2023-11-28T18:03:02.585 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T18:03:02.585 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457 --owner scheduled_vshankar@teuthology --verbose --name vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} -- /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/orig.config.yaml 2023-11-28T18:03:02.636 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T18:03:02.727 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457 branch: wip-vshankar-testing-20231127.102654 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} email: vshankar@redhat.com first_in_suite: false job_id: '7469457' kernel: flavor: default kdb: true sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-vshankar-testing-20231127.102654 ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds - SLOW_OPS - slow request sha1: 47c1c547372ed5bff7984647dc200bac0e019caf ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 47c1c547372ed5bff7984647dc200bac0e019caf thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20231127.102654 sha1: 47c1c547372ed5bff7984647dc200bac0e019caf owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1098 sha1: 47c1c547372ed5bff7984647dc200bac0e019caf sleep_before_teardown: 0 subset: 854/1024 suite: fs suite_branch: wip-vshankar-testing-20231127.102654 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 47c1c547372ed5bff7984647dc200bac0e019caf syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ6dCvqETJOdN7f1Epo6cniZht7c5goNuYAOZUBrh9thfzQcM2F3QKi5mYaL+zO29hg6o7/fjXX4Y1jFJrsaJyc= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBgCbSH1Nj/42ryCnkWgSNJF0gbI6Oe+/+KDqHSBCnArI3dbGEbt4LSUv3/4QJAvpTe6bSYl7wBvSQEb1HWidsk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 5ae09a5cc7064ca2bf994df19f3006b79794789a timestamp: 2023-11-28_06:23:42 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-28T18:03:02.727 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa; will attempt to use it 2023-11-28T18:03:02.728 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks 2023-11-28T18:03:02.728 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-28T18:03:02.730 INFO:teuthology.task.internal:Checking packages... 2023-11-28T18:03:02.757 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '47c1c547372ed5bff7984647dc200bac0e019caf' 2023-11-28T18:03:02.758 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-28T18:03:02.758 INFO:teuthology.packaging:ref: None 2023-11-28T18:03:02.758 INFO:teuthology.packaging:tag: None 2023-11-28T18:03:02.758 INFO:teuthology.packaging:branch: wip-vshankar-testing-20231127.102654 2023-11-28T18:03:02.759 INFO:teuthology.packaging:sha1: 47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T18:03:02.759 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-vshankar-testing-20231127.102654 2023-11-28T18:03:03.091 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:03:03.092 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-28T18:03:03.097 INFO:teuthology.task.internal:no buildpackages task found 2023-11-28T18:03:03.097 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-28T18:03:03.146 INFO:teuthology.task.internal:Saving configuration 2023-11-28T18:03:03.166 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-28T18:03:03.180 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T18:03:03.207 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 17:57:36.705776', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEGMzGuWGeet2sEw4NU5pT5KzbFLcWcsXfHQK2HttNhX'} 2023-11-28T18:03:03.231 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 17:57:36.703938', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILyjC7P5dzpgr8hhB9ZNt6LMZeeSKEbaKicV+99XYFzf'} 2023-11-28T18:03:03.231 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-28T18:03:03.238 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-11-28T18:03:03.238 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-11-28T18:03:03.238 INFO:teuthology.run_tasks:Running task console_log... 2023-11-28T18:03:03.340 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4305f6f040>, signals=[15]) 2023-11-28T18:03:03.340 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-28T18:03:03.387 INFO:teuthology.task.internal:Opening connections... 2023-11-28T18:03:03.388 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:03:03.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:03:03.505 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:03:03.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:03:03.579 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-28T18:03:03.586 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2023-11-28T18:03:03.591 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2023-11-28T18:03:03.592 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="22.04" 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=jammy 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-28T18:03:03.639 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-28T18:03:03.640 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-28T18:03:03.640 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-28T18:03:03.640 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=jammy 2023-11-28T18:03:03.640 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2023-11-28T18:03:03.661 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2023-11-28T18:03:03.667 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2023-11-28T18:03:03.667 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2023-11-28T18:03:03.715 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-28T18:03:03.716 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="Ubuntu" 2023-11-28T18:03:03.716 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="22.04" 2023-11-28T18:03:03.716 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-28T18:03:03.716 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_CODENAME=jammy 2023-11-28T18:03:03.716 INFO:teuthology.orchestra.run.smithi193.stdout:ID=ubuntu 2023-11-28T18:03:03.716 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE=debian 2023-11-28T18:03:03.717 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-28T18:03:03.717 INFO:teuthology.orchestra.run.smithi193.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-28T18:03:03.717 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-28T18:03:03.717 INFO:teuthology.orchestra.run.smithi193.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-28T18:03:03.717 INFO:teuthology.orchestra.run.smithi193.stdout:UBUNTU_CODENAME=jammy 2023-11-28T18:03:03.717 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2023-11-28T18:03:03.736 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-28T18:03:03.746 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-28T18:03:03.772 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-28T18:03:03.772 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2023-11-28T18:03:03.775 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2023-11-28T18:03:03.781 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-28T18:03:03.803 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-28T18:03:03.804 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2023-11-28T18:03:03.821 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2023-11-28T18:03:03.842 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-28T18:03:03.900 INFO:teuthology.run_tasks:Running task kernel... 2023-11-28T18:03:03.915 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'} 2023-11-28T18:03:03.915 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, timeout 300 2023-11-28T18:03:03.915 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T18:03:03.916 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T18:03:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:03:03.920 DEBUG:teuthology.task.kernel:sha1 for mon.a is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T18:03:03.920 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-28T18:03:03.920 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-11-28T18:03:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:03:03.922 DEBUG:teuthology.task.kernel:sha1 for mon.b is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T18:03:03.922 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-28T18:03:03.922 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2023-11-28T18:03:03.968 INFO:teuthology.orchestra.run.smithi007.stdout:5.15.0-56-generic 2023-11-28T18:03:03.968 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T18:03:03.969 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-28T18:03:03.969 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T18:03:03.973 INFO:teuthology.orchestra.run.smithi193.stdout:5.15.0-56-generic 2023-11-28T18:03:03.973 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T18:03:03.974 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-28T18:03:03.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T18:03:04.187 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2023-11-28T18:03:04.188 DEBUG:teuthology.orchestra.run.smithi007:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-11-28T18:03:04.200 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2023-11-28T18:03:04.200 DEBUG:teuthology.orchestra.run.smithi193:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-11-28T18:03:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-11-28T18:03:04.225 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-11-28T18:03:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-11-28T18:03:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-11-28T18:03:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-11-28T18:03:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:hint: 2023-11-28T18:03:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:hint: git config --global init.defaultBranch 2023-11-28T18:03:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:hint: 2023-11-28T18:03:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-11-28T18:03:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-11-28T18:03:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:hint: 2023-11-28T18:03:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:hint: git branch -m 2023-11-28T18:03:04.347 INFO:teuthology.orchestra.run.smithi007.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-11-28T18:03:04.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-11-28T18:03:04.360 INFO:teuthology.orchestra.run.smithi193.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-11-28T18:03:04.360 INFO:teuthology.orchestra.run.smithi193.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-11-28T18:03:04.360 INFO:teuthology.orchestra.run.smithi193.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-11-28T18:03:04.360 INFO:teuthology.orchestra.run.smithi193.stderr:hint: 2023-11-28T18:03:04.360 INFO:teuthology.orchestra.run.smithi193.stderr:hint: git config --global init.defaultBranch 2023-11-28T18:03:04.361 INFO:teuthology.orchestra.run.smithi193.stderr:hint: 2023-11-28T18:03:04.361 INFO:teuthology.orchestra.run.smithi193.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-11-28T18:03:04.361 INFO:teuthology.orchestra.run.smithi193.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-11-28T18:03:04.361 INFO:teuthology.orchestra.run.smithi193.stderr:hint: 2023-11-28T18:03:04.361 INFO:teuthology.orchestra.run.smithi193.stderr:hint: git branch -m 2023-11-28T18:03:04.362 INFO:teuthology.orchestra.run.smithi193.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-11-28T18:03:04.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-11-28T18:03:04.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-11-28T18:03:04.454 DEBUG:teuthology.orchestra.run.smithi193:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-11-28T18:03:04.467 DEBUG:teuthology.orchestra.run.smithi007:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-11-28T18:03:04.512 DEBUG:teuthology.orchestra.run.smithi193:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-11-28T18:07:18.665 INFO:teuthology.orchestra.run.smithi193.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-11-28T18:07:18.666 INFO:teuthology.orchestra.run.smithi193.stderr: * [new branch] main -> origin/main 2023-11-28T18:07:18.697 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20190312 -> 20190312 2023-11-28T18:07:18.697 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20190416 -> 20190416 2023-11-28T18:07:18.697 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20190514 -> 20190514 2023-11-28T18:07:18.697 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20190618 -> 20190618 2023-11-28T18:07:18.699 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20190717 -> 20190717 2023-11-28T18:07:18.699 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20190815 -> 20190815 2023-11-28T18:07:18.699 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20190923 -> 20190923 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20191022 -> 20191022 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20191215 -> 20191215 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20200122 -> 20200122 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20200316 -> 20200316 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20200421 -> 20200421 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20200519 -> 20200519 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20200619 -> 20200619 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20200721 -> 20200721 2023-11-28T18:07:18.700 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20200817 -> 20200817 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20200918 -> 20200918 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20201022 -> 20201022 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20201118 -> 20201118 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20201218 -> 20201218 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20210208 -> 20210208 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20210315 -> 20210315 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20210511 -> 20210511 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20210716 -> 20210716 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20210818 -> 20210818 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20210919 -> 20210919 2023-11-28T18:07:18.702 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20211027 -> 20211027 2023-11-28T18:07:18.703 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20211216 -> 20211216 2023-11-28T18:07:18.703 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20220209 -> 20220209 2023-11-28T18:07:18.703 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20220310 -> 20220310 2023-11-28T18:07:18.704 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20220411 -> 20220411 2023-11-28T18:07:18.704 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20220509 -> 20220509 2023-11-28T18:07:18.704 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20220610 -> 20220610 2023-11-28T18:07:18.705 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20220708 -> 20220708 2023-11-28T18:07:18.705 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20220815 -> 20220815 2023-11-28T18:07:18.705 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20220913 -> 20220913 2023-11-28T18:07:18.705 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20221012 -> 20221012 2023-11-28T18:07:18.705 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20221109 -> 20221109 2023-11-28T18:07:18.705 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20221214 -> 20221214 2023-11-28T18:07:18.706 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20230117 -> 20230117 2023-11-28T18:07:18.706 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20230210 -> 20230210 2023-11-28T18:07:18.706 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20230310 -> 20230310 2023-11-28T18:07:18.707 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20230404 -> 20230404 2023-11-28T18:07:18.707 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20230515 -> 20230515 2023-11-28T18:07:18.707 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20230625 -> 20230625 2023-11-28T18:07:18.707 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20230804 -> 20230804 2023-11-28T18:07:18.707 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20230919 -> 20230919 2023-11-28T18:07:18.707 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20231030 -> 20231030 2023-11-28T18:07:18.708 INFO:teuthology.orchestra.run.smithi193.stderr: * [new tag] 20231111 -> 20231111 2023-11-28T18:07:22.707 INFO:teuthology.orchestra.run.smithi193.stderr:Updating files: 50% (1637/3250) Updating files: 51% (1658/3250) Updating files: 52% (1690/3250) Updating files: 53% (1723/3250) Updating files: 54% (1755/3250) Updating files: 55% (1788/3250) Updating files: 56% (1820/3250) Updating files: 57% (1853/3250) Updating files: 58% (1885/3250) Updating files: 59% (1918/3250) Updating files: 60% (1950/3250) Updating files: 61% (1983/3250) Updating files: 62% (2015/3250) Updating files: 63% (2048/3250) Updating files: 64% (2080/3250) Updating files: 64% (2104/3250) Updating files: 65% (2113/3250) Updating files: 66% (2145/3250) Updating files: 67% (2178/3250) Updating files: 68% (2210/3250) Updating files: 69% (2243/3250) Updating files: 70% (2275/3250) Updating files: 71% (2308/3250) Updating files: 72% (2340/3250) Updating files: 73% (2373/3250) Updating files: 74% (2405/3250) Updating files: 75% (2438/3250) Updating files: 76% (2470/3250) Updating files: 77% (2503/3250) Updating files: 78% (2535/3250) Updating files: 79% (2568/3250) Updating files: 80% (2600/3250) Updating files: 81% (2633/3250) Updating files: 81% (2650/3250) Updating files: 82% (2665/3250) Updating files: 83% (2698/3250) Updating files: 84% (2730/3250) Updating files: 85% (2763/3250) Updating files: 86% (2795/3250) Updating files: 87% (2828/3250) Updating files: 88% (2860/3250) Updating files: 89% (2893/3250) Updating files: 90% (2925/3250) Updating files: 91% (2958/3250) Updating files: 92% (2990/3250) Updating files: 93% (3023/3250) Updating files: 94% (3055/3250) Updating files: 95% (3088/3250) Updating files: 96% (3120/3250) Updating files: 97% (3153/3250) Updating files: 98% (3185/3250) Updating files: 99% (3218/3250) Updating files: 100% (3250/3250) Updating files: 100% (3250/3250), done. 2023-11-28T18:07:22.711 INFO:teuthology.orchestra.run.smithi193.stdout:HEAD is now at aae60524 Merge branch 'make-opts' into 'main' 2023-11-28T18:07:22.733 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.b... 2023-11-28T18:07:22.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T18:07:22.883 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g8247bbb5b470/ 2023-11-28T18:07:22.884 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-g8247bbb5b470_6.6.0-g8247bbb5b470-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g8247bbb5b470/ --input-file=- 2023-11-28T18:07:22.887 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi193.front.sepia.ceph.com to complete... 2023-11-28T18:07:24.588 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28 18:07:24 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g8247bbb5b470/linux-image-6.6.0-g8247bbb5b470_6.6.0-g8247bbb5b470-1_amd64.deb [38938726/38938726] -> "/tmp/linux-image.deb" [1] 2023-11-28T18:07:24.588 INFO:teuthology.orchestra.run.smithi193.stderr:FINISHED --2023-11-28 18:07:24-- 2023-11-28T18:07:24.588 INFO:teuthology.orchestra.run.smithi193.stderr:Total wall clock time: 1.7s 2023-11-28T18:07:24.588 INFO:teuthology.orchestra.run.smithi193.stderr:Downloaded: 1 files, 37M in 1.5s (25.2 MB/s) 2023-11-28T18:07:24.591 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.b... 2023-11-28T18:07:24.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo dpkg -i /tmp/linux-image.deb 2023-11-28T18:07:24.836 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package linux-image-6.6.0-g8247bbb5b470. 2023-11-28T18:07:26.799 INFO:teuthology.orchestra.run.smithi193.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-11-28T18:07:26.804 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-11-28T18:07:26.949 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking linux-image-6.6.0-g8247bbb5b470 (6.6.0-g8247bbb5b470-1) ... 2023-11-28T18:07:28.021 INFO:teuthology.orchestra.run.smithi007.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-11-28T18:07:28.021 INFO:teuthology.orchestra.run.smithi007.stderr: * [new branch] main -> origin/main 2023-11-28T18:07:28.065 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20190312 -> 20190312 2023-11-28T18:07:28.065 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20190416 -> 20190416 2023-11-28T18:07:28.065 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20190514 -> 20190514 2023-11-28T18:07:28.066 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20190618 -> 20190618 2023-11-28T18:07:28.066 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20190717 -> 20190717 2023-11-28T18:07:28.066 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20190815 -> 20190815 2023-11-28T18:07:28.066 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20190923 -> 20190923 2023-11-28T18:07:28.066 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20191022 -> 20191022 2023-11-28T18:07:28.067 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20191215 -> 20191215 2023-11-28T18:07:28.067 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20200122 -> 20200122 2023-11-28T18:07:28.067 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20200316 -> 20200316 2023-11-28T18:07:28.067 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20200421 -> 20200421 2023-11-28T18:07:28.067 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20200519 -> 20200519 2023-11-28T18:07:28.068 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20200619 -> 20200619 2023-11-28T18:07:28.068 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20200721 -> 20200721 2023-11-28T18:07:28.068 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20200817 -> 20200817 2023-11-28T18:07:28.068 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20200918 -> 20200918 2023-11-28T18:07:28.068 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20201022 -> 20201022 2023-11-28T18:07:28.069 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20201118 -> 20201118 2023-11-28T18:07:28.069 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20201218 -> 20201218 2023-11-28T18:07:28.069 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20210208 -> 20210208 2023-11-28T18:07:28.069 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20210315 -> 20210315 2023-11-28T18:07:28.069 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20210511 -> 20210511 2023-11-28T18:07:28.070 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20210716 -> 20210716 2023-11-28T18:07:28.070 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20210818 -> 20210818 2023-11-28T18:07:28.070 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20210919 -> 20210919 2023-11-28T18:07:28.070 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20211027 -> 20211027 2023-11-28T18:07:28.070 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20211216 -> 20211216 2023-11-28T18:07:28.070 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20220209 -> 20220209 2023-11-28T18:07:28.071 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20220310 -> 20220310 2023-11-28T18:07:28.071 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20220411 -> 20220411 2023-11-28T18:07:28.071 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20220509 -> 20220509 2023-11-28T18:07:28.071 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20220610 -> 20220610 2023-11-28T18:07:28.071 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20220708 -> 20220708 2023-11-28T18:07:28.072 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20220815 -> 20220815 2023-11-28T18:07:28.072 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20220913 -> 20220913 2023-11-28T18:07:28.072 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20221012 -> 20221012 2023-11-28T18:07:28.072 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20221109 -> 20221109 2023-11-28T18:07:28.072 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20221214 -> 20221214 2023-11-28T18:07:28.073 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20230117 -> 20230117 2023-11-28T18:07:28.073 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20230210 -> 20230210 2023-11-28T18:07:28.073 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20230310 -> 20230310 2023-11-28T18:07:28.073 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20230404 -> 20230404 2023-11-28T18:07:28.073 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20230515 -> 20230515 2023-11-28T18:07:28.074 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20230625 -> 20230625 2023-11-28T18:07:28.074 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20230804 -> 20230804 2023-11-28T18:07:28.074 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20230919 -> 20230919 2023-11-28T18:07:28.074 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20231030 -> 20231030 2023-11-28T18:07:28.074 INFO:teuthology.orchestra.run.smithi007.stderr: * [new tag] 20231111 -> 20231111 2023-11-28T18:07:30.603 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up linux-image-6.6.0-g8247bbb5b470 (6.6.0-g8247bbb5b470-1) ... 2023-11-28T18:07:30.697 INFO:teuthology.orchestra.run.smithi193.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-g8247bbb5b470 2023-11-28T18:07:32.263 INFO:teuthology.orchestra.run.smithi007.stderr:Updating files: 48% (1590/3250) Updating files: 49% (1593/3250) Updating files: 50% (1625/3250) Updating files: 51% (1658/3250) Updating files: 52% (1690/3250) Updating files: 53% (1723/3250) Updating files: 54% (1755/3250) Updating files: 55% (1788/3250) Updating files: 56% (1820/3250) Updating files: 57% (1853/3250) Updating files: 58% (1885/3250) Updating files: 59% (1918/3250) Updating files: 60% (1950/3250) Updating files: 61% (1983/3250) Updating files: 62% (2015/3250) Updating files: 63% (2048/3250) Updating files: 63% (2074/3250) Updating files: 64% (2080/3250) Updating files: 65% (2113/3250) Updating files: 66% (2145/3250) Updating files: 67% (2178/3250) Updating files: 68% (2210/3250) Updating files: 69% (2243/3250) Updating files: 70% (2275/3250) Updating files: 71% (2308/3250) Updating files: 72% (2340/3250) Updating files: 73% (2373/3250) Updating files: 74% (2405/3250) Updating files: 75% (2438/3250) Updating files: 76% (2470/3250) Updating files: 77% (2503/3250) Updating files: 78% (2535/3250) Updating files: 79% (2568/3250) Updating files: 79% (2583/3250) Updating files: 80% (2600/3250) Updating files: 81% (2633/3250) Updating files: 82% (2665/3250) Updating files: 83% (2698/3250) Updating files: 84% (2730/3250) Updating files: 85% (2763/3250) Updating files: 85% (2792/3250) Updating files: 86% (2795/3250) Updating files: 87% (2828/3250) Updating files: 88% (2860/3250) Updating files: 89% (2893/3250) Updating files: 90% (2925/3250) Updating files: 91% (2958/3250) Updating files: 92% (2990/3250) Updating files: 93% (3023/3250) Updating files: 94% (3055/3250) Updating files: 95% (3088/3250) Updating files: 96% (3120/3250) Updating files: 97% (3153/3250) Updating files: 98% (3185/3250) Updating files: 99% (3218/3250) Updating files: 100% (3250/3250) Updating files: 100% (3250/3250), done. 2023-11-28T18:07:32.268 INFO:teuthology.orchestra.run.smithi007.stdout:HEAD is now at aae60524 Merge branch 'make-opts' into 'main' 2023-11-28T18:07:32.290 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.a... 2023-11-28T18:07:32.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T18:07:32.431 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g8247bbb5b470/ 2023-11-28T18:07:32.431 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-g8247bbb5b470_6.6.0-g8247bbb5b470-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g8247bbb5b470/ --input-file=- 2023-11-28T18:07:32.434 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi007.front.sepia.ceph.com to complete... 2023-11-28T18:07:34.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28 18:07:34 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g8247bbb5b470/linux-image-6.6.0-g8247bbb5b470_6.6.0-g8247bbb5b470-1_amd64.deb [38938726/38938726] -> "/tmp/linux-image.deb" [1] 2023-11-28T18:07:34.976 INFO:teuthology.orchestra.run.smithi007.stderr:FINISHED --2023-11-28 18:07:34-- 2023-11-28T18:07:34.977 INFO:teuthology.orchestra.run.smithi007.stderr:Total wall clock time: 2.5s 2023-11-28T18:07:34.977 INFO:teuthology.orchestra.run.smithi007.stderr:Downloaded: 1 files, 37M in 2.3s (16.3 MB/s) 2023-11-28T18:07:34.979 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.a... 2023-11-28T18:07:34.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo dpkg -i /tmp/linux-image.deb 2023-11-28T18:07:35.212 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-6.6.0-g8247bbb5b470. 2023-11-28T18:07:37.266 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-11-28T18:07:37.270 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-11-28T18:07:37.417 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-6.6.0-g8247bbb5b470 (6.6.0-g8247bbb5b470-1) ... 2023-11-28T18:07:40.898 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-6.6.0-g8247bbb5b470 (6.6.0-g8247bbb5b470-1) ... 2023-11-28T18:07:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-g8247bbb5b470 2023-11-28T18:07:52.007 INFO:teuthology.orchestra.run.smithi193.stderr:Sourcing file `/etc/default/grub' 2023-11-28T18:07:52.024 INFO:teuthology.orchestra.run.smithi193.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-28T18:07:52.054 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2023-11-28T18:07:52.372 INFO:teuthology.orchestra.run.smithi193.stderr:Found linux image: /boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T18:07:52.388 INFO:teuthology.orchestra.run.smithi193.stderr:Found initrd image: /boot/initrd.img-6.6.0-g8247bbb5b470 2023-11-28T18:07:52.697 INFO:teuthology.orchestra.run.smithi193.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-28T18:07:52.705 INFO:teuthology.orchestra.run.smithi193.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-28T18:07:52.764 INFO:teuthology.orchestra.run.smithi193.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-28T18:07:52.769 INFO:teuthology.orchestra.run.smithi193.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-28T18:07:52.908 INFO:teuthology.orchestra.run.smithi193.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-28T18:07:52.909 INFO:teuthology.orchestra.run.smithi193.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-28T18:07:52.909 INFO:teuthology.orchestra.run.smithi193.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-28T18:07:52.959 INFO:teuthology.orchestra.run.smithi193.stderr:done 2023-11-28T18:07:53.176 DEBUG:teuthology.orchestra.run.smithi193:> dpkg-deb -c /tmp/linux-image.deb 2023-11-28T18:07:53.184 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./ 2023-11-28T18:07:53.184 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./boot/ 2023-11-28T18:07:53.184 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 4415617 2023-11-23 03:47 ./boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T18:07:53.197 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 169214 2023-11-23 03:47 ./boot/config-6.6.0-g8247bbb5b470 2023-11-28T18:07:53.197 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11352832 2023-11-23 03:47 ./boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T18:07:53.214 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/ 2023-11-28T18:07:53.214 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/ 2023-11-28T18:07:53.214 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/postinst.d/ 2023-11-28T18:07:53.214 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/postrm.d/ 2023-11-28T18:07:53.214 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/preinst.d/ 2023-11-28T18:07:53.214 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/prerm.d/ 2023-11-28T18:07:53.214 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/ 2023-11-28T18:07:53.215 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/ 2023-11-28T18:07:53.215 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/ 2023-11-28T18:07:53.215 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/ 2023-11-28T18:07:53.215 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/ 2023-11-28T18:07:53.215 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/ 2023-11-28T18:07:53.215 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ 2023-11-28T18:07:53.215 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 490233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/aesni-intel.ko 2023-11-28T18:07:53.217 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-28T18:07:53.217 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-28T18:07:53.217 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-28T18:07:53.217 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 93113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-28T18:07:53.217 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 106729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 152241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 83889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 58585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-28T18:07:53.218 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/ 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/ 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/ 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpuid.ko 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/msr.ko 2023-11-28T18:07:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/ 2023-11-28T18:07:53.221 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 354433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-28T18:07:53.221 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 881625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-28T18:07:53.222 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 2275865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/af_alg.ko 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/algif_hash.ko 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/algif_skcipher.ko 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/asymmetric_keys/ 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-11-28T18:07:53.233 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/ 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-28T18:07:53.234 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cryptd.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_simd.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-28T18:07:53.235 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lrw.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-28T18:07:53.236 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 149193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xts.ko 2023-11-28T18:07:53.237 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_extlog.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/apei/ 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/apei/einj.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-28T18:07:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 103025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 88865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 88289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-28T18:07:53.239 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-28T18:07:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-28T18:07:53.241 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-28T18:07:53.241 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_legacy.ko 2023-11-28T18:07:53.241 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-28T18:07:53.241 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-28T18:07:53.241 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-28T18:07:53.242 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-28T18:07:53.243 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_radisys.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rz1000.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-28T18:07:53.244 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 83345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ 2023-11-28T18:07:53.245 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/ 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 124473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/ 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/ 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 113665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/ 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 965585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 173225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/ 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 94297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 115793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-28T18:07:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/ 2023-11-28T18:07:53.247 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 110385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-28T18:07:53.247 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 98193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-28T18:07:53.247 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 263769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-28T18:07:53.247 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/ 2023-11-28T18:07:53.247 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 98145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-28T18:07:53.410 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/ 2023-11-28T18:07:53.411 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-28T18:07:53.411 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ 2023-11-28T18:07:53.411 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/applicom.ko 2023-11-28T18:07:53.411 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-28T18:07:53.411 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/ 2023-11-28T18:07:53.411 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 148601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 139313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-28T18:07:53.412 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/ 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 84097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/nvram.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/ 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-28T18:07:53.413 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-28T18:07:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-11-28T18:07:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/xillybus/ 2023-11-28T18:07:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/xillybus/xillybus_class.ko 2023-11-28T18:07:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/xillybus/xillybus_core.ko 2023-11-28T18:07:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-11-28T18:07:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/ 2023-11-28T18:07:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/clk-palmas.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/clk-si5351.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/clk-twl6040.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/clk-wm831x.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/ 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 243025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/ 2023-11-28T18:07:53.415 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/ 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/ 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 438761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/ 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/ 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/ 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-28T18:07:53.416 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ 2023-11-28T18:07:53.417 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 122569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-28T18:07:53.417 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ 2023-11-28T18:07:53.417 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 79953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-28T18:07:53.417 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-28T18:07:53.417 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 63617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-28T18:07:53.417 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-28T18:07:53.417 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-28T18:07:53.417 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-28T18:07:53.418 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-28T18:07:53.418 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-28T18:07:53.418 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 52041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-28T18:07:53.418 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-28T18:07:53.418 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-28T18:07:53.418 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 56825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/ 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-gpio.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-max14577.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-max77693.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-max8997.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-palmas.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-rt8973a.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-sm5502.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/ 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/dmi-sysfs.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/efi/ 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/efi/efi-pstore.ko 2023-11-28T18:07:53.419 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/ 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-adp5520.ko 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-amd8111.ko 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-arizona.ko 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-da9052.ko 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-da9055.ko 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-dln2.ko 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 77849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-f7188x.ko 2023-11-28T18:07:53.420 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-generic.ko 2023-11-28T18:07:53.421 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-28T18:07:53.421 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-11-28T18:07:53.421 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-kempld.ko 2023-11-28T18:07:53.421 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-lp3943.ko 2023-11-28T18:07:53.421 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-max7300.ko 2023-11-28T18:07:53.421 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-max730x.ko 2023-11-28T18:07:53.421 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-max732x.ko 2023-11-28T18:07:53.422 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-11-28T18:07:53.422 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-pca953x.ko 2023-11-28T18:07:53.422 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-pcf857x.ko 2023-11-28T18:07:53.422 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-rdc321x.ko 2023-11-28T18:07:53.422 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-sch.ko 2023-11-28T18:07:53.422 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-sch311x.ko 2023-11-28T18:07:53.423 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-tps65912.ko 2023-11-28T18:07:53.423 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-twl4030.ko 2023-11-28T18:07:53.423 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-twl6040.ko 2023-11-28T18:07:53.423 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-28T18:07:53.423 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-vx855.ko 2023-11-28T18:07:53.423 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-wm831x.ko 2023-11-28T18:07:53.423 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-wm8350.ko 2023-11-28T18:07:53.424 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-wm8994.ko 2023-11-28T18:07:53.424 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/ 2023-11-28T18:07:53.424 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-a4tech.ko 2023-11-28T18:07:53.424 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-apple.ko 2023-11-28T18:07:53.424 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-28T18:07:53.424 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-28T18:07:53.425 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-28T18:07:53.425 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-belkin.ko 2023-11-28T18:07:53.425 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-cherry.ko 2023-11-28T18:07:53.425 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-chicony.ko 2023-11-28T18:07:53.425 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-cp2112.ko 2023-11-28T18:07:53.425 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-cypress.ko 2023-11-28T18:07:53.426 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-28T18:07:53.426 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-28T18:07:53.426 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-28T18:07:53.426 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-28T18:07:53.426 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ezkey.ko 2023-11-28T18:07:53.426 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-28T18:07:53.427 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-generic.ko 2023-11-28T18:07:53.427 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-28T18:07:53.427 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-28T18:07:53.427 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-28T18:07:53.427 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-28T18:07:53.427 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-28T18:07:53.427 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-28T18:07:53.427 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-28T18:07:53.428 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kensington.ko 2023-11-28T18:07:53.428 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-28T18:07:53.428 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-28T18:07:53.428 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-28T18:07:53.428 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-28T18:07:53.428 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-28T18:07:53.428 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lg-g15.ko 2023-11-28T18:07:53.428 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 58185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 101481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 80993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-magicmouse.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-microsoft.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-monterey.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ntrig.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-28T18:07:53.429 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 72937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-28T18:07:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-28T18:07:53.431 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-28T18:07:53.431 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-28T18:07:53.431 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-28T18:07:53.431 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-28T18:07:53.431 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-28T18:07:53.431 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-28T18:07:53.431 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-28T18:07:53.431 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-28T18:07:53.432 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-28T18:07:53.432 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-28T18:07:53.432 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-28T18:07:53.432 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-28T18:07:53.432 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 71073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 143473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-28T18:07:53.433 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-28T18:07:53.434 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-28T18:07:53.434 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-28T18:07:53.434 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 267825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid.ko 2023-11-28T18:07:53.434 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-28T18:07:53.434 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/usbhid/ 2023-11-28T18:07:53.434 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 125721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/usbhid/usbhid.ko 2023-11-28T18:07:53.434 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/usbhid/usbkbd.ko 2023-11-28T18:07:53.434 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/usbhid/usbmouse.ko 2023-11-28T18:07:53.435 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 217873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-28T18:07:53.435 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/ 2023-11-28T18:07:53.435 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 64201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-28T18:07:53.435 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 89625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-28T18:07:53.435 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 305465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-28T18:07:53.435 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ 2023-11-28T18:07:53.436 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-28T18:07:53.436 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 61393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-28T18:07:53.436 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-28T18:07:53.436 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-28T18:07:53.436 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-28T18:07:53.436 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-28T18:07:53.437 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-28T18:07:53.437 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 97977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-28T18:07:53.437 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-28T18:07:53.437 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 45553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-28T18:07:53.437 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-28T18:07:53.437 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-28T18:07:53.437 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-28T18:07:53.438 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-28T18:07:53.438 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 74433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-28T18:07:53.438 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-28T18:07:53.438 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-28T18:07:53.438 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-28T18:07:53.438 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-28T18:07:53.439 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 55201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-28T18:07:53.439 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-28T18:07:53.439 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 58617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-28T18:07:53.439 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-28T18:07:53.439 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-28T18:07:53.439 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-28T18:07:53.440 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/da9052-hwmon.ko 2023-11-28T18:07:53.440 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/da9055-hwmon.ko 2023-11-28T18:07:53.440 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-28T18:07:53.440 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 99601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-28T18:07:53.440 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-28T18:07:53.441 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-28T18:07:53.441 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-28T18:07:53.441 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc2103.ko 2023-11-28T18:07:53.441 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-28T18:07:53.441 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-28T18:07:53.442 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 94457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-28T18:07:53.442 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-28T18:07:53.442 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-28T18:07:53.442 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 55449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-28T18:07:53.442 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-28T18:07:53.442 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-28T18:07:53.443 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 46801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-28T18:07:53.443 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 45529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-28T18:07:53.443 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hih6130.ko 2023-11-28T18:07:53.443 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-28T18:07:53.443 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-28T18:07:53.443 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-28T18:07:53.444 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-28T18:07:53.444 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-28T18:07:53.444 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-28T18:07:53.444 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-28T18:07:53.444 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 130449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-28T18:07:53.444 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/jc42.ko 2023-11-28T18:07:53.445 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-28T18:07:53.445 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-28T18:07:53.445 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-28T18:07:53.445 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 48601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-28T18:07:53.445 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-28T18:07:53.446 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-28T18:07:53.446 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-28T18:07:53.446 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-28T18:07:53.446 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-28T18:07:53.446 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-28T18:07:53.446 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 72833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-28T18:07:53.447 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 46153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-28T18:07:53.447 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-28T18:07:53.447 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-28T18:07:53.447 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 108841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-28T18:07:53.447 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-28T18:07:53.447 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-28T18:07:53.448 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-28T18:07:53.448 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-28T18:07:53.448 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-28T18:07:53.448 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-28T18:07:53.448 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-28T18:07:53.448 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mc13783-adc.ko 2023-11-28T18:07:53.449 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 119817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 59737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 82337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-28T18:07:53.450 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 82577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-28T18:07:53.451 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-28T18:07:53.452 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 52457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 57449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 71345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 63225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 71321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 101473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 119505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-28T18:07:53.453 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/ 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/ 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/ 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-28T18:07:53.454 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 6929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 62817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-28T18:07:53.455 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-11-28T18:07:53.456 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/muxes/ 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-11-28T18:07:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 273609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 879969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 69273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 324217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 97985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 282273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/ 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/ 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 493073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-28T18:07:53.458 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/ 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 419737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/ 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 739025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 296625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/ocrdma/ 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 153697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/usnic/ 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 168969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 263081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-28T18:07:53.459 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 100241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 111425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 128753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 129529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/ 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/evbug.ko 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/ff-memless.ko 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/ 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-28T18:07:53.460 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/ 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/lkkbd.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/lm8323.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/lm8333.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-11-28T18:07:53.461 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/newtonkbd.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/qt1070.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/qt2160.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/stowaway.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/sunkbd.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-11-28T18:07:53.462 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/xtkbd.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/matrix-keymap.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/ 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 123681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 89449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/gpio_mouse.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 335929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/psmouse.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-28T18:07:53.463 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/ 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 109873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/ 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/ct82c710.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/parkbd.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/pcips2.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/ps2mult.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serport.ko 2023-11-28T18:07:53.464 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/ 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/ 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/ 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-88pm860x.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-adp5520.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 59753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-bd2802.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-da903x.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-da9052.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-gpio.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-28T18:07:53.465 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3533.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm355x.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3642.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp8788.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-max8997.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-mc13783.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-menf21bmc.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-pca9532.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-pca955x.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-pca963x.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-28T18:07:53.466 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-tca6507.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-wm831x-status.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-wm8350.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-28T18:07:53.467 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-28T18:07:53.469 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/ 2023-11-28T18:07:53.469 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/ 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 654969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 89601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bufio.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 156009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 106897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-era.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-28T18:07:53.470 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-mirror.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-region-hash.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 109529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-snapshot.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-28T18:07:53.471 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 181865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-zero.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/ 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 199937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 113073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 158681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 395721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-28T18:07:53.472 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/ 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/ 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 178137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 63385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 123321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 79041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/ 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/88pm800.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/88pm805.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/88pm80x.ko 2023-11-28T18:07:53.473 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/arizona-i2c.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 123721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/arizona.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/bcm590xx.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/cros_ec_dev.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/dln2.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/janz-cmodio.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/kempld-core.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lm3533-core.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lp3943.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/max8907.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mc13xxx-core.ko 2023-11-28T18:07:53.474 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/menf21bmc.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/pcf50633-adc.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/pcf50633-gpio.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/pcf50633.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/retu-mfd.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 80425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/si476x-core.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 48457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/tps65010.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/tps6507x.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-28T18:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wm8994.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 48489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ad525x_dpot.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/ 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 46585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 45593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/c2port/ 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/c2port/core.ko 2023-11-28T18:07:53.476 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/ 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ds1682.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/dummy-irq.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/echo/ 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/echo/echo.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/ 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/genwqe/ 2023-11-28T18:07:53.477 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 152201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hmc6352.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ibmasm/ 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 72129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ics932s401.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/ 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/ 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 125209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-28T18:07:53.478 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 56521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 310113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/phantom.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ti-st/ 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 74265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ti-st/st_drv.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 48697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/ 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 183609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/ 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 490129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-28T18:07:53.479 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/ 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/ 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 106033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/ 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 453737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/ 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 45785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-28T18:07:53.480 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 130553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 68849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 59689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 99049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/ 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/ 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 69137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/ 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 59585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/ 2023-11-28T18:07:53.481 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/ 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 87297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/ 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/ 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 97337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 104673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 90257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/ 2023-11-28T18:07:53.482 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 81569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 69729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/ 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 176761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1421497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 128569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/ 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 132089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 312081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/ 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/ 2023-11-28T18:07:53.483 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 323513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/ 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 193497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 347417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1044497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 166193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/ 2023-11-28T18:07:53.484 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/ 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 204345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/ 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/ 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 69449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 61489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 141225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 62817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/ 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-28T18:07:53.485 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 62833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 45721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ec_bhf.ko 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/ 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/ 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 281137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fujitsu/ 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 45345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 90465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/ 2023-11-28T18:07:53.486 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 295577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/ 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 584497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/ 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 246209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/ 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 925825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/ 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 302937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/ 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 514641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/ 2023-11-28T18:07:53.487 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 93593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 734305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 151769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 86921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/ 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 102289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 122649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/ 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-11-28T18:07:53.488 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 727601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 278937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1113681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 95665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/ 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 108649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-28T18:07:53.489 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 73441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/ 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 150905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/ 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 136721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/ 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 43145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/ 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-11-28T18:07:53.490 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 227897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 87081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 672049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/ 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 43017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/ 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 83985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 189801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/samsung/ 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-11-28T18:07:53.491 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 104241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/ 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1052497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/ 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/ 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 63337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/ 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 61537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 73409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 68177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-28T18:07:53.492 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/ 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 487025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/ 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 103337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 166097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/ 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-28T18:07:53.493 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/ 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 86529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/ 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 80273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 83633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/ 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 46241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/ 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 59993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/fddi/ 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/fddi/defxx.ko 2023-11-28T18:07:53.494 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/fddi/skfp/ 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 216985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/fddi/skfp/skfp.ko 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 77345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/ 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 235521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/ 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 83769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 62145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/ 2023-11-28T18:07:53.495 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-gpio.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 43953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/ 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-28T18:07:53.496 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 82049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 76113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 97857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-28T18:07:53.497 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-28T18:07:53.498 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-28T18:07:53.498 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-28T18:07:53.498 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-28T18:07:53.498 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/plip/ 2023-11-28T18:07:53.498 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 43505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/plip/plip.ko 2023-11-28T18:07:53.498 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ 2023-11-28T18:07:53.498 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-28T18:07:53.498 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sb1000.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/ 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 64449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-28T18:07:53.499 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/ 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 106169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 87017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/ 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 144297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/ 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 297329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/ 2023-11-28T18:07:53.500 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 144585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/ 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/ 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 265121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 102209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/ 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 133841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 84465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/ 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/ 2023-11-28T18:07:53.501 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pci-stub.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 46465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/ 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 145809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pcmcia.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 62137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pcmcia_core.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 112889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/ 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/broadcom/ 2023-11-28T18:07:53.502 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/samsung/ 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/ 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/ 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/ 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/ 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-11-28T18:07:53.503 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/ 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 59665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 127441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-28T18:07:53.504 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/ibm_rtl.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/ 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 43497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-28T18:07:53.505 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 57569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/ 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/88pm860x_battery.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/88pm860x_charger.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 55057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/bq2415x_charger.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/bq24190_charger.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/bq24735-charger.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/da9030_battery.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/da9052-battery.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ds2760_battery.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ds2780_battery.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ds2781_battery.ko 2023-11-28T18:07:53.506 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ds2782_battery.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/gpio-charger.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/isp1704_charger.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/lp8727_charger.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max14577_charger.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max17040_battery.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max17042_battery.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max8903_charger.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max8925_power.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/pcf50633-charger.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/sbs-battery.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/test_power.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/tps65090-charger.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/wm831x_backup.ko 2023-11-28T18:07:53.507 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/wm831x_power.ko 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/wm8350_power.ko 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/ 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 57729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/ 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/ 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ 2023-11-28T18:07:53.508 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/ 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-88pm80x.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-88pm860x.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-da9052.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-da9055.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-da9063.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 68577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-28T18:07:53.509 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-lp8788.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t86.ko 2023-11-28T18:07:53.510 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max8907.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max8925.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max8997.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max8998.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-palmas.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 46633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf50633.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-28T18:07:53.511 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rc5t583.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-s35390a.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-tps6586x.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-tps65910.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-wm831x.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-wm8350.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ 2023-11-28T18:07:53.512 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 77617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 57665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 55105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 135185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/ 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 272305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 100617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/ 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 356201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 295801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic94xx/ 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 189185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-28T18:07:53.513 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/ 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 106881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/ 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 246985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/ 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1171945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/ 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 237273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/ 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 115369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/ 2023-11-28T18:07:53.514 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 308921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/ 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 76145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 101449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 146689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 75553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/ 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 212425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/ 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-28T18:07:53.515 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 158505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fdomain.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/ 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 263489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 201817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/imm.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 266537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 101881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/ 2023-11-28T18:07:53.516 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 317153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 52961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/ 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 314865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 145585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/ 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 216665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/ 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 2197249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/ 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 73569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-28T18:07:53.517 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 310617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 84721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/ 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 659825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/ 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 119561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/ 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 81625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-11-28T18:07:53.518 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/ 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 364105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 123617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ppa.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 55361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/ 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1876689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 85289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/ 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 559033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qlogicfas408.ko 2023-11-28T18:07:53.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 238761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 173937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 279313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 97729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 73881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 115889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 46825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/ 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 162121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-28T18:07:53.520 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/xen-scsifront.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 129441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/ 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/ 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 736601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/ 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 43025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-28T18:07:53.521 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 930257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 122521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/ 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 75593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/ 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/ 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/ 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-28T18:07:53.522 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ 2023-11-28T18:07:53.523 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 57865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/moxa.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 50953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/mxser.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 108601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/ 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/ 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 59497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/altera_uart.ko 2023-11-28T18:07:53.524 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 72425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/ 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 103073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/rp2.ko 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 52817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/sc16is7xx.ko 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/ 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 531265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-11-28T18:07:53.525 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/ 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_dmem_genirq.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_mf624.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_netx.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/ 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/c67x00/ 2023-11-28T18:07:53.526 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 73297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/c67x00/c67x00.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 144377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/ 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 87089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 89857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/common/ 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/common/ulpi.ko 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc2/ 2023-11-28T18:07:53.527 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc3/ 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 448505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc3/dwc3.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/ 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/ 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/u_ether.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/u_serial.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-11-28T18:07:53.528 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 92489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 122081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 57681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 57729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 85609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 58113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/ 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-11-28T18:07:53.529 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 62185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-11-28T18:07:53.530 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 161225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/libcomposite.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/ 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/bdc/ 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 157681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 52721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 69705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 56033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 76057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/net2272.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 75073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/net2280.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 73169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-11-28T18:07:53.531 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 65809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 81505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 156073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/ 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/bcma-hcd.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 80113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/isp116x-hcd.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 87353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811_cs.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/ssb-hcd.ko 2023-11-28T18:07:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/ 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/cytherm.ko 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ehset.ko 2023-11-28T18:07:53.533 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/lvstest.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/ 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 64841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-28T18:07:53.534 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbtest.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/mon/ 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 72953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/mon/usbmon.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/musb/ 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 301697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/musb/musb_hdrc.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/ 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-11-28T18:07:53.535 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-isp1301.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-tahvo.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 58217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/f81232.ko 2023-11-28T18:07:53.536 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 175137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 137081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 124833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 58233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 107521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-28T18:07:53.537 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/metro-usb.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 78217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 56337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mxuport.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 191137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-28T18:07:53.538 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 51625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 61489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 125609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usbserial.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-28T18:07:53.539 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/wishbone-serial.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 52913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-28T18:07:53.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 179969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/ 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 72177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 107001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/ 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/ 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 154297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-28T18:07:53.541 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 108737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 77609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/ 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 113785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 58089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 89249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/ 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/ 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/aat2870_bl.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/adp5520_bl.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/adp8860_bl.ko 2023-11-28T18:07:53.542 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/adp8870_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/as3711_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/bd6107.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/da903x_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/da9052_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/gpio_backlight.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/kb3886_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lm3533_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lm3639_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lv5207lp.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/max8925_bl.ko 2023-11-28T18:07:53.543 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/pandora_bl.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/wm831x_bl.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/ 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/matrox_w1.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/w1-gpio.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/ 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-28T18:07:53.544 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 56425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 102673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/acquirewdt.ko 2023-11-28T18:07:53.545 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/advantechwdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/cpu5wdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/da9052_wdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/da9055_wdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/da9063_wdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/dw_wdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/eurotechwdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-28T18:07:53.546 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/kempld_wdt.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/mena21_wdt.ko 2023-11-28T18:07:53.547 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pc87413_wdt.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/retu_wdt.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sc1200wdt.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-28T18:07:53.548 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/twl4030_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wafer5823wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wm831x_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wm8350_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-28T18:07:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/ 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/ 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 176377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 71249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-scsiback.ko 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/ 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/ 2023-11-28T18:07:53.550 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 166425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/adfs/ 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 61073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/adfs/adfs.ko 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/ 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 163593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/afs/ 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 932817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/afs/kafs.ko 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/autofs/ 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 101057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/autofs/autofs4.ko 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/ 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 64545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/bfs/ 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/bfs/bfs.ko 2023-11-28T18:07:53.551 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/ 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 3402729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/ 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 306009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1333017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/ 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 100801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/configfs/ 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 108393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/configfs/configfs.ko 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/ 2023-11-28T18:07:53.552 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 29625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/ 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 642513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/efs/ 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/efs/efs.ko 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/ 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1505025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/ 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/freevxfs/ 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/freevxfs/freevxfs.ko 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/ 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 226481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-28T18:07:53.553 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/ 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/ 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1231529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/ 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 141209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/ 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 222817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hpfs/ 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 192329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hpfs/hpfs.ko 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/ 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 74065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-28T18:07:53.554 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/ 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 419497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/ 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 280313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/ 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 89193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/ 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 124649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/ 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/ 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 81449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/ 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 59449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-28T18:07:53.555 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/ 2023-11-28T18:07:53.556 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 117465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-28T18:07:53.556 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1055721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-28T18:07:53.556 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv2.ko 2023-11-28T18:07:53.556 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 120177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-28T18:07:53.556 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 2042137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-28T18:07:53.595 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/ 2023-11-28T18:07:53.595 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-28T18:07:53.595 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-28T18:07:53.595 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/ 2023-11-28T18:07:53.595 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1516873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-28T18:07:53.595 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/ 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 592833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/ 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-28T18:07:53.596 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ascii.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-28T18:07:53.597 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 94137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 142673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 165745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 118121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-28T18:07:53.598 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-28T18:07:53.599 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ntfs/ 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 219417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ntfs/ntfs.ko 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 244409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 440737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 2592433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-28T18:07:53.600 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/omfs/ 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/omfs/omfs.ko 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/ 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 352161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/qnx4/ 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/qnx4/qnx4.ko 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/qnx6/ 2023-11-28T18:07:53.601 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 46689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/qnx6/qnx6.ko 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/quota/ 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/quota/quota_tree.ko 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/quota/quota_v1.ko 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/quota/quota_v2.ko 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/ 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 532673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/ 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/ 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/ 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 2668401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/ 2023-11-28T18:07:53.602 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/ 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 134641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/ 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 90721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/ 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 247737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 162497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/ 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 4011017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/ 2023-11-28T18:07:53.603 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/locking/ 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 75489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/locking/locktorture.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/time/ 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/time/test_udelay.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 6833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 6529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc7.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/ 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/interval_tree_test.ko 2023-11-28T18:07:53.604 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/ 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/ 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 6617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/memory-notifier-error-inject.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/notifier-error-inject.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/percpu_test.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/pm-notifier-error-inject.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/ 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 143433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-28T18:07:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/rbtree_test.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/ 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test-kstrtox.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test-string_helpers.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 4760425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test_bpf.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test_firmware.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 6369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test_module.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test_user_copy.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-28T18:07:53.606 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/xz/ 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/xz/xz_dec_test.ko 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/mm/ 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/ 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 55737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 6985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-28T18:07:53.607 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/ 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/ 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 81737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/ 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 198337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-28T18:07:53.608 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/ 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 649297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/ 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 63809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 777689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-28T18:07:53.609 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-28T18:07:53.610 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 55929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/ 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1046833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/ 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 125993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/ 2023-11-28T18:07:53.611 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 194209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 57881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/ 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 390473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/hsr/ 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 99729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/hsr/hsr.ko 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ 2023-11-28T18:07:53.612 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 45233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 254625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 48337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 48657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 61225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-28T18:07:53.613 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ip_tables.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-28T18:07:53.614 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-28T18:07:53.615 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-28T18:07:53.616 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 41665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 80897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_gre.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 93577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-28T18:07:53.617 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/mip6.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-28T18:07:53.618 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-28T18:07:53.619 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 74073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/ 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 82833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/ 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 125281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-28T18:07:53.620 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 25953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 53241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/lapb/ 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 47977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/lapb/lapb.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/ 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 267913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc2.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/ 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 290417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-28T18:07:53.621 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/ 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 119585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 100297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 101441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 102817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 103417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 112849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-28T18:07:53.622 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 60329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 109593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 113129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 112585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 111585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 114465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 411729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-28T18:07:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 408257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-28T18:07:53.624 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 118145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 107233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 89425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-28T18:07:53.625 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 37129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 95353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 44065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 615649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 42561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-28T18:07:53.626 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 39785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 40721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-28T18:07:53.627 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_osf.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_synproxy.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tproxy.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 89313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/x_tables.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-28T18:07:53.628 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 18169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-28T18:07:53.629 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 14337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-28T18:07:53.630 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-28T18:07:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 43289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-28T18:07:53.632 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-28T18:07:53.633 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 16089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 8553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpudp.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/ 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/ 2023-11-28T18:07:53.634 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 12425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/ 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 380033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 11793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/ 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/phonet/ 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 110201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/phonet/phonet.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/phonet/pn_pep.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/ 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 251481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-28T18:07:53.635 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 164937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 67377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rxrpc/ 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 819657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rxrpc/rxrpc.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/ 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 43449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-28T18:07:53.636 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_cgroup.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 38425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 61409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 13409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-28T18:07:53.637 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 21649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 34633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 33609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq_codel.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 35433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 49817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 66393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 30273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-28T18:07:53.638 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 36137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 27497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 24729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 48729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 32905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 26881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 31353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 28105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/ 2023-11-28T18:07:53.639 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 929745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 19033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/ 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/ 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 336529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 54737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 1520097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/ 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 707657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/ 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 862665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/ 2023-11-28T18:07:53.640 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/ 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 85409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 68049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 96193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 10705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 17521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/x25/ 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 125177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/x25/x25.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/ 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 22657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_algo.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 20649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 102257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_user.ko 2023-11-28T18:07:53.641 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/virt/ 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/ 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 15537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 640081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 624107 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 7663 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 23428 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 9821 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 70004 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 146313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 232548 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 210 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 72606 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-28T18:07:53.642 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 879 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-28T18:07:53.643 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 254438 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-28T18:07:53.643 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 309363 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-28T18:07:53.643 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./usr/ 2023-11-28T18:07:53.643 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./usr/share/ 2023-11-28T18:07:53.643 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./usr/share/doc/ 2023-11-28T18:07:53.643 INFO:teuthology.orchestra.run.smithi193.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./usr/share/doc/linux-image-6.6.0-g8247bbb5b470/ 2023-11-28T18:07:53.643 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 172 2023-11-23 03:47 ./usr/share/doc/linux-image-6.6.0-g8247bbb5b470/changelog.Debian.gz 2023-11-28T18:07:53.643 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r-- root/root 691 2023-11-23 03:47 ./usr/share/doc/linux-image-6.6.0-g8247bbb5b470/copyright 2023-11-28T18:07:53.644 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-28T18:07:53.644 INFO:teuthology.task.kernel:searching for kernel 6.6.0-g8247bbb5b470 2023-11-28T18:07:53.644 DEBUG:teuthology.orchestra.run.smithi193:> egrep '(submenu|menuentry.*6.6.0-g8247bbb5b470).*{' /boot/grub/grub.cfg 2023-11-28T18:07:53.651 INFO:teuthology.orchestra.run.smithi193.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-11-28T18:07:53.651 INFO:teuthology.orchestra.run.smithi193.stdout: menuentry 'Ubuntu, with Linux 6.6.0-g8247bbb5b470' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-g8247bbb5b470-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-11-28T18:07:53.651 INFO:teuthology.orchestra.run.smithi193.stdout: menuentry 'Ubuntu, with Linux 6.6.0-g8247bbb5b470 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-g8247bbb5b470-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-11-28T18:07:53.652 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-11-28T18:07:53.652 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-g8247bbb5b470 2023-11-28T18:07:53.652 DEBUG:teuthology.orchestra.run.smithi193:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-11-28T18:07:53.693 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi193.front.sepia.ceph.com to complete... 2023-11-28T18:07:54.361 INFO:teuthology.orchestra.run.smithi193.stderr:Sourcing file `/etc/default/grub' 2023-11-28T18:07:54.362 INFO:teuthology.orchestra.run.smithi193.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-28T18:07:54.382 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2023-11-28T18:07:54.764 INFO:teuthology.orchestra.run.smithi193.stderr:Found linux image: /boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T18:07:54.771 INFO:teuthology.orchestra.run.smithi193.stderr:Found initrd image: /boot/initrd.img-6.6.0-g8247bbb5b470 2023-11-28T18:07:55.775 INFO:teuthology.orchestra.run.smithi193.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-28T18:07:55.780 INFO:teuthology.orchestra.run.smithi193.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-28T18:07:55.820 INFO:teuthology.orchestra.run.smithi193.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-28T18:07:55.825 INFO:teuthology.orchestra.run.smithi193.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-28T18:07:56.295 INFO:teuthology.orchestra.run.smithi193.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-28T18:07:56.295 INFO:teuthology.orchestra.run.smithi193.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-28T18:07:56.295 INFO:teuthology.orchestra.run.smithi193.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-28T18:07:56.318 INFO:teuthology.orchestra.run.smithi193.stderr:done 2023-11-28T18:08:01.634 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2023-11-28T18:08:01.651 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-28T18:08:01.676 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2023-11-28T18:08:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T18:08:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-6.6.0-g8247bbb5b470 2023-11-28T18:08:02.345 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-28T18:08:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-28T18:08:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-28T18:08:02.430 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-28T18:08:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-28T18:08:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-28T18:08:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-28T18:08:02.636 INFO:teuthology.orchestra.run.smithi007.stderr:done 2023-11-28T18:08:02.839 DEBUG:teuthology.orchestra.run.smithi007:> dpkg-deb -c /tmp/linux-image.deb 2023-11-28T18:08:02.846 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./ 2023-11-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./boot/ 2023-11-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 4415617 2023-11-23 03:47 ./boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T18:08:02.860 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 169214 2023-11-23 03:47 ./boot/config-6.6.0-g8247bbb5b470 2023-11-28T18:08:02.860 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11352832 2023-11-23 03:47 ./boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T18:08:02.883 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/ 2023-11-28T18:08:02.883 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/ 2023-11-28T18:08:02.883 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/postinst.d/ 2023-11-28T18:08:02.883 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/postrm.d/ 2023-11-28T18:08:02.883 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/preinst.d/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./etc/kernel/prerm.d/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 490233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/aesni-intel.ko 2023-11-28T18:08:02.884 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 93113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 106729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 152241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 83889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-28T18:08:02.885 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 58585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/ 2023-11-28T18:08:02.886 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/ 2023-11-28T18:08:02.887 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/ 2023-11-28T18:08:02.887 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-28T18:08:02.887 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpuid.ko 2023-11-28T18:08:02.887 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/msr.ko 2023-11-28T18:08:02.887 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/ 2023-11-28T18:08:02.887 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 354433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-28T18:08:02.887 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 881625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-28T18:08:02.888 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 2275865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-28T18:08:02.948 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/af_alg.ko 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/algif_hash.ko 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/algif_skcipher.ko 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/asymmetric_keys/ 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/ 2023-11-28T18:08:02.949 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-28T18:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cryptd.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_simd.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-28T18:08:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lrw.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-28T18:08:02.952 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 149193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xts.ko 2023-11-28T18:08:02.953 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_extlog.ko 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/apei/ 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/apei/einj.ko 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-28T18:08:02.954 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 103025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 88865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 88289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-28T18:08:02.955 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-28T18:08:02.956 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-28T18:08:02.957 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-28T18:08:02.957 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_legacy.ko 2023-11-28T18:08:02.957 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-28T18:08:02.957 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-28T18:08:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_radisys.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rz1000.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-28T18:08:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 83345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-28T18:08:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-28T18:08:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-28T18:08:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-28T18:08:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ 2023-11-28T18:08:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-28T18:08:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/ 2023-11-28T18:08:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 124473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-28T18:08:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/ 2023-11-28T18:08:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/ 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 113665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/ 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 965585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 173225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/ 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 94297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 115793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/ 2023-11-28T18:08:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 110385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 98193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 263769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/ 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 98145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/ 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/applicom.ko 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/ 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-28T18:08:02.963 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 148601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 139313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/ 2023-11-28T18:08:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 84097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-28T18:08:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/nvram.ko 2023-11-28T18:08:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-28T18:08:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-28T18:08:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/ 2023-11-28T18:08:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-28T18:08:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-11-28T18:08:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-11-28T18:08:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-11-28T18:08:02.966 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-28T18:08:02.966 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-28T18:08:02.966 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-11-28T18:08:02.966 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/xillybus/ 2023-11-28T18:08:02.967 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/xillybus/xillybus_class.ko 2023-11-28T18:08:02.967 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/xillybus/xillybus_core.ko 2023-11-28T18:08:02.967 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-11-28T18:08:02.967 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/ 2023-11-28T18:08:02.967 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/clk-palmas.ko 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/clk-si5351.ko 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/clk-twl6040.ko 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/clk/clk-wm831x.ko 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/ 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-28T18:08:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 243025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/ 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/ 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/ 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 438761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/ 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/ 2023-11-28T18:08:02.969 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-28T18:08:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ 2023-11-28T18:08:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/ 2023-11-28T18:08:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-28T18:08:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ 2023-11-28T18:08:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 122569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-28T18:08:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ 2023-11-28T18:08:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 79953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-28T18:08:02.971 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 63617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 52041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-28T18:08:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 56825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/ 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-gpio.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-max14577.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-max77693.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-max8997.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-palmas.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-rt8973a.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/extcon/extcon-sm5502.ko 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/ 2023-11-28T18:08:02.973 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/dmi-sysfs.ko 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/efi/ 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/efi/efi-pstore.ko 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/ 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-adp5520.ko 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-amd8111.ko 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-arizona.ko 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-da9052.ko 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-da9055.ko 2023-11-28T18:08:02.974 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-dln2.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 77849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-f7188x.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-generic.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-kempld.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-lp3943.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-max7300.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-max730x.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-max732x.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-pca953x.ko 2023-11-28T18:08:02.975 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-pcf857x.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-rdc321x.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-sch.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-sch311x.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-tps65912.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-twl4030.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-twl6040.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-vx855.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-wm831x.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-wm8350.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-wm8994.ko 2023-11-28T18:08:02.976 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/ 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-a4tech.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-apple.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-belkin.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-cherry.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-chicony.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-cp2112.ko 2023-11-28T18:08:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-cypress.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ezkey.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-generic.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-28T18:08:02.978 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kensington.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lg-g15.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 58185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-28T18:08:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 101481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 80993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-magicmouse.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-microsoft.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-monterey.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ntrig.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-28T18:08:02.980 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-28T18:08:02.981 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 72937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-28T18:08:02.981 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-28T18:08:02.981 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-28T18:08:02.981 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-28T18:08:02.981 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-28T18:08:02.981 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-28T18:08:02.981 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-28T18:08:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-28T18:08:02.983 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 71073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 143473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 267825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/usbhid/ 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 125721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/usbhid/usbhid.ko 2023-11-28T18:08:02.984 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/usbhid/usbkbd.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/usbhid/usbmouse.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 217873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/ 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 64201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 89625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 305465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 61393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-28T18:08:02.985 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 97977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 45553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 74433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-28T18:08:02.986 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 55201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 58617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/da9052-hwmon.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/da9055-hwmon.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 99601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-28T18:08:02.987 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc2103.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 94457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 55449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-28T18:08:02.988 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 46801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 45529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hih6130.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 130449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/jc42.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-28T18:08:02.989 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 48601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 72833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 46153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-28T18:08:02.990 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 108841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-28T18:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-28T18:08:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mc13783-adc.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 119817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 59737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 82337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-28T18:08:02.993 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-28T18:08:02.994 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 82577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-28T18:08:02.995 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-28T18:08:02.996 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 52457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 57449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 71345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 63225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 71321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 101473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 119505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-28T18:08:02.997 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/ 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/ 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/ 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-11-28T18:08:02.998 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 6929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-11-28T18:08:02.999 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 62817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-28T18:08:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-28T18:08:03.001 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/muxes/ 2023-11-28T18:08:03.002 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-11-28T18:08:03.002 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-11-28T18:08:03.002 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-11-28T18:08:03.002 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ 2023-11-28T18:08:03.002 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ 2023-11-28T18:08:03.002 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 273609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 879969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 69273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 324217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 97985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 282273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/ 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/ 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 493073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/ 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 419737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-28T18:08:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/ 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 739025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 296625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/ocrdma/ 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 153697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/usnic/ 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 168969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 263081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ 2023-11-28T18:08:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 100241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 111425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 128753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 129529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/ 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/evbug.ko 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/ff-memless.ko 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/ 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-28T18:08:03.005 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/ 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/lkkbd.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/lm8323.ko 2023-11-28T18:08:03.006 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/lm8333.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/newtonkbd.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/qt1070.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/qt2160.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/stowaway.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/sunkbd.ko 2023-11-28T18:08:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/xtkbd.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/matrix-keymap.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/ 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 123681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 89449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/gpio_mouse.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 335929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/psmouse.ko 2023-11-28T18:08:03.008 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/ 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 109873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/ 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/ct82c710.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/parkbd.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/pcips2.ko 2023-11-28T18:08:03.009 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/ps2mult.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serport.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/ 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/ 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/ 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-88pm860x.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-adp5520.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 59753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-bd2802.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-28T18:08:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-da903x.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-da9052.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-gpio.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3533.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm355x.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3642.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp8788.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-max8997.ko 2023-11-28T18:08:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-mc13783.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-menf21bmc.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-pca9532.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-pca955x.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-pca963x.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-tca6507.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-wm831x-status.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-wm8350.ko 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ 2023-11-28T18:08:03.012 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/ 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/ 2023-11-28T18:08:03.013 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 654969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 89601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bufio.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 156009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 106897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-era.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-28T18:08:03.014 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-28T18:08:03.015 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log.ko 2023-11-28T18:08:03.015 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-mirror.ko 2023-11-28T18:08:03.015 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-28T18:08:03.015 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-28T18:08:03.015 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-region-hash.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 109529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-snapshot.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 181865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-zero.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-28T18:08:03.016 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-28T18:08:03.017 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/ 2023-11-28T18:08:03.017 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 199937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-28T18:08:03.017 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-28T18:08:03.017 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 113073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-28T18:08:03.017 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 158681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-28T18:08:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 395721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-28T18:08:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/ 2023-11-28T18:08:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/ 2023-11-28T18:08:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 178137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-28T18:08:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 63385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-28T18:08:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-28T18:08:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-28T18:08:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 123321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 79041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/ 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/88pm800.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/88pm805.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/88pm80x.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/arizona-i2c.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 123721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/arizona.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/bcm590xx.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/cros_ec_dev.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/dln2.ko 2023-11-28T18:08:03.093 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/janz-cmodio.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/kempld-core.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lm3533-core.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lp3943.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/max8907.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mc13xxx-core.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/menf21bmc.ko 2023-11-28T18:08:03.094 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/pcf50633-adc.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/pcf50633-gpio.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/pcf50633.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/retu-mfd.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 80425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/si476x-core.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 48457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/tps65010.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/tps6507x.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-28T18:08:03.095 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wm8994.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 48489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ad525x_dpot.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/ 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 46585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 45593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/c2port/ 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-11-28T18:08:03.096 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/c2port/core.ko 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/ 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ds1682.ko 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/dummy-irq.ko 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/echo/ 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/echo/echo.ko 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/ 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-28T18:08:03.097 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/genwqe/ 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 152201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hmc6352.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ibmasm/ 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 72129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ics932s401.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-28T18:08:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/ 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/ 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 125209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 56521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 310113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/phantom.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ti-st/ 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 74265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/ti-st/st_drv.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-28T18:08:03.099 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 48697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/ 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 183609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/ 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 490129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/ 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/ 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 106033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-28T18:08:03.100 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/ 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 453737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/ 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 45785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 130553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 68849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ 2023-11-28T18:08:03.101 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 59689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 99049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/ 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/ 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 69137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/ 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 59585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/ 2023-11-28T18:08:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-28T18:08:03.103 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/ 2023-11-28T18:08:03.103 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-28T18:08:03.103 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-28T18:08:03.103 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 87297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/ 2023-11-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/ 2023-11-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 97337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-11-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 104673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 90257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/ 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 81569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 69729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/ 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 176761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-11-28T18:08:03.106 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1421497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 128569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/ 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 132089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 312081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/ 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/ 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 323513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/ 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 193497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-28T18:08:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 347417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1044497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 166193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/ 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/ 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 204345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/ 2023-11-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/ 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 69449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 61489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 141225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 62817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/ 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 62833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 45721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ec_bhf.ko 2023-11-28T18:08:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/ 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/ 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 281137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fujitsu/ 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 45345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 90465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/ 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 295577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/ 2023-11-28T18:08:03.110 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 584497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/ 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 246209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/ 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 925825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/ 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 302937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/ 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 514641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/ 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 93593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-11-28T18:08:03.111 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 734305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 151769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 86921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/ 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 102289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 122649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/ 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 727601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-28T18:08:03.112 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 278937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1113681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 95665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/ 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 108649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-28T18:08:03.113 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 73441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/ 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 150905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/ 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 136721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/ 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 43145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/ 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-11-28T18:08:03.114 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 227897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-28T18:08:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 87081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-28T18:08:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-11-28T18:08:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 672049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-28T18:08:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/ 2023-11-28T18:08:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 43017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-28T18:08:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/ 2023-11-28T18:08:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-28T18:08:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 83985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 189801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/samsung/ 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 104241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/ 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1052497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/ 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/ 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-28T18:08:03.116 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 63337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/ 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 61537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 73409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 68177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/ 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 487025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-28T18:08:03.117 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/ 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 103337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 166097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/ 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/ 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 86529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/ 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 80273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-28T18:08:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 83633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/ 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 46241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/ 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 59993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/fddi/ 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/fddi/defxx.ko 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/fddi/skfp/ 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 216985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/fddi/skfp/skfp.ko 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 77345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/ 2023-11-28T18:08:03.119 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 235521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/ 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 83769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 62145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/ 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-gpio.ko 2023-11-28T18:08:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 43953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/ 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-28T18:08:03.121 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 82049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 76113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-28T18:08:03.122 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 97857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/plip/ 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 43505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/plip/plip.ko 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ 2023-11-28T18:08:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sb1000.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/ 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-28T18:08:03.124 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 64449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/ 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 106169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-28T18:08:03.125 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 87017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/ 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 144297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/ 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 297329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/ 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 144585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/ 2023-11-28T18:08:03.126 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/ 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 265121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 102209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/ 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 133841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 84465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/ 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/ 2023-11-28T18:08:03.127 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pci-stub.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 46465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/ 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 145809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pcmcia.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 62137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pcmcia_core.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-28T18:08:03.128 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 112889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/ 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/broadcom/ 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/samsung/ 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/ 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/ 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/ 2023-11-28T18:08:03.129 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/ 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec.ko 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/ 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-28T18:08:03.130 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 59665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 127441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-28T18:08:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/ibm_rtl.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/ 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 43497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-28T18:08:03.132 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 57569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/ 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/88pm860x_battery.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/88pm860x_charger.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 55057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/bq2415x_charger.ko 2023-11-28T18:08:03.133 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/bq24190_charger.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/bq24735-charger.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/da9030_battery.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/da9052-battery.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ds2760_battery.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ds2780_battery.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ds2781_battery.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/ds2782_battery.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/gpio-charger.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/isp1704_charger.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/lp8727_charger.ko 2023-11-28T18:08:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max14577_charger.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max17040_battery.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max17042_battery.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max8903_charger.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/max8925_power.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/pcf50633-charger.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/sbs-battery.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/test_power.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/tps65090-charger.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/wm831x_backup.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/wm831x_power.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/power/supply/wm8350_power.ko 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/ 2023-11-28T18:08:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 57729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/ 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/ 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/ 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-88pm80x.ko 2023-11-28T18:08:03.136 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-88pm860x.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-da9052.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-da9055.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-da9063.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 68577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-28T18:08:03.137 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-lp8788.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t86.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max8907.ko 2023-11-28T18:08:03.138 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max8925.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max8997.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max8998.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-palmas.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 46633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf50633.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rc5t583.ko 2023-11-28T18:08:03.139 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-s35390a.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-tps6586x.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-tps65910.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-wm831x.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-wm8350.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ 2023-11-28T18:08:03.140 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 77617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 57665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 55105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 135185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/ 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 272305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 100617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/ 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 356201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 295801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic94xx/ 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 189185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-11-28T18:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/ 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 106881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/ 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 246985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/ 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1171945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/ 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 237273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/ 2023-11-28T18:08:03.142 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 115369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/ 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 308921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/ 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 76145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 101449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 146689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 75553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-28T18:08:03.143 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/ 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 212425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/ 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 158505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fdomain.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/ 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 263489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 201817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/imm.ko 2023-11-28T18:08:03.144 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 266537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 101881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/ 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 317153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 52961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/ 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 314865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 145585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/ 2023-11-28T18:08:03.145 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 216665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/ 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 2197249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/ 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 73569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 310617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 84721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/ 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 659825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/ 2023-11-28T18:08:03.146 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 119561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/ 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 81625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/ 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 364105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 123617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ppa.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 55361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-28T18:08:03.147 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/ 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1876689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 85289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/ 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 559033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qlogicfas408.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 238761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 173937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 279313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 97729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 73881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-28T18:08:03.148 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 115889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 46825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/ 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 162121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/xen-scsifront.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 129441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-28T18:08:03.149 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/ 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/ 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 736601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/ 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 43025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 930257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 122521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/ 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 75593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/ 2023-11-28T18:08:03.150 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/ 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/ 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-28T18:08:03.151 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 57865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/moxa.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 50953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/mxser.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 108601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-28T18:08:03.152 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/ 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/ 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 59497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/altera_uart.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 72425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/ 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 103073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/rp2.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 52817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/sc16is7xx.ko 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/ 2023-11-28T18:08:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 531265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/ 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_dmem_genirq.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_mf624.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_netx.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-28T18:08:03.154 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/ 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/c67x00/ 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 73297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/c67x00/c67x00.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 144377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/ 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 87089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-28T18:08:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 89857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/common/ 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/common/ulpi.ko 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc2/ 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc3/ 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 448505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/dwc3/dwc3.ko 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/ 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/ 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/u_ether.ko 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/u_serial.ko 2023-11-28T18:08:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 92489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 122081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 57681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 57729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 85609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-11-28T18:08:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 58113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/ 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-11-28T18:08:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 62185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 161225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/libcomposite.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/ 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/bdc/ 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 157681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 52721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 69705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 56033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 76057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/net2272.ko 2023-11-28T18:08:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 75073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/net2280.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 73169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 65809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 81505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 156073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/ 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/bcma-hcd.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 80113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/isp116x-hcd.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 87353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811_cs.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/ssb-hcd.ko 2023-11-28T18:08:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-11-28T18:08:03.161 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/ 2023-11-28T18:08:03.161 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-28T18:08:03.161 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-28T18:08:03.161 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ 2023-11-28T18:08:03.161 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/cytherm.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ehset.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-28T18:08:03.162 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/lvstest.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/ 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 64841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbtest.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/mon/ 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 72953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/mon/usbmon.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/musb/ 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 301697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/musb/musb_hdrc.ko 2023-11-28T18:08:03.163 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/ 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-isp1301.ko 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-tahvo.ko 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-28T18:08:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 58217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/f81232.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 175137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 137081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 124833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-28T18:08:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 58233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 107521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/metro-usb.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 78217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 56337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mxuport.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-28T18:08:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 191137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 51625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 61489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-28T18:08:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 125609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usbserial.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/wishbone-serial.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 52913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-28T18:08:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 179969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/ 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 72177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 107001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-28T18:08:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/ 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/ 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 154297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 108737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 77609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/ 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 113785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 58089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 89249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/ 2023-11-28T18:08:03.170 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/ 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/aat2870_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/adp5520_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/adp8860_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/adp8870_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/as3711_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/bd6107.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/da903x_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/da9052_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/gpio_backlight.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/kb3886_bl.ko 2023-11-28T18:08:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lm3533_bl.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lm3639_bl.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lv5207lp.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/max8925_bl.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/pandora_bl.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/wm831x_bl.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/ 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-28T18:08:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/matrox_w1.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/w1-gpio.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/ 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14153 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-28T18:08:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 56425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 102673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/acquirewdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/advantechwdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/cpu5wdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/da9052_wdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/da9055_wdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/da9063_wdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/dw_wdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/eurotechwdt.ko 2023-11-28T18:08:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-28T18:08:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/kempld_wdt.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/mena21_wdt.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22265 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pc87413_wdt.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/retu_wdt.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-11-28T18:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sc1200wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/twl4030_wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wafer5823wdt.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-28T18:08:03.177 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wm831x_wdt.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wm8350_wdt.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/ 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/ 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 176377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 71249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-scsiback.ko 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/ 2023-11-28T18:08:03.178 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/ 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 166425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/adfs/ 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 61073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/adfs/adfs.ko 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/ 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 163593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/afs/ 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 932817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/afs/kafs.ko 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/autofs/ 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 101057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/autofs/autofs4.ko 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/ 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 64545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-28T18:08:03.179 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/bfs/ 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/bfs/bfs.ko 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/ 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 3402729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/ 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 306009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1333017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/ 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 100801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/configfs/ 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 108393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/configfs/configfs.ko 2023-11-28T18:08:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/ 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 29625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/ 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 642513 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/efs/ 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/efs/efs.ko 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/ 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1505025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/ 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/freevxfs/ 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/freevxfs/freevxfs.ko 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/ 2023-11-28T18:08:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 226481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-28T18:08:03.182 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/ 2023-11-28T18:08:03.182 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-28T18:08:03.182 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/ 2023-11-28T18:08:03.182 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1231529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-28T18:08:03.221 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/ 2023-11-28T18:08:03.221 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 141209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-28T18:08:03.221 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/ 2023-11-28T18:08:03.221 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 222817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hpfs/ 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 192329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hpfs/hpfs.ko 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/ 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 74065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/ 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 419497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/ 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 280313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/ 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 89193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/ 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 124649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-28T18:08:03.222 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/ 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/ 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 81449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/ 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 59449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/ 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 117465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1055721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv2.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 120177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 2042137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/ 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-28T18:08:03.223 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/ 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1516873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/ 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 592833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/ 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-28T18:08:03.224 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ascii.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-28T18:08:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 94137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 142673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 165745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 118121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-28T18:08:03.226 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-28T18:08:03.227 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ntfs/ 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 219417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ntfs/ntfs.ko 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 244409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 440737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 2592433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-28T18:08:03.228 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/omfs/ 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/omfs/omfs.ko 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/ 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 352161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/qnx4/ 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/qnx4/qnx4.ko 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/qnx6/ 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 46689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/qnx6/qnx6.ko 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/quota/ 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/quota/quota_tree.ko 2023-11-28T18:08:03.229 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/quota/quota_v1.ko 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/quota/quota_v2.ko 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/ 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 532673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/ 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/ 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/ 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 2668401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/ 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/ 2023-11-28T18:08:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 134641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/ 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 90721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/ 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 247737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 162497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/ 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 4011017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/ 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/locking/ 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 75489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/locking/locktorture.ko 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/time/ 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/time/test_udelay.ko 2023-11-28T18:08:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 6833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 6529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc7.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/ 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/interval_tree_test.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/ 2023-11-28T18:08:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/ 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 6617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/memory-notifier-error-inject.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/notifier-error-inject.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/percpu_test.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/pm-notifier-error-inject.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/ 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 143433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/rbtree_test.ko 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/ 2023-11-28T18:08:03.233 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test-kstrtox.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test-string_helpers.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 4760425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test_bpf.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test_firmware.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 6369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test_module.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/test_user_copy.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/xz/ 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/lib/xz/xz_dec_test.ko 2023-11-28T18:08:03.234 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/mm/ 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/ 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 55737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 6985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-28T18:08:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/ 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/ 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 81737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/ 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 198337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26345 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-28T18:08:03.236 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/ 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 649297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/ 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 63809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 777689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-28T18:08:03.237 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13217 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8385 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-28T18:08:03.238 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 55929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/ 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1046833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/ 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 125993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/ 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 194209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-28T18:08:03.239 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 57881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/ 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 390473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/hsr/ 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 99729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/hsr/hsr.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 45233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-28T18:08:03.240 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 254625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 48337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 48657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 61225 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-28T18:08:03.241 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ip_tables.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11305 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-11-28T18:08:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18161 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-28T18:08:03.243 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-28T18:08:03.244 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 41665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 80897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_gre.ko 2023-11-28T18:08:03.245 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 93577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19017 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/mip6.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-28T18:08:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11121 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-28T18:08:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25945 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9009 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 74073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/ 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 82833 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/ 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 125281 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-28T18:08:03.248 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 25953 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36713 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 53241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/lapb/ 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 47977 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/lapb/lapb.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/ 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 267913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc2.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/ 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 290417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/ 2023-11-28T18:08:03.249 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 119585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26913 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 100297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 101441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 102817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 103417 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 112849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 60329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-28T18:08:03.250 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 109593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 113129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 112585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 111585 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 114465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 411729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-28T18:08:03.251 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7689 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10681 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7697 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10993 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 408257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33761 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-28T18:08:03.252 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 118145 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 107233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66641 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10113 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 89425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9593 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 37129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 95353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-28T18:08:03.253 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11857 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28929 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 44065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 615649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 42561 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30233 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30569 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 39785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10457 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-28T18:08:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 40721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9601 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14185 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13673 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11841 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_osf.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-28T18:08:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_synproxy.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tproxy.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 89313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/x_tables.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 18169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-28T18:08:03.256 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13369 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13889 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11489 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11753 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-28T18:08:03.257 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 14337 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17361 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8321 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10465 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-28T18:08:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13129 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8801 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 43289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7865 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11921 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-28T18:08:03.259 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11785 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16041 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8001 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10209 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7545 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11825 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-28T18:08:03.260 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11985 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13329 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 16089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9505 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9473 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8809 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 8553 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15777 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpudp.ko 2023-11-28T18:08:03.261 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/ 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13969 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/ 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 12425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/ 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 380033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 11793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/ 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13057 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-28T18:08:03.262 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/phonet/ 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 110201 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/phonet/phonet.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54721 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/phonet/pn_pep.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/ 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 251481 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 164937 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 67377 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rxrpc/ 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 819657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/rxrpc/rxrpc.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/ 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32241 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23577 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26873 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-28T18:08:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 43449 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35297 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33961 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19169 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_cgroup.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31401 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31769 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 38425 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 61409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7441 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 13409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-28T18:08:03.264 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33289 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7249 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 21649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20849 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 34633 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 33609 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31617 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq_codel.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 35433 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 49817 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22089 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 66393 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-28T18:08:03.265 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26073 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 30273 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10793 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26065 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 36137 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 27497 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10625 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 24729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 48729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 32905 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 26881 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 31353 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-28T18:08:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28897 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 28105 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/ 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 929745 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 19033 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/ 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/ 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 336529 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 54737 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 1520097 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/ 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 707657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/ 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10025 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-28T18:08:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 862665 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/ 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15729 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/ 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 85409 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 68049 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 96193 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 10705 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 17521 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/x25/ 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 125177 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/x25/x25.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/ 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 22657 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_algo.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 20649 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-28T18:08:03.268 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 102257 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_user.ko 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/virt/ 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/ 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 15537 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 640081 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 624107 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 7663 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 23428 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 9821 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 70004 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 146313 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 232548 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 210 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 72606 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-28T18:08:03.269 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 879 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-28T18:08:03.270 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 254438 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-28T18:08:03.270 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 309363 2023-11-23 03:47 ./lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-28T18:08:03.270 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./usr/ 2023-11-28T18:08:03.270 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./usr/share/ 2023-11-28T18:08:03.270 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./usr/share/doc/ 2023-11-28T18:08:03.270 INFO:teuthology.orchestra.run.smithi007.stdout:drwxr-xr-x root/root 0 2023-11-23 03:47 ./usr/share/doc/linux-image-6.6.0-g8247bbb5b470/ 2023-11-28T18:08:03.270 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 172 2023-11-23 03:47 ./usr/share/doc/linux-image-6.6.0-g8247bbb5b470/changelog.Debian.gz 2023-11-28T18:08:03.270 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r-- root/root 691 2023-11-23 03:47 ./usr/share/doc/linux-image-6.6.0-g8247bbb5b470/copyright 2023-11-28T18:08:03.271 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-28T18:08:03.271 INFO:teuthology.task.kernel:searching for kernel 6.6.0-g8247bbb5b470 2023-11-28T18:08:03.271 DEBUG:teuthology.orchestra.run.smithi007:> egrep '(submenu|menuentry.*6.6.0-g8247bbb5b470).*{' /boot/grub/grub.cfg 2023-11-28T18:08:03.278 INFO:teuthology.orchestra.run.smithi007.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-11-28T18:08:03.279 INFO:teuthology.orchestra.run.smithi007.stdout: menuentry 'Ubuntu, with Linux 6.6.0-g8247bbb5b470' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-g8247bbb5b470-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-11-28T18:08:03.279 INFO:teuthology.orchestra.run.smithi007.stdout: menuentry 'Ubuntu, with Linux 6.6.0-g8247bbb5b470 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-g8247bbb5b470-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-11-28T18:08:03.279 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-11-28T18:08:03.280 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-g8247bbb5b470 2023-11-28T18:08:03.280 DEBUG:teuthology.orchestra.run.smithi007:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-11-28T18:08:03.323 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi007.front.sepia.ceph.com to complete... 2023-11-28T18:08:03.766 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2023-11-28T18:08:03.768 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-28T18:08:03.790 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2023-11-28T18:08:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T18:08:03.990 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-6.6.0-g8247bbb5b470 2023-11-28T18:08:04.265 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-28T18:08:04.271 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-28T18:08:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-28T18:08:04.323 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-28T18:08:04.561 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-28T18:08:04.561 INFO:teuthology.orchestra.run.smithi007.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-28T18:08:04.561 INFO:teuthology.orchestra.run.smithi007.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-28T18:08:04.582 INFO:teuthology.orchestra.run.smithi007.stderr:done 2023-11-28T18:08:27.485 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-28T18:08:27.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-28T18:08:27.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:08:35.770 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-28T18:08:35.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-28T18:08:35.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:09:02.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-11-28T18:09:10.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2023-11-28T18:09:11.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-28T18:09:11.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:09:19.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-28T18:09:19.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:09:29.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-11-28T18:09:37.634 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2023-11-28T18:09:41.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-28T18:09:41.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:09:41.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-11-28T18:09:49.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-28T18:09:49.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:09:52.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2023-11-28T18:09:56.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-28T18:09:56.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:09:56.748 DEBUG:teuthology.orchestra.run.smithi193:> true 2023-11-28T18:09:57.423 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-28T18:09:57.423 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.6.0-g8247bbb5b470"... 2023-11-28T18:09:57.423 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2023-11-28T18:09:57.472 INFO:teuthology.orchestra.run.smithi193.stdout:6.6.0-g8247bbb5b470 2023-11-28T18:09:57.472 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-28T18:09:57.472 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T18:09:57.473 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-28T18:09:58.473 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-28T18:09:58.473 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T18:09:58.537 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2023-11-28T18:09:58.608 DEBUG:teuthology.parallel:result is None 2023-11-28T18:10:07.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-28T18:10:07.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:10:08.011 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-11-28T18:10:08.725 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-28T18:10:08.725 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.6.0-g8247bbb5b470"... 2023-11-28T18:10:08.726 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-11-28T18:10:08.776 INFO:teuthology.orchestra.run.smithi007.stdout:6.6.0-g8247bbb5b470 2023-11-28T18:10:08.776 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-28T18:10:08.776 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T18:10:08.776 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-28T18:10:09.778 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-28T18:10:09.778 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T18:10:09.876 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2023-11-28T18:10:09.935 DEBUG:teuthology.parallel:result is None 2023-11-28T18:10:09.935 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-28T18:10:09.943 INFO:teuthology.task.internal:Creating test directory... 2023-11-28T18:10:09.944 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T18:10:09.948 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T18:10:09.959 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-28T18:10:09.982 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-28T18:10:09.999 INFO:teuthology.task.internal:Creating archive directory... 2023-11-28T18:10:09.999 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T18:10:10.003 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T18:10:10.093 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-28T18:10:10.102 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-28T18:10:10.102 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T18:10:10.106 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T18:10:10.132 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T18:10:10.145 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T18:10:10.146 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T18:10:10.160 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T18:10:10.162 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-28T18:10:10.170 INFO:teuthology.task.internal:Configuring sudo... 2023-11-28T18:10:10.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T18:10:10.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T18:10:10.225 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-28T18:10:10.235 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-28T18:10:10.235 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T18:10:10.263 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T18:10:10.278 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T18:10:10.325 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T18:10:10.379 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:10:10.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T18:10:10.456 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T18:10:10.465 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T18:10:10.517 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:10:10.517 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T18:10:10.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2023-11-28T18:10:10.592 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2023-11-28T18:10:10.710 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-28T18:10:10.718 INFO:teuthology.task.internal:Starting timer... 2023-11-28T18:10:10.718 INFO:teuthology.run_tasks:Running task pcp... 2023-11-28T18:10:10.753 INFO:teuthology.run_tasks:Running task selinux... 2023-11-28T18:10:10.768 DEBUG:teuthology.task.selinux:Excluding smithi007: OS 'ubuntu' does not support SELinux 2023-11-28T18:10:10.768 DEBUG:teuthology.task.selinux:Excluding smithi193: OS 'ubuntu' does not support SELinux 2023-11-28T18:10:10.768 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-28T18:10:10.769 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-11-28T18:10:10.769 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-28T18:10:10.769 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-28T18:10:10.787 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-28T18:10:10.787 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-28T18:10:10.809 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-28T18:10:10.810 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-28T18:19:01.495 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2023-11-28T18:19:01.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-28T18:19:01.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:19:01.580 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-11-28T18:19:01.693 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-28T18:19:01.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-28T18:19:01.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T18:19:01.771 DEBUG:teuthology.orchestra.run.smithi193:> true 2023-11-28T18:19:01.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-28T18:19:01.885 INFO:teuthology.run_tasks:Running task clock... 2023-11-28T18:19:01.896 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-28T18:19:01.896 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T18:19:01.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T18:19:01.900 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T18:19:01.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T18:19:01.937 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-28T18:19:01.938 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Command line: ntpd -gq 2023-11-28T18:19:01.938 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: ---------------------------------------------------- 2023-11-28T18:19:01.938 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: ntp-4 is maintained by Network Time Foundation, 2023-11-28T18:19:01.938 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-28T18:19:01.938 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: corporation. Support and training for ntp-4 are 2023-11-28T18:19:01.938 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: available at https://www.nwtime.org/support 2023-11-28T18:19:01.938 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: ---------------------------------------------------- 2023-11-28T18:19:01.939 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: proto: precision = 0.042 usec (-24) 2023-11-28T18:19:01.939 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: basedate set to 2022-02-04 2023-11-28T18:19:01.939 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: gps base set to 2022-02-06 (week 2196) 2023-11-28T18:19:01.939 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-28T18:19:01.940 INFO:teuthology.orchestra.run.smithi007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-28T18:19:01.940 INFO:teuthology.orchestra.run.smithi007.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-28T18:19:01.941 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: restrict ::: KOD does nothing without LIMITED. 2023-11-28T18:19:01.941 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Listen and drop on 0 v6wildcard [::]:123 2023-11-28T18:19:01.941 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-28T18:19:01.941 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Listen normally on 2 lo 127.0.0.1:123 2023-11-28T18:19:01.941 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Listen normally on 3 ens1f0 172.21.15.7:123 2023-11-28T18:19:01.942 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Listen normally on 4 lo [::1]:123 2023-11-28T18:19:01.942 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15ea%4]:123 2023-11-28T18:19:01.942 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:01 ntpd[27894]: Listening on routing socket on fd #22 for interface updates 2023-11-28T18:19:01.966 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-28T18:19:01.966 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Command line: ntpd -gq 2023-11-28T18:19:01.966 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: ---------------------------------------------------- 2023-11-28T18:19:01.966 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: ntp-4 is maintained by Network Time Foundation, 2023-11-28T18:19:01.966 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-28T18:19:01.967 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: corporation. Support and training for ntp-4 are 2023-11-28T18:19:01.967 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: available at https://www.nwtime.org/support 2023-11-28T18:19:01.967 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: ---------------------------------------------------- 2023-11-28T18:19:01.967 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: proto: precision = 0.043 usec (-24) 2023-11-28T18:19:01.968 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: basedate set to 2022-02-04 2023-11-28T18:19:01.968 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: gps base set to 2022-02-06 (week 2196) 2023-11-28T18:19:01.968 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-28T18:19:01.968 INFO:teuthology.orchestra.run.smithi193.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-28T18:19:01.968 INFO:teuthology.orchestra.run.smithi193.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-28T18:19:01.969 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: restrict ::: KOD does nothing without LIMITED. 2023-11-28T18:19:01.969 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Listen and drop on 0 v6wildcard [::]:123 2023-11-28T18:19:01.969 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-28T18:19:01.969 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Listen normally on 2 lo 127.0.0.1:123 2023-11-28T18:19:01.969 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Listen normally on 3 enp3s0f1 172.21.15.193:123 2023-11-28T18:19:01.970 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Listen normally on 4 lo [::1]:123 2023-11-28T18:19:01.970 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7253%5]:123 2023-11-28T18:19:01.970 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:01 ntpd[27983]: Listening on routing socket on fd #22 for interface updates 2023-11-28T18:19:02.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:02.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:02.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:02.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:02.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:02 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:03 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:03 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:03 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:03 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:03 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:03 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:03.967 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:03.967 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:03.967 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:03.967 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:03.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:03.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:03.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:03.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:03.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:03.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:03.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:03.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:03 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:04.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:04.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:04.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:04.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:04.941 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:04 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:05.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:05 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:05 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:05 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:05 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:05 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:05.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:05 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:05.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:05.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:05.970 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:05.970 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:05 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:06.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:06.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:06.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:06.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:06.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:06.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:06.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:06.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:06.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:06.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:06.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:06.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:06 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:07.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:07 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:07.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:07 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:07.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:07 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:07.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:07 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:07.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:07 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:07.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:07 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:07.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:07.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:07.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:07.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:07.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:07.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:07.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:07.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:07.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:07 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stdout:28 Nov 18:19:08 ntpd[27894]: ntpd: time slew -0.000433 s 2023-11-28T18:19:08.939 INFO:teuthology.orchestra.run.smithi007.stdout:ntpd: time slew -0.000433s 2023-11-28T18:19:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-28T18:19:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:28 Nov 18:19:08 ntpd[27894]: can't open /var/log/ntpstats/loopstats.20231128: Permission denied 2023-11-28T18:19:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2023-11-28T18:19:09.049 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2023-11-28T18:19:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-28T18:19:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-28T18:19:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-28T18:19:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-28T18:19:09.967 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: can't open /var/log/ntpstats/rawstats.20231128: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-28T18:19:09.968 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: can't open /var/log/ntpstats/peerstats.20231128: Permission denied 2023-11-28T18:19:09.969 INFO:teuthology.orchestra.run.smithi193.stdout:28 Nov 18:19:09 ntpd[27983]: ntpd: time slew -0.000810 s 2023-11-28T18:19:09.969 INFO:teuthology.orchestra.run.smithi193.stdout:ntpd: time slew -0.000810s 2023-11-28T18:19:09.969 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-28T18:19:09.970 INFO:teuthology.orchestra.run.smithi193.stderr:28 Nov 18:19:09 ntpd[27983]: can't open /var/log/ntpstats/loopstats.20231128: Permission denied 2023-11-28T18:19:10.054 INFO:teuthology.orchestra.run.smithi193.stdout: remote refid st t when poll reach delay offset jitter 2023-11-28T18:19:10.054 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================== 2023-11-28T18:19:10.054 INFO:teuthology.orchestra.run.smithi193.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-28T18:19:10.054 INFO:teuthology.orchestra.run.smithi193.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-28T18:19:10.055 INFO:teuthology.orchestra.run.smithi193.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-28T18:19:10.055 INFO:teuthology.orchestra.run.smithi193.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-28T18:19:10.056 INFO:teuthology.run_tasks:Running task install... 2023-11-28T18:19:10.067 DEBUG:teuthology.task.install:project ceph 2023-11-28T18:19:10.067 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}} 2023-11-28T18:19:10.067 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'} 2023-11-28T18:19:10.067 INFO:teuthology.task.install:Using flavor: default 2023-11-28T18:19:10.074 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-28T18:19:10.074 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-28T18:19:10.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-key list | grep Ceph 2023-11-28T18:19:10.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo apt-key list | grep Ceph 2023-11-28T18:19:10.187 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-28T18:19:10.202 INFO:teuthology.orchestra.run.smithi193.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-28T18:19:10.242 INFO:teuthology.orchestra.run.smithi007.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-28T18:19:10.243 INFO:teuthology.orchestra.run.smithi007.stdout:uid [ unknown] Ceph.com (release key) 2023-11-28T18:19:10.244 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-11-28T18:19:10.244 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-11-28T18:19:10.244 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T18:19:10.255 INFO:teuthology.orchestra.run.smithi193.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-28T18:19:10.255 INFO:teuthology.orchestra.run.smithi193.stdout:uid [ unknown] Ceph.com (release key) 2023-11-28T18:19:10.256 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-11-28T18:19:10.256 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-11-28T18:19:10.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T18:19:10.398 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default/ 2023-11-28T18:19:10.398 INFO:teuthology.task.install.deb:Package version is 18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:19:10.469 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default/ 2023-11-28T18:19:10.469 INFO:teuthology.task.install.deb:Package version is 18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:19:10.628 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:19:10.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-28T18:19:10.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2023-11-28T18:19:10.728 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:19:10.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-28T18:19:10.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo apt-get update 2023-11-28T18:19:10.841 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-28T18:19:10.847 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-28T18:19:10.864 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-28T18:19:10.940 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-28T18:19:10.940 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-28T18:19:10.951 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-28T18:19:10.958 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-28T18:19:10.973 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-28T18:19:11.136 INFO:teuthology.orchestra.run.smithi007.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy InRelease 2023-11-28T18:19:11.182 INFO:teuthology.orchestra.run.smithi193.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy InRelease 2023-11-28T18:19:11.222 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-28T18:19:11.256 INFO:teuthology.orchestra.run.smithi193.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-28T18:19:11.305 INFO:teuthology.orchestra.run.smithi007.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-28T18:19:11.320 INFO:teuthology.orchestra.run.smithi193.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-28T18:19:11.357 INFO:teuthology.orchestra.run.smithi193.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-28T18:19:11.372 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-28T18:19:11.436 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,403 B] 2023-11-28T18:19:11.437 INFO:teuthology.orchestra.run.smithi193.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,403 B] 2023-11-28T18:19:11.547 INFO:teuthology.orchestra.run.smithi193.stdout:Fetched 28.6 kB in 1s (49.8 kB/s) 2023-11-28T18:19:11.547 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 28.6 kB in 1s (42.7 kB/s) 2023-11-28T18:19:12.469 INFO:teuthology.orchestra.run.smithi193.stdout:Reading package lists... 2023-11-28T18:19:12.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7531-g47c1c547-1jammy cephadm=18.0.0-7531-g47c1c547-1jammy ceph-mds=18.0.0-7531-g47c1c547-1jammy ceph-mgr=18.0.0-7531-g47c1c547-1jammy ceph-common=18.0.0-7531-g47c1c547-1jammy ceph-fuse=18.0.0-7531-g47c1c547-1jammy ceph-test=18.0.0-7531-g47c1c547-1jammy ceph-volume=18.0.0-7531-g47c1c547-1jammy radosgw=18.0.0-7531-g47c1c547-1jammy python3-rados=18.0.0-7531-g47c1c547-1jammy python3-rgw=18.0.0-7531-g47c1c547-1jammy python3-cephfs=18.0.0-7531-g47c1c547-1jammy python3-rbd=18.0.0-7531-g47c1c547-1jammy libcephfs2=18.0.0-7531-g47c1c547-1jammy libcephfs-dev=18.0.0-7531-g47c1c547-1jammy librados2=18.0.0-7531-g47c1c547-1jammy librbd1=18.0.0-7531-g47c1c547-1jammy rbd-fuse=18.0.0-7531-g47c1c547-1jammy python3-cephfs=18.0.0-7531-g47c1c547-1jammy cephfs-shell=18.0.0-7531-g47c1c547-1jammy cephfs-top=18.0.0-7531-g47c1c547-1jammy cephfs-mirror=18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:19:12.499 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2023-11-28T18:19:12.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7531-g47c1c547-1jammy cephadm=18.0.0-7531-g47c1c547-1jammy ceph-mds=18.0.0-7531-g47c1c547-1jammy ceph-mgr=18.0.0-7531-g47c1c547-1jammy ceph-common=18.0.0-7531-g47c1c547-1jammy ceph-fuse=18.0.0-7531-g47c1c547-1jammy ceph-test=18.0.0-7531-g47c1c547-1jammy ceph-volume=18.0.0-7531-g47c1c547-1jammy radosgw=18.0.0-7531-g47c1c547-1jammy python3-rados=18.0.0-7531-g47c1c547-1jammy python3-rgw=18.0.0-7531-g47c1c547-1jammy python3-cephfs=18.0.0-7531-g47c1c547-1jammy python3-rbd=18.0.0-7531-g47c1c547-1jammy libcephfs2=18.0.0-7531-g47c1c547-1jammy libcephfs-dev=18.0.0-7531-g47c1c547-1jammy librados2=18.0.0-7531-g47c1c547-1jammy librbd1=18.0.0-7531-g47c1c547-1jammy rbd-fuse=18.0.0-7531-g47c1c547-1jammy python3-cephfs=18.0.0-7531-g47c1c547-1jammy cephfs-shell=18.0.0-7531-g47c1c547-1jammy cephfs-top=18.0.0-7531-g47c1c547-1jammy cephfs-mirror=18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:19:12.557 INFO:teuthology.orchestra.run.smithi193.stdout:Reading package lists... 2023-11-28T18:19:12.590 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2023-11-28T18:19:12.723 INFO:teuthology.orchestra.run.smithi193.stdout:Building dependency tree... 2023-11-28T18:19:12.723 INFO:teuthology.orchestra.run.smithi193.stdout:Reading state information... 2023-11-28T18:19:12.760 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2023-11-28T18:19:15.132 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2023-11-28T18:19:15.132 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2023-11-28T18:19:15.132 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-28T18:19:15.132 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-28T18:19:15.132 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-28T18:19:15.132 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-28T18:19:15.132 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-28T18:19:15.133 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-11-28T18:19:15.134 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-11-28T18:19:15.135 INFO:teuthology.orchestra.run.smithi007.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2023-11-28T18:19:15.135 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2023-11-28T18:19:15.135 INFO:teuthology.orchestra.run.smithi007.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-11-28T18:19:15.135 INFO:teuthology.orchestra.run.smithi007.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-11-28T18:19:15.135 INFO:teuthology.orchestra.run.smithi007.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-11-28T18:19:15.135 INFO:teuthology.orchestra.run.smithi007.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-11-28T18:19:15.135 INFO:teuthology.orchestra.run.smithi007.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-11-28T18:19:15.136 INFO:teuthology.orchestra.run.smithi007.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-11-28T18:19:15.136 INFO:teuthology.orchestra.run.smithi007.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-11-28T18:19:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:Recommended packages: 2023-11-28T18:19:15.136 INFO:teuthology.orchestra.run.smithi007.stdout: btrfs-tools 2023-11-28T18:19:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2023-11-28T18:19:15.136 INFO:teuthology.orchestra.run.smithi007.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-28T18:19:15.136 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-28T18:19:15.136 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-28T18:19:15.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout: socat unzip xmlstarlet zip 2023-11-28T18:19:15.138 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 124 newly installed, 0 to remove and 264 not upgraded. 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 187 MB of archives. 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 835 MB of additional disk space will be used. 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7531-g47c1c547-1jammy [3,466 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7531-g47c1c547-1jammy [3,111 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7531-g47c1c547-1jammy [690 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7531-g47c1c547-1jammy [356 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7531-g47c1c547-1jammy [32.8 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7531-g47c1c547-1jammy [180 kB] 2023-11-28T18:19:15.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7531-g47c1c547-1jammy [63.5 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7531-g47c1c547-1jammy [330 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7531-g47c1c547-1jammy [6,233 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7531-g47c1c547-1jammy [112 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7531-g47c1c547-1jammy [439 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7531-g47c1c547-1jammy [22.6 MB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-28T18:19:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-28T18:19:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-28T18:19:15.147 INFO:teuthology.orchestra.run.smithi193.stdout:The following packages were automatically installed and are no longer required: 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout:The following additional packages will be installed: 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-28T18:19:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-11-28T18:19:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout:Suggested packages: 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-11-28T18:19:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout:Recommended packages: 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: btrfs-tools 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout:The following NEW packages will be installed: 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-28T18:19:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-28T18:19:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-11-28T18:19:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-11-28T18:19:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-28T18:19:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-28T18:19:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-28T18:19:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-28T18:19:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-28T18:19:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-28T18:19:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: socat unzip xmlstarlet zip 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout:0 upgraded, 124 newly installed, 0 to remove and 264 not upgraded. 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout:Need to get 187 MB of archives. 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout:After this operation, 835 MB of additional disk space will be used. 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-28T18:19:15.155 INFO:teuthology.orchestra.run.smithi193.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7531-g47c1c547-1jammy [3,466 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-11-28T18:19:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-28T18:19:15.157 INFO:teuthology.orchestra.run.smithi193.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-28T18:19:15.158 INFO:teuthology.orchestra.run.smithi193.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-28T18:19:15.159 INFO:teuthology.orchestra.run.smithi193.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-28T18:19:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-28T18:19:15.161 INFO:teuthology.orchestra.run.smithi193.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-28T18:19:15.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7531-g47c1c547-1jammy [3,111 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7531-g47c1c547-1jammy [690 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7531-g47c1c547-1jammy [356 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7531-g47c1c547-1jammy [32.8 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7531-g47c1c547-1jammy [180 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7531-g47c1c547-1jammy [63.5 kB] 2023-11-28T18:19:15.163 INFO:teuthology.orchestra.run.smithi193.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7531-g47c1c547-1jammy [330 kB] 2023-11-28T18:19:15.164 INFO:teuthology.orchestra.run.smithi193.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7531-g47c1c547-1jammy [6,233 kB] 2023-11-28T18:19:15.192 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-28T18:19:15.220 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-28T18:19:15.231 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-28T18:19:15.274 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-28T18:19:15.451 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-28T18:19:15.465 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-28T18:19:15.472 INFO:teuthology.orchestra.run.smithi007.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-28T18:19:15.478 INFO:teuthology.orchestra.run.smithi007.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-28T18:19:15.486 INFO:teuthology.orchestra.run.smithi007.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7531-g47c1c547-1jammy [6,900 kB] 2023-11-28T18:19:15.609 INFO:teuthology.orchestra.run.smithi193.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7531-g47c1c547-1jammy [112 kB] 2023-11-28T18:19:15.620 INFO:teuthology.orchestra.run.smithi193.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7531-g47c1c547-1jammy [439 kB] 2023-11-28T18:19:15.661 INFO:teuthology.orchestra.run.smithi007.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-28T18:19:15.665 INFO:teuthology.orchestra.run.smithi193.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7531-g47c1c547-1jammy [22.6 MB] 2023-11-28T18:19:15.743 INFO:teuthology.orchestra.run.smithi007.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-28T18:19:15.758 INFO:teuthology.orchestra.run.smithi007.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-28T18:19:15.814 INFO:teuthology.orchestra.run.smithi007.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7531-g47c1c547-1jammy [238 kB] 2023-11-28T18:19:15.824 INFO:teuthology.orchestra.run.smithi007.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7531-g47c1c547-1jammy [124 kB] 2023-11-28T18:19:15.828 INFO:teuthology.orchestra.run.smithi007.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7531-g47c1c547-1jammy [1,496 kB] 2023-11-28T18:19:15.901 INFO:teuthology.orchestra.run.smithi007.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7531-g47c1c547-1jammy [9,957 kB] 2023-11-28T18:19:15.944 INFO:teuthology.orchestra.run.smithi007.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-28T18:19:15.995 INFO:teuthology.orchestra.run.smithi007.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-28T18:19:16.008 INFO:teuthology.orchestra.run.smithi007.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-28T18:19:16.166 INFO:teuthology.orchestra.run.smithi007.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-28T18:19:16.166 INFO:teuthology.orchestra.run.smithi007.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-28T18:19:16.171 INFO:teuthology.orchestra.run.smithi007.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-28T18:19:16.177 INFO:teuthology.orchestra.run.smithi007.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-28T18:19:16.191 INFO:teuthology.orchestra.run.smithi007.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-28T18:19:16.336 INFO:teuthology.orchestra.run.smithi007.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7531-g47c1c547-1jammy [31.9 MB] 2023-11-28T18:19:16.375 INFO:teuthology.orchestra.run.smithi007.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-28T18:19:16.628 INFO:teuthology.orchestra.run.smithi007.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-11-28T18:19:16.761 INFO:teuthology.orchestra.run.smithi007.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-28T18:19:16.805 INFO:teuthology.orchestra.run.smithi007.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-28T18:19:16.896 INFO:teuthology.orchestra.run.smithi007.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-28T18:19:16.905 INFO:teuthology.orchestra.run.smithi007.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-28T18:19:16.918 INFO:teuthology.orchestra.run.smithi007.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-28T18:19:16.961 INFO:teuthology.orchestra.run.smithi007.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-28T18:19:17.224 INFO:teuthology.orchestra.run.smithi007.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-28T18:19:17.224 INFO:teuthology.orchestra.run.smithi007.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-28T18:19:17.226 INFO:teuthology.orchestra.run.smithi007.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-28T18:19:17.391 INFO:teuthology.orchestra.run.smithi007.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-28T18:19:18.170 INFO:teuthology.orchestra.run.smithi007.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-28T18:19:18.170 INFO:teuthology.orchestra.run.smithi007.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7531-g47c1c547-1jammy [14.0 kB] 2023-11-28T18:19:18.170 INFO:teuthology.orchestra.run.smithi007.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7531-g47c1c547-1jammy [839 kB] 2023-11-28T18:19:18.170 INFO:teuthology.orchestra.run.smithi007.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7531-g47c1c547-1jammy [2,144 kB] 2023-11-28T18:19:18.170 INFO:teuthology.orchestra.run.smithi007.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7531-g47c1c547-1jammy [802 kB] 2023-11-28T18:19:18.170 INFO:teuthology.orchestra.run.smithi007.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7531-g47c1c547-1jammy [139 kB] 2023-11-28T18:19:18.170 INFO:teuthology.orchestra.run.smithi007.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7531-g47c1c547-1jammy [1,900 kB] 2023-11-28T18:19:18.170 INFO:teuthology.orchestra.run.smithi007.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7531-g47c1c547-1jammy [8,625 kB] 2023-11-28T18:19:18.171 INFO:teuthology.orchestra.run.smithi007.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-28T18:19:18.171 INFO:teuthology.orchestra.run.smithi007.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-28T18:19:18.171 INFO:teuthology.orchestra.run.smithi007.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-28T18:19:18.212 INFO:teuthology.orchestra.run.smithi007.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-28T18:19:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-28T18:19:18.214 INFO:teuthology.orchestra.run.smithi007.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-28T18:19:18.215 INFO:teuthology.orchestra.run.smithi007.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-28T18:19:18.215 INFO:teuthology.orchestra.run.smithi007.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-28T18:19:18.217 INFO:teuthology.orchestra.run.smithi007.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-28T18:19:18.381 INFO:teuthology.orchestra.run.smithi007.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-28T18:19:18.469 INFO:teuthology.orchestra.run.smithi007.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7531-g47c1c547-1jammy [14.2 kB] 2023-11-28T18:19:18.470 INFO:teuthology.orchestra.run.smithi007.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7531-g47c1c547-1jammy [48.7 MB] 2023-11-28T18:19:18.585 INFO:teuthology.orchestra.run.smithi007.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-28T18:19:18.909 INFO:teuthology.orchestra.run.smithi007.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-28T18:19:18.956 INFO:teuthology.orchestra.run.smithi007.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-28T18:19:18.982 INFO:teuthology.orchestra.run.smithi007.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-28T18:19:18.991 INFO:teuthology.orchestra.run.smithi007.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-28T18:19:19.039 INFO:teuthology.orchestra.run.smithi007.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-28T18:19:19.064 INFO:teuthology.orchestra.run.smithi007.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-11-28T18:19:19.066 INFO:teuthology.orchestra.run.smithi007.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-11-28T18:19:19.074 INFO:teuthology.orchestra.run.smithi007.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-11-28T18:19:19.077 INFO:teuthology.orchestra.run.smithi007.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-28T18:19:19.214 INFO:teuthology.orchestra.run.smithi193.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7531-g47c1c547-1jammy [6,900 kB] 2023-11-28T18:19:19.273 INFO:teuthology.orchestra.run.smithi007.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-11-28T18:19:19.532 INFO:teuthology.orchestra.run.smithi007.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-28T18:19:19.794 INFO:teuthology.orchestra.run.smithi007.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-28T18:19:19.796 INFO:teuthology.orchestra.run.smithi007.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-28T18:19:19.812 INFO:teuthology.orchestra.run.smithi007.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-28T18:19:19.818 INFO:teuthology.orchestra.run.smithi007.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-28T18:19:19.856 INFO:teuthology.orchestra.run.smithi007.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-28T18:19:19.870 INFO:teuthology.orchestra.run.smithi007.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-28T18:19:19.961 INFO:teuthology.orchestra.run.smithi007.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-28T18:19:19.962 INFO:teuthology.orchestra.run.smithi007.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-28T18:19:19.964 INFO:teuthology.orchestra.run.smithi007.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-28T18:19:20.143 INFO:teuthology.orchestra.run.smithi007.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-28T18:19:20.234 INFO:teuthology.orchestra.run.smithi193.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7531-g47c1c547-1jammy [238 kB] 2023-11-28T18:19:20.268 INFO:teuthology.orchestra.run.smithi193.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7531-g47c1c547-1jammy [124 kB] 2023-11-28T18:19:20.284 INFO:teuthology.orchestra.run.smithi193.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7531-g47c1c547-1jammy [1,496 kB] 2023-11-28T18:19:20.406 INFO:teuthology.orchestra.run.smithi007.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-28T18:19:20.509 INFO:teuthology.orchestra.run.smithi193.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7531-g47c1c547-1jammy [9,957 kB] 2023-11-28T18:19:20.537 INFO:teuthology.orchestra.run.smithi007.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7531-g47c1c547-1jammy [121 kB] 2023-11-28T18:19:20.538 INFO:teuthology.orchestra.run.smithi007.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-7531-g47c1c547-1jammy [225 kB] 2023-11-28T18:19:20.540 INFO:teuthology.orchestra.run.smithi007.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-7531-g47c1c547-1jammy [34.0 kB] 2023-11-28T18:19:20.540 INFO:teuthology.orchestra.run.smithi007.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-7531-g47c1c547-1jammy [23.4 kB] 2023-11-28T18:19:20.540 INFO:teuthology.orchestra.run.smithi007.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7531-g47c1c547-1jammy [38.3 kB] 2023-11-28T18:19:20.541 INFO:teuthology.orchestra.run.smithi007.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7531-g47c1c547-1jammy [12.3 MB] 2023-11-28T18:19:21.284 INFO:teuthology.orchestra.run.smithi007.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7531-g47c1c547-1jammy [91.4 kB] 2023-11-28T18:19:21.949 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 187 MB in 8s (22.8 MB/s) 2023-11-28T18:19:21.968 INFO:teuthology.orchestra.run.smithi193.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7531-g47c1c547-1jammy [31.9 MB] 2023-11-28T18:19:22.070 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyperclip. 2023-11-28T18:19:22.141 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132930 files and directories currently installed.) 2023-11-28T18:19:22.144 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-11-28T18:19:22.177 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-11-28T18:19:22.463 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-28T18:19:22.478 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-28T18:19:22.521 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-28T18:19:22.957 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cmd2. 2023-11-28T18:19:22.972 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-11-28T18:19:23.016 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-11-28T18:19:23.360 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-28T18:19:23.376 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-28T18:19:23.461 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-28T18:19:23.739 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-28T18:19:23.754 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-28T18:19:23.789 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-28T18:19:24.101 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-28T18:19:24.116 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-28T18:19:24.177 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-28T18:19:24.812 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-28T18:19:24.828 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-28T18:19:24.879 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:19:25.188 INFO:teuthology.orchestra.run.smithi193.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7531-g47c1c547-1jammy [14.0 kB] 2023-11-28T18:19:25.189 INFO:teuthology.orchestra.run.smithi193.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7531-g47c1c547-1jammy [839 kB] 2023-11-28T18:19:25.198 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-28T18:19:25.214 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-28T18:19:25.249 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:19:25.261 INFO:teuthology.orchestra.run.smithi193.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7531-g47c1c547-1jammy [2,144 kB] 2023-11-28T18:19:25.462 INFO:teuthology.orchestra.run.smithi193.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7531-g47c1c547-1jammy [802 kB] 2023-11-28T18:19:25.533 INFO:teuthology.orchestra.run.smithi193.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7531-g47c1c547-1jammy [139 kB] 2023-11-28T18:19:25.545 INFO:teuthology.orchestra.run.smithi193.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7531-g47c1c547-1jammy [1,900 kB] 2023-11-28T18:19:25.577 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-28T18:19:25.592 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-28T18:19:25.635 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:19:25.712 INFO:teuthology.orchestra.run.smithi193.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7531-g47c1c547-1jammy [8,625 kB] 2023-11-28T18:19:25.997 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-28T18:19:26.012 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-28T18:19:26.055 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-28T18:19:26.341 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librados2. 2023-11-28T18:19:26.357 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../010-librados2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:26.391 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librados2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:26.505 INFO:teuthology.orchestra.run.smithi193.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7531-g47c1c547-1jammy [14.2 kB] 2023-11-28T18:19:26.505 INFO:teuthology.orchestra.run.smithi193.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7531-g47c1c547-1jammy [48.7 MB] 2023-11-28T18:19:26.819 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librbd1. 2023-11-28T18:19:26.834 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../011-librbd1_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:26.877 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librbd1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:27.313 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libcephfs2. 2023-11-28T18:19:27.329 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../012-libcephfs2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:27.372 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libcephfs2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:27.649 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rados. 2023-11-28T18:19:27.665 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../013-python3-rados_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:27.699 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rados (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:27.952 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-28T18:19:27.968 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:28.023 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-ceph-argparse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:28.271 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cephfs. 2023-11-28T18:19:28.287 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../015-python3-cephfs_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:28.330 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cephfs (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:28.624 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-28T18:19:28.639 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../016-python3-ceph-common_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:28.682 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:29.019 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-prettytable. 2023-11-28T18:19:29.034 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2023-11-28T18:19:29.077 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-28T18:19:29.347 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rbd. 2023-11-28T18:19:29.362 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../018-python3-rbd_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:29.505 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rbd (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:30.075 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-28T18:19:30.090 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-28T18:19:30.133 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-28T18:19:30.154 INFO:teuthology.orchestra.run.smithi193.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7531-g47c1c547-1jammy [121 kB] 2023-11-28T18:19:30.155 INFO:teuthology.orchestra.run.smithi193.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-7531-g47c1c547-1jammy [225 kB] 2023-11-28T18:19:30.169 INFO:teuthology.orchestra.run.smithi193.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-7531-g47c1c547-1jammy [34.0 kB] 2023-11-28T18:19:30.170 INFO:teuthology.orchestra.run.smithi193.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-7531-g47c1c547-1jammy [23.4 kB] 2023-11-28T18:19:30.172 INFO:teuthology.orchestra.run.smithi193.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7531-g47c1c547-1jammy [38.3 kB] 2023-11-28T18:19:30.174 INFO:teuthology.orchestra.run.smithi193.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7531-g47c1c547-1jammy [12.3 MB] 2023-11-28T18:19:30.411 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-11-28T18:19:30.426 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2023-11-28T18:19:30.469 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-11-28T18:19:30.805 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-11-28T18:19:30.821 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-11-28T18:19:30.837 INFO:teuthology.orchestra.run.smithi193.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7531-g47c1c547-1jammy [91.4 kB] 2023-11-28T18:19:30.863 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-11-28T18:19:31.175 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua5.1. 2023-11-28T18:19:31.191 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-11-28T18:19:31.234 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-11-28T18:19:31.473 INFO:teuthology.orchestra.run.smithi193.stdout:Fetched 187 MB in 18s (10.5 MB/s) 2023-11-28T18:19:31.537 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-any. 2023-11-28T18:19:31.552 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2023-11-28T18:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-any (27ubuntu1) ... 2023-11-28T18:19:31.601 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pyperclip. 2023-11-28T18:19:31.670 INFO:teuthology.orchestra.run.smithi193.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132930 files and directories currently installed.) 2023-11-28T18:19:31.673 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-11-28T18:19:31.708 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-11-28T18:19:31.872 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package zip. 2023-11-28T18:19:31.888 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2023-11-28T18:19:31.931 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking zip (3.0-12build2) ... 2023-11-28T18:19:32.002 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-28T18:19:32.015 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-28T18:19:32.060 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-28T18:19:32.226 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package unzip. 2023-11-28T18:19:32.241 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-11-28T18:19:32.276 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-11-28T18:19:32.372 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-cmd2. 2023-11-28T18:19:32.385 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-11-28T18:19:32.430 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-11-28T18:19:32.680 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package luarocks. 2023-11-28T18:19:32.695 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-11-28T18:19:32.738 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-11-28T18:19:32.774 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-28T18:19:32.788 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-28T18:19:32.866 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-28T18:19:33.113 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librgw2. 2023-11-28T18:19:33.127 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../027-librgw2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:33.180 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librgw2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:33.220 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-28T18:19:33.233 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-28T18:19:33.287 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-28T18:19:33.564 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-28T18:19:33.578 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-28T18:19:33.640 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rgw. 2023-11-28T18:19:33.655 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../028-python3-rgw_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:33.698 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:33.823 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-28T18:19:34.043 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-28T18:19:34.060 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-28T18:19:34.102 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-28T18:19:34.259 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-28T18:19:34.273 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-28T18:19:34.342 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:19:34.363 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libradosstriper1. 2023-11-28T18:19:34.378 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../030-libradosstriper1_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:34.446 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libradosstriper1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:34.721 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-28T18:19:34.735 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-28T18:19:34.779 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:19:34.874 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-common. 2023-11-28T18:19:34.890 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../031-ceph-common_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:34.933 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:35.074 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-28T18:19:35.088 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-28T18:19:35.132 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:19:35.460 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-28T18:19:35.474 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-28T18:19:35.543 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-28T18:19:35.863 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package librados2. 2023-11-28T18:19:35.878 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../010-librados2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:35.921 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking librados2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:35.980 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-base. 2023-11-28T18:19:35.995 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../032-ceph-base_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:36.048 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-base (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:36.299 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package librbd1. 2023-11-28T18:19:36.313 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../011-librbd1_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:36.349 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking librbd1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:36.734 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-28T18:19:36.749 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-28T18:19:36.777 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libcephfs2. 2023-11-28T18:19:36.790 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../012-libcephfs2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:36.793 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-28T18:19:36.835 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libcephfs2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:37.104 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cheroot. 2023-11-28T18:19:37.119 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-28T18:19:37.121 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-rados. 2023-11-28T18:19:37.134 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../013-python3-rados_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:37.154 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-28T18:19:37.171 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-rados (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:37.449 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-28T18:19:37.463 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:37.466 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-28T18:19:37.482 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-28T18:19:37.507 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-ceph-argparse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:37.515 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-28T18:19:37.726 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-cephfs. 2023-11-28T18:19:37.739 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../015-python3-cephfs_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:37.768 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-cephfs (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:37.802 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-28T18:19:37.818 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-28T18:19:37.860 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-28T18:19:38.087 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-28T18:19:38.100 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../016-python3-ceph-common_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:38.145 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:38.163 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-28T18:19:38.178 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-28T18:19:38.221 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-28T18:19:38.473 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-prettytable. 2023-11-28T18:19:38.487 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2023-11-28T18:19:38.499 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-tempora. 2023-11-28T18:19:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2023-11-28T18:19:38.532 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-28T18:19:38.549 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-28T18:19:38.784 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-rbd. 2023-11-28T18:19:38.798 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../018-python3-rbd_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:38.842 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-rbd (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:38.852 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-portend. 2023-11-28T18:19:38.867 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2023-11-28T18:19:38.910 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-28T18:19:39.213 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-28T18:19:39.229 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-28T18:19:39.272 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-28T18:19:39.304 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-28T18:19:39.318 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-28T18:19:39.354 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-28T18:19:39.600 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-28T18:19:39.615 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-28T18:19:39.631 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-11-28T18:19:39.645 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2023-11-28T18:19:39.690 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-11-28T18:19:39.841 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-28T18:19:39.959 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-11-28T18:19:39.973 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-11-28T18:19:40.017 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-11-28T18:19:40.195 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-natsort. 2023-11-28T18:19:40.210 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2023-11-28T18:19:40.253 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-28T18:19:40.362 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package lua5.1. 2023-11-28T18:19:40.377 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-11-28T18:19:40.420 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-11-28T18:19:40.548 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-logutils. 2023-11-28T18:19:40.563 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2023-11-28T18:19:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-28T18:19:40.710 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package lua-any. 2023-11-28T18:19:40.721 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2023-11-28T18:19:40.765 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking lua-any (27ubuntu1) ... 2023-11-28T18:19:40.900 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-28T18:19:40.916 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-28T18:19:40.951 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-28T18:19:41.018 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package zip. 2023-11-28T18:19:41.032 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2023-11-28T18:19:41.076 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking zip (3.0-12build2) ... 2023-11-28T18:19:41.212 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-mako. 2023-11-28T18:19:41.227 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-28T18:19:41.270 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-28T18:19:41.396 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package unzip. 2023-11-28T18:19:41.411 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-11-28T18:19:41.471 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-11-28T18:19:41.607 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-28T18:19:41.623 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-28T18:19:41.665 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-28T18:19:41.850 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package luarocks. 2023-11-28T18:19:41.864 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-11-28T18:19:41.899 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-11-28T18:19:41.968 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-28T18:19:41.984 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-28T18:19:42.026 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-28T18:19:42.274 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package librgw2. 2023-11-28T18:19:42.287 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../027-librgw2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:42.332 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking librgw2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:42.346 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-webob. 2023-11-28T18:19:42.362 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-28T18:19:42.396 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-28T18:19:42.699 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-waitress. 2023-11-28T18:19:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-28T18:19:42.768 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-28T18:19:42.834 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-rgw. 2023-11-28T18:19:42.848 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../028-python3-rgw_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:42.893 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-rgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:43.110 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-tempita. 2023-11-28T18:19:43.126 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-28T18:19:43.169 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-28T18:19:43.229 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-28T18:19:43.244 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-28T18:19:43.296 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-28T18:19:43.497 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-paste. 2023-11-28T18:19:43.512 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-28T18:19:43.555 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-28T18:19:43.598 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libradosstriper1. 2023-11-28T18:19:43.613 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../030-libradosstriper1_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:43.657 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libradosstriper1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:43.858 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-28T18:19:43.873 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-28T18:19:43.916 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-28T18:19:44.102 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-common. 2023-11-28T18:19:44.117 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../031-ceph-common_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:44.160 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:44.220 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-28T18:19:44.235 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-28T18:19:44.278 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-28T18:19:44.581 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-webtest. 2023-11-28T18:19:44.598 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2023-11-28T18:19:44.639 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-28T18:19:44.934 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pecan. 2023-11-28T18:19:44.949 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-28T18:19:44.983 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-28T18:19:45.157 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-base. 2023-11-28T18:19:45.172 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../032-ceph-base_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:45.225 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-base (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:45.877 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-28T18:19:45.891 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-28T18:19:45.972 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-certifi. 2023-11-28T18:19:45.988 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2023-11-28T18:19:46.030 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-28T18:19:46.375 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-idna. 2023-11-28T18:19:46.390 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2023-11-28T18:19:46.433 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-28T18:19:46.486 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-28T18:19:46.728 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-urllib3. 2023-11-28T18:19:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-11-28T18:19:46.778 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-11-28T18:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-cheroot. 2023-11-28T18:19:46.796 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-28T18:19:46.839 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-28T18:19:47.081 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-requests. 2023-11-28T18:19:47.097 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-28T18:19:47.139 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-28T18:19:47.159 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-28T18:19:47.173 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-28T18:19:47.217 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-28T18:19:47.426 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-28T18:19:47.441 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-28T18:19:47.484 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-28T18:19:47.486 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-28T18:19:47.501 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-28T18:19:47.536 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-28T18:19:47.746 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-28T18:19:47.761 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:47.795 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:47.856 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-28T18:19:47.871 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-28T18:19:47.914 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-28T18:19:48.184 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-28T18:19:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:48.217 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-tempora. 2023-11-28T18:19:48.231 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2023-11-28T18:19:48.242 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:48.276 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-28T18:19:48.538 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-28T18:19:48.554 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-28T18:19:48.578 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-portend. 2023-11-28T18:19:48.593 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2023-11-28T18:19:48.596 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-28T18:19:48.636 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-28T18:19:48.875 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-openssl. 2023-11-28T18:19:48.890 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2023-11-28T18:19:48.924 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-28T18:19:48.931 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-28T18:19:48.945 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-28T18:19:48.989 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-28T18:19:49.235 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr. 2023-11-28T18:19:49.251 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../065-ceph-mgr_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:49.294 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:49.492 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-28T18:19:49.507 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-28T18:19:49.550 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-28T18:19:49.689 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mon. 2023-11-28T18:19:49.704 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../066-ceph-mon_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:49.747 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mon (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:49.862 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-natsort. 2023-11-28T18:19:49.876 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2023-11-28T18:19:49.912 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-28T18:19:50.248 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-logutils. 2023-11-28T18:19:50.263 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2023-11-28T18:19:50.307 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-28T18:19:50.484 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-osd. 2023-11-28T18:19:50.499 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../067-ceph-osd_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:50.533 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-osd (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:50.626 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-28T18:19:50.641 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-28T18:19:50.684 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-28T18:19:50.987 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-mako. 2023-11-28T18:19:51.002 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-28T18:19:51.045 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-28T18:19:51.374 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-28T18:19:51.389 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-28T18:19:51.424 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-28T18:19:51.710 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-28T18:19:51.725 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-28T18:19:51.753 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph. 2023-11-28T18:19:51.768 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-28T18:19:51.768 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../068-ceph_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:51.811 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:52.071 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-webob. 2023-11-28T18:19:52.081 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-fuse. 2023-11-28T18:19:52.086 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-28T18:19:52.096 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../069-ceph-fuse_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:52.129 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-28T18:19:52.139 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:52.507 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-waitress. 2023-11-28T18:19:52.522 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-28T18:19:52.525 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mds. 2023-11-28T18:19:52.541 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../070-ceph-mds_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:52.568 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-28T18:19:52.583 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mds (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:52.885 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-tempita. 2023-11-28T18:19:52.900 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-28T18:19:52.943 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-28T18:19:52.952 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephadm. 2023-11-28T18:19:52.968 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../071-cephadm_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:53.011 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:53.188 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-paste. 2023-11-28T18:19:53.203 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-28T18:19:53.246 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-28T18:19:53.314 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-28T18:19:53.330 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-28T18:19:53.363 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-28T18:19:53.591 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-28T18:19:53.605 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-28T18:19:53.649 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-28T18:19:53.692 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-28T18:19:53.707 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-28T18:19:53.750 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-28T18:19:53.910 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-28T18:19:53.924 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-28T18:19:53.960 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-28T18:19:54.213 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-webtest. 2023-11-28T18:19:54.227 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2023-11-28T18:19:54.263 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-28T18:19:54.549 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pecan. 2023-11-28T18:19:54.563 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-28T18:19:54.607 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-28T18:19:54.721 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-babel. 2023-11-28T18:19:54.736 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-28T18:19:54.770 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-28T18:19:54.987 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-certifi. 2023-11-28T18:19:55.001 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2023-11-28T18:19:55.037 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-28T18:19:55.082 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jinja2. 2023-11-28T18:19:55.097 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2023-11-28T18:19:55.132 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-28T18:19:55.298 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-idna. 2023-11-28T18:19:55.312 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2023-11-28T18:19:55.356 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-28T18:19:55.626 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-28T18:19:55.643 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:55.667 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-urllib3. 2023-11-28T18:19:55.676 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:55.681 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-11-28T18:19:55.726 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-11-28T18:19:55.971 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-28T18:19:55.987 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2023-11-28T18:19:56.021 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-28T18:19:56.071 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-requests. 2023-11-28T18:19:56.085 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-28T18:19:56.129 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-28T18:19:56.316 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-routes. 2023-11-28T18:19:56.331 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-28T18:19:56.382 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-28T18:19:56.440 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-28T18:19:56.454 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-28T18:19:56.498 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-28T18:19:56.668 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-28T18:19:56.685 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:56.718 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:56.801 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-28T18:19:56.815 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:56.859 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:57.130 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-28T18:19:57.145 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-28T18:19:57.188 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-28T18:19:57.239 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-28T18:19:57.253 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:57.289 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:57.601 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-28T18:19:57.614 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-28T18:19:57.659 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-28T18:19:57.716 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-joblib. 2023-11-28T18:19:57.733 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-28T18:19:57.774 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-28T18:19:57.961 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-openssl. 2023-11-28T18:19:57.975 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2023-11-28T18:19:58.020 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-28T18:19:58.127 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-28T18:19:58.143 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-28T18:19:58.263 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-28T18:19:58.383 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-mgr. 2023-11-28T18:19:58.397 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../065-ceph-mgr_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:58.441 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-mgr (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:58.566 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-sklearn. 2023-11-28T18:19:58.582 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-28T18:19:58.624 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-28T18:19:58.811 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-mon. 2023-11-28T18:19:58.825 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../066-ceph-mon_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:58.869 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-mon (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:59.325 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-28T18:19:59.339 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:19:59.383 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:19:59.689 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-osd. 2023-11-28T18:19:59.704 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../067-ceph-osd_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:19:59.856 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-osd (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:00.402 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cachetools. 2023-11-28T18:20:00.418 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2023-11-28T18:20:00.461 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-28T18:20:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-28T18:20:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-28T18:20:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-28T18:20:01.100 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph. 2023-11-28T18:20:01.115 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../068-ceph_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:01.125 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-28T18:20:01.142 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-28T18:20:01.150 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:01.217 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-28T18:20:01.478 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-fuse. 2023-11-28T18:20:01.493 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../069-ceph-fuse_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:01.520 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rsa. 2023-11-28T18:20:01.536 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2023-11-28T18:20:01.544 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:01.570 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-28T18:20:01.882 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-google-auth. 2023-11-28T18:20:01.898 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2023-11-28T18:20:01.915 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-mds. 2023-11-28T18:20:01.929 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../070-ceph-mds_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:01.941 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-28T18:20:01.972 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-mds (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:02.243 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-28T18:20:02.259 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-28T18:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-28T18:20:02.342 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package cephadm. 2023-11-28T18:20:02.356 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../071-cephadm_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:02.392 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:02.604 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-websocket. 2023-11-28T18:20:02.620 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2023-11-28T18:20:02.644 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-28T18:20:02.658 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-28T18:20:02.662 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-28T18:20:02.702 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-28T18:20:02.997 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-28T18:20:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-28T18:20:03.011 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-28T18:20:03.023 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-28T18:20:03.055 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-28T18:20:03.057 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-28T18:20:03.745 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-28T18:20:03.760 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:03.812 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:04.002 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-babel. 2023-11-28T18:20:04.016 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-28T18:20:04.060 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-28T18:20:04.157 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-28T18:20:04.172 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-28T18:20:04.215 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-28T18:20:04.354 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-jinja2. 2023-11-28T18:20:04.368 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2023-11-28T18:20:04.412 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-28T18:20:04.526 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-28T18:20:04.541 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-28T18:20:04.584 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-28T18:20:04.749 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-28T18:20:04.763 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:04.807 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:04.845 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package jq. 2023-11-28T18:20:04.860 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-28T18:20:04.903 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-28T18:20:05.068 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-28T18:20:05.082 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2023-11-28T18:20:05.127 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-28T18:20:05.156 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package socat. 2023-11-28T18:20:05.172 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-28T18:20:05.223 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-28T18:20:05.429 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-routes. 2023-11-28T18:20:05.443 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-28T18:20:05.487 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-28T18:20:05.693 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package xmlstarlet. 2023-11-28T18:20:05.709 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-28T18:20:05.751 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-28T18:20:05.849 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-28T18:20:05.863 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:05.907 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:06.105 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-test. 2023-11-28T18:20:06.120 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../099-ceph-test_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:06.163 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-test (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:06.319 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-28T18:20:06.333 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-28T18:20:06.526 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-28T18:20:07.030 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-joblib. 2023-11-28T18:20:07.044 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-28T18:20:07.088 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-28T18:20:07.450 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-28T18:20:07.464 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-28T18:20:07.593 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-28T18:20:07.854 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-sklearn. 2023-11-28T18:20:07.869 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-28T18:20:07.913 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-28T18:20:08.067 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-volume. 2023-11-28T18:20:08.083 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../100-ceph-volume_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:08.125 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-volume (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:08.496 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephfs-mirror. 2023-11-28T18:20:08.512 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../101-cephfs-mirror_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:08.562 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephfs-mirror (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:08.563 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-28T18:20:08.577 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:08.621 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:08.857 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-colorama. 2023-11-28T18:20:08.872 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2023-11-28T18:20:08.915 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-11-28T18:20:09.176 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephfs-shell. 2023-11-28T18:20:09.191 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../103-cephfs-shell_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:09.259 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephfs-shell (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:09.478 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephfs-top. 2023-11-28T18:20:09.495 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../104-cephfs-top_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:09.528 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephfs-top (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:09.566 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-cachetools. 2023-11-28T18:20:09.581 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2023-11-28T18:20:09.617 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-28T18:20:09.789 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-28T18:20:09.805 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../105-libcephfs-dev_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:09.847 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libcephfs-dev (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:09.919 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-28T18:20:09.935 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-28T18:20:09.977 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-28T18:20:10.166 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-socket:amd64. 2023-11-28T18:20:10.182 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-11-28T18:20:10.224 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-11-28T18:20:10.297 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-28T18:20:10.313 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-28T18:20:10.355 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-28T18:20:10.662 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-sec:amd64. 2023-11-28T18:20:10.678 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2023-11-28T18:20:10.708 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-rsa. 2023-11-28T18:20:10.723 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2023-11-28T18:20:10.767 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-28T18:20:10.795 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-11-28T18:20:11.087 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-google-auth. 2023-11-28T18:20:11.103 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2023-11-28T18:20:11.145 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-28T18:20:11.149 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package nvme-cli. 2023-11-28T18:20:11.165 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-28T18:20:11.207 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-28T18:20:11.465 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-28T18:20:11.480 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-28T18:20:11.514 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-28T18:20:11.621 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package pkg-config. 2023-11-28T18:20:11.636 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-11-28T18:20:11.679 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-11-28T18:20:11.842 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-websocket. 2023-11-28T18:20:11.858 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2023-11-28T18:20:11.901 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-28T18:20:11.965 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-28T18:20:11.981 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-28T18:20:12.023 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-28T18:20:12.220 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-28T18:20:12.236 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-28T18:20:12.282 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-28T18:20:12.418 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-28T18:20:12.433 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-28T18:20:12.468 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-28T18:20:12.754 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pastescript. 2023-11-28T18:20:12.769 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2023-11-28T18:20:12.812 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-28T18:20:12.984 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-28T18:20:12.998 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:13.042 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:13.132 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pluggy. 2023-11-28T18:20:13.148 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-28T18:20:13.182 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-28T18:20:13.328 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-28T18:20:13.342 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-28T18:20:13.378 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-28T18:20:13.460 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-psutil. 2023-11-28T18:20:13.476 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-28T18:20:13.518 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-28T18:20:13.681 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-28T18:20:13.695 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-28T18:20:13.739 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-28T18:20:14.001 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package jq. 2023-11-28T18:20:14.014 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-28T18:20:14.021 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-py. 2023-11-28T18:20:14.037 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2023-11-28T18:20:14.079 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-28T18:20:14.083 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-28T18:20:14.336 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package socat. 2023-11-28T18:20:14.350 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-28T18:20:14.394 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-28T18:20:14.432 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pygments. 2023-11-28T18:20:14.449 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-28T18:20:14.491 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-28T18:20:14.697 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package xmlstarlet. 2023-11-28T18:20:14.711 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-28T18:20:14.755 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-28T18:20:14.936 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-28T18:20:14.951 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-28T18:20:14.994 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-28T18:20:15.272 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-toml. 2023-11-28T18:20:15.287 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2023-11-28T18:20:15.322 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-28T18:20:15.368 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-test. 2023-11-28T18:20:15.382 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../099-ceph-test_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:15.426 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-test (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:15.574 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pytest. 2023-11-28T18:20:15.590 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-28T18:20:15.666 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-28T18:20:16.019 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-simplejson. 2023-11-28T18:20:16.036 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-28T18:20:16.077 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-28T18:20:16.339 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-28T18:20:16.355 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-28T18:20:16.390 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-28T18:20:17.118 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package radosgw. 2023-11-28T18:20:17.133 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../122-radosgw_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking radosgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:17.230 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ceph-volume. 2023-11-28T18:20:17.245 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../100-ceph-volume_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:17.296 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ceph-volume (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:17.659 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package cephfs-mirror. 2023-11-28T18:20:17.673 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../101-cephfs-mirror_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:17.739 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking cephfs-mirror (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:17.753 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package rbd-fuse. 2023-11-28T18:20:17.768 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../123-rbd-fuse_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:17.811 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking rbd-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:18.012 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-colorama. 2023-11-28T18:20:18.026 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2023-11-28T18:20:18.061 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-11-28T18:20:18.089 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-28T18:20:18.322 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package cephfs-shell. 2023-11-28T18:20:18.333 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-28T18:20:18.336 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../103-cephfs-shell_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:18.380 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking cephfs-shell (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:18.459 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-28T18:20:18.633 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package cephfs-top. 2023-11-28T18:20:18.648 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../104-cephfs-top_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-11-28T18:20:18.691 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking cephfs-top (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:18.730 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-28T18:20:18.944 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-28T18:20:18.959 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../105-libcephfs-dev_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:19.001 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libcephfs-dev (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:19.151 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-28T18:20:19.279 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package lua-socket:amd64. 2023-11-28T18:20:19.365 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-11-28T18:20:19.365 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-11-28T18:20:19.600 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package lua-sec:amd64. 2023-11-28T18:20:19.615 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2023-11-28T18:20:19.658 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-11-28T18:20:19.766 INFO:teuthology.orchestra.run.smithi007.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-28T18:20:19.800 INFO:teuthology.orchestra.run.smithi007.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-11-28T18:20:19.844 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:20.028 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package nvme-cli. 2023-11-28T18:20:20.044 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-28T18:20:20.087 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-28T18:20:20.423 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user cephadm....done 2023-11-28T18:20:20.542 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package pkg-config. 2023-11-28T18:20:20.558 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-11-28T18:20:20.583 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-28T18:20:20.600 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-11-28T18:20:20.853 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-28T18:20:20.868 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-28T18:20:20.911 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-28T18:20:21.193 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-28T18:20:21.314 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-28T18:20:21.328 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-28T18:20:21.397 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-28T18:20:21.461 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-28T18:20:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-28T18:20:21.708 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pastescript. 2023-11-28T18:20:21.723 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2023-11-28T18:20:21.766 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-28T18:20:21.930 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-28T18:20:22.086 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pluggy. 2023-11-28T18:20:22.100 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-28T18:20:22.144 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-28T18:20:22.231 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-28T18:20:22.348 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-28T18:20:22.430 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-psutil. 2023-11-28T18:20:22.445 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-28T18:20:22.488 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-28T18:20:22.667 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-28T18:20:22.825 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-py. 2023-11-28T18:20:22.840 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2023-11-28T18:20:22.883 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-28T18:20:23.044 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-28T18:20:23.178 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pygments. 2023-11-28T18:20:23.192 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-28T18:20:23.236 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-28T18:20:23.337 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-11-28T18:20:23.613 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-28T18:20:23.656 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-28T18:20:23.671 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-28T18:20:23.706 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-28T18:20:23.897 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-11-28T18:20:24.034 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-toml. 2023-11-28T18:20:24.048 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2023-11-28T18:20:24.092 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-28T18:20:24.157 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-28T18:20:24.400 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-28T18:20:24.412 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-pytest. 2023-11-28T18:20:24.426 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-28T18:20:24.470 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-28T18:20:24.676 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-28T18:20:24.781 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package python3-simplejson. 2023-11-28T18:20:24.795 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-28T18:20:24.839 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-28T18:20:24.943 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-ceph-argparse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:25.134 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-28T18:20:25.148 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-28T18:20:25.192 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-28T18:20:25.220 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-28T18:20:25.345 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-11-28T18:20:25.472 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-11-28T18:20:25.715 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-11-28T18:20:25.840 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-28T18:20:25.854 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package radosgw. 2023-11-28T18:20:25.868 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../122-radosgw_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:25.912 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking radosgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:26.141 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-11-28T18:20:26.515 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package rbd-fuse. 2023-11-28T18:20:26.529 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../123-rbd-fuse_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-11-28T18:20:26.557 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-11-28T18:20:26.573 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking rbd-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:26.588 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-11-28T18:20:26.644 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-28T18:20:26.770 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-28T18:20:26.866 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-28T18:20:27.120 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-28T18:20:27.172 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-28T18:20:27.221 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-28T18:20:27.273 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-28T18:20:27.461 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-28T18:20:27.541 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-28T18:20:27.800 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-28T18:20:27.872 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-28T18:20:28.109 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up zip (3.0-12build2) ... 2023-11-28T18:20:28.235 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-28T18:20:28.467 INFO:teuthology.orchestra.run.smithi193.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-28T18:20:28.521 INFO:teuthology.orchestra.run.smithi193.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-11-28T18:20:28.589 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:28.954 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-28T18:20:29.093 INFO:teuthology.orchestra.run.smithi193.stdout:Adding system user cephadm....done 2023-11-28T18:20:29.213 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-28T18:20:29.253 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-28T18:20:29.314 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-28T18:20:29.432 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-28T18:20:29.538 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-28T18:20:29.758 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-28T18:20:29.806 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-28T18:20:29.915 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-28T18:20:30.166 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-28T18:20:30.167 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-28T18:20:30.434 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-28T18:20:30.435 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-28T18:20:30.560 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-28T18:20:30.836 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-28T18:20:30.945 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-28T18:20:31.155 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-11-28T18:20:31.330 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-28T18:20:31.472 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-28T18:20:31.581 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-11-28T18:20:31.849 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-28T18:20:31.860 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-28T18:20:32.108 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-28T18:20:32.142 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-11-28T18:20:32.234 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:32.418 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-28T18:20:32.577 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-idna (3.3-1) ... 2023-11-28T18:20:32.669 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-28T18:20:32.928 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-28T18:20:33.046 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-28T18:20:33.296 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-ceph-argparse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:33.581 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-28T18:20:33.706 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-11-28T18:20:33.834 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-11-28T18:20:34.093 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-11-28T18:20:34.218 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-28T18:20:34.219 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-11-28T18:20:34.469 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-11-28T18:20:34.471 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:20:34.589 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-28T18:20:34.858 INFO:teuthology.orchestra.run.smithi193.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-11-28T18:20:34.873 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-28T18:20:34.916 INFO:teuthology.orchestra.run.smithi193.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-11-28T18:20:34.980 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-28T18:20:34.999 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-28T18:20:35.106 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-28T18:20:35.125 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-11-28T18:20:35.435 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-28T18:20:35.483 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-28T18:20:35.608 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-28T18:20:35.711 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-28T18:20:35.901 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-28T18:20:35.987 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:20:36.139 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-28T18:20:36.169 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-28T18:20:36.478 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up zip (3.0-12build2) ... 2023-11-28T18:20:36.587 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-28T18:20:36.665 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-28T18:20:36.933 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-28T18:20:37.209 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-28T18:20:37.264 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-28T18:20:37.485 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-28T18:20:37.532 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-28T18:20:37.658 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-28T18:20:37.761 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-28T18:20:37.784 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-28T18:20:38.046 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-28T18:20:38.102 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-28T18:20:38.163 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-28T18:20:38.448 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-28T18:20:38.520 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-28T18:20:38.574 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-28T18:20:38.779 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-28T18:20:38.934 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-28T18:20:39.147 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-28T18:20:39.260 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-28T18:20:39.440 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-11-28T18:20:39.527 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-11-28T18:20:39.653 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-11-28T18:20:39.767 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-28T18:20:39.773 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-28T18:20:39.852 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-28T18:20:40.106 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-28T18:20:40.118 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-28T18:20:40.232 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-28T18:20:40.377 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-28T18:20:40.478 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:40.642 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-28T18:20:40.779 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-idna (3.3-1) ... 2023-11-28T18:20:40.918 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-any (27ubuntu1) ... 2023-11-28T18:20:41.044 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-28T18:20:41.097 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-28T18:20:41.312 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:20:41.430 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-28T18:20:41.757 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-28T18:20:42.074 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-28T18:20:42.312 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-11-28T18:20:42.359 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-28T18:20:42.531 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:20:42.595 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-28T18:20:42.640 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-28T18:20:42.872 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-28T18:20:42.933 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-28T18:20:43.058 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-28T18:20:43.148 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-28T18:20:43.186 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-11-28T18:20:43.416 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-28T18:20:43.469 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-28T18:20:43.720 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-28T18:20:43.835 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-28T18:20:43.972 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:20:44.122 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-28T18:20:44.136 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-28T18:20:44.441 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-28T18:20:44.505 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-28T18:20:44.606 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librados2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:44.708 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-28T18:20:44.732 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:44.858 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-11-28T18:20:44.984 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-28T18:20:45.220 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libcephfs2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:45.252 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-28T18:20:45.346 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libradosstriper1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:45.455 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-28T18:20:45.528 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-28T18:20:45.741 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librbd1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:45.795 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-28T18:20:45.866 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-modules-core (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:45.905 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-28T18:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:46.189 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-28T18:20:46.211 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-28T18:20:46.211 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-28T18:20:46.315 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-28T18:20:46.608 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-28T18:20:46.869 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libcephfs-dev (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:46.926 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-28T18:20:46.995 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rados (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:47.121 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librgw2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:47.194 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-11-28T18:20:47.263 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephfs-top (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:47.320 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-11-28T18:20:47.365 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-28T18:20:47.437 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-28T18:20:47.509 INFO:teuthology.orchestra.run.smithi193.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-28T18:20:47.747 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-28T18:20:47.848 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-28T18:20:48.241 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-28T18:20:48.517 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up lua-any (27ubuntu1) ... 2023-11-28T18:20:48.651 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-28T18:20:48.710 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rbd (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:48.836 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up rbd-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:48.902 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-28T18:20:48.970 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:49.003 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-28T18:20:49.071 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cephfs (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:49.198 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephfs-shell (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:49.304 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-28T18:20:49.323 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:49.597 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-28T18:20:49.881 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-28T18:20:49.944 INFO:teuthology.orchestra.run.smithi007.stdout:Adding group ceph....done 2023-11-28T18:20:50.126 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-28T18:20:50.320 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user ceph....done 2023-11-28T18:20:50.403 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-28T18:20:50.498 INFO:teuthology.orchestra.run.smithi007.stdout:Setting system user ceph properties....done 2023-11-28T18:20:50.519 INFO:teuthology.orchestra.run.smithi007.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-28T18:20:50.668 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-28T18:20:50.670 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-28T18:20:50.955 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-28T18:20:51.069 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-28T18:20:51.331 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-28T18:20:51.624 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-28T18:20:51.717 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephfs-mirror (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-11-28T18:20:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-11-28T18:20:51.985 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-28T18:20:52.111 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up librados2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:52.220 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:52.322 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-11-28T18:20:52.534 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-test (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:52.661 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up radosgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:52.666 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libcephfs2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:52.792 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libradosstriper1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:52.926 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-28T18:20:53.211 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up librbd1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:53.228 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-28T18:20:53.228 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-28T18:20:53.329 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-mgr-modules-core (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-28T18:20:53.623 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-28T18:20:53.912 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-base (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:54.264 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libcephfs-dev (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:54.306 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-28T18:20:54.382 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-rados (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:54.508 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up librgw2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:54.634 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up cephfs-top (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:54.785 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-28T18:20:54.948 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mds (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:55.195 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-28T18:20:55.195 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-28T18:20:55.824 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:56.044 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-28T18:20:56.045 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-28T18:20:56.097 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-rbd (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:56.197 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up rbd-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:56.315 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-rgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:56.441 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up python3-cephfs (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:56.575 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up cephfs-shell (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:56.659 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-osd (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:56.701 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:56.998 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-28T18:20:56.998 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-28T18:20:57.255 INFO:teuthology.orchestra.run.smithi193.stdout:Adding group ceph....done 2023-11-28T18:20:57.605 INFO:teuthology.orchestra.run.smithi193.stdout:Adding system user ceph....done 2023-11-28T18:20:57.653 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:57.762 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:57.791 INFO:teuthology.orchestra.run.smithi193.stdout:Setting system user ceph properties....done 2023-11-28T18:20:57.813 INFO:teuthology.orchestra.run.smithi193.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-28T18:20:57.914 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mon (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:57.955 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-28T18:20:58.130 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-28T18:20:58.130 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-28T18:20:58.361 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-28T18:20:58.756 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:58.899 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:59.025 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-dashboard (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:59.035 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up cephfs-mirror (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:59.193 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-volume (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:59.244 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-11-28T18:20:59.245 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-11-28T18:20:59.554 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-11-28T18:20:59.663 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-28T18:20:59.815 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-28T18:20:59.819 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-test (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:20:59.946 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up radosgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:00.482 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-28T18:21:00.482 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-28T18:21:00.950 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-11-28T18:21:01.081 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-base (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:01.432 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-28T18:21:01.644 INFO:teuthology.orchestra.run.smithi007.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-28T18:21:01.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-11-28T18:21:01.726 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2023-11-28T18:21:01.896 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2023-11-28T18:21:01.897 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-28T18:21:02.061 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-28T18:21:02.062 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-28T18:21:02.062 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-28T18:21:02.062 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-28T18:21:02.062 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2023-11-28T18:21:02.063 INFO:teuthology.orchestra.run.smithi007.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-11-28T18:21:02.063 INFO:teuthology.orchestra.run.smithi007.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-11-28T18:21:02.063 INFO:teuthology.orchestra.run.smithi007.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-11-28T18:21:02.063 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-11-28T18:21:02.063 INFO:teuthology.orchestra.run.smithi007.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-11-28T18:21:02.064 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2023-11-28T18:21:02.064 INFO:teuthology.orchestra.run.smithi007.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-11-28T18:21:02.064 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-11-28T18:21:02.064 INFO:teuthology.orchestra.run.smithi007.stdout: libengine-pkcs11-openssl 2023-11-28T18:21:02.113 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2023-11-28T18:21:02.114 INFO:teuthology.orchestra.run.smithi007.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-11-28T18:21:02.114 INFO:teuthology.orchestra.run.smithi007.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-11-28T18:21:02.114 INFO:teuthology.orchestra.run.smithi007.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-11-28T18:21:02.115 INFO:teuthology.orchestra.run.smithi007.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-11-28T18:21:02.115 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-11-28T18:21:02.115 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-11-28T18:21:02.115 INFO:teuthology.orchestra.run.smithi007.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-11-28T18:21:02.115 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages will be upgraded: 2023-11-28T18:21:02.116 INFO:teuthology.orchestra.run.smithi007.stdout: libssl3 2023-11-28T18:21:02.174 INFO:teuthology.orchestra.run.smithi007.stdout:1 upgraded, 39 newly installed, 0 to remove and 263 not upgraded. 2023-11-28T18:21:02.174 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 54.6 MB of archives. 2023-11-28T18:21:02.175 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 202 MB of additional disk space will be used. 2023-11-28T18:21:02.175 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-11-28T18:21:02.301 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-11-28T18:21:02.378 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-11-28T18:21:02.378 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-mds (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:02.391 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-11-28T18:21:02.392 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-11-28T18:21:02.394 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-11-28T18:21:02.395 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-11-28T18:21:02.396 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-11-28T18:21:02.398 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-11-28T18:21:02.399 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-11-28T18:21:02.399 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-11-28T18:21:02.406 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-11-28T18:21:02.424 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-11-28T18:21:02.593 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-28T18:21:02.593 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-28T18:21:02.801 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-11-28T18:21:02.803 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-11-28T18:21:02.803 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-11-28T18:21:02.806 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-11-28T18:21:02.806 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-11-28T18:21:02.811 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-11-28T18:21:02.813 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-11-28T18:21:02.820 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-11-28T18:21:02.820 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-11-28T18:21:02.859 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-11-28T18:21:02.865 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-11-28T18:21:02.879 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-11-28T18:21:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-11-28T18:21:02.964 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-11-28T18:21:02.965 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-11-28T18:21:02.966 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-11-28T18:21:02.966 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-11-28T18:21:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-11-28T18:21:02.979 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-11-28T18:21:03.213 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-mgr (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:03.215 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-11-28T18:21:03.215 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-11-28T18:21:03.215 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-11-28T18:21:03.215 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-11-28T18:21:03.216 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-11-28T18:21:03.216 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-11-28T18:21:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-11-28T18:21:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-11-28T18:21:03.409 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-28T18:21:03.409 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-28T18:21:03.636 INFO:teuthology.orchestra.run.smithi007.stdout:Preconfiguring packages ... 2023-11-28T18:21:03.784 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 54.6 MB in 1s (48.8 MB/s) 2023-11-28T18:21:03.908 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package flex. 2023-11-28T18:21:03.984 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140566 files and directories currently installed.) 2023-11-28T18:21:03.988 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-11-28T18:21:04.023 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking flex (2.6.4-8build2) ... 2023-11-28T18:21:04.156 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-osd (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:04.508 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-28T18:21:04.509 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-28T18:21:04.593 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-11-28T18:21:04.770 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-11-28T18:21:05.170 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-11-28T18:21:05.191 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:05.325 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:05.493 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-mon (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:05.702 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-28T18:21:05.702 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-28T18:21:05.755 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package bison. 2023-11-28T18:21:05.830 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140639 files and directories currently installed.) 2023-11-28T18:21:05.834 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-11-28T18:21:05.882 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-11-28T18:21:06.291 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-11-28T18:21:06.294 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-mgr-cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:06.306 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-11-28T18:21:06.350 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-11-28T18:21:06.437 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:06.546 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-mgr-dashboard (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:06.661 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package dump. 2023-11-28T18:21:06.676 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-11-28T18:21:06.697 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ceph-volume (18.0.0-7531-g47c1c547-1jammy) ... 2023-11-28T18:21:06.745 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking dump (0.4b47-3) ... 2023-11-28T18:21:07.048 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package indent. 2023-11-28T18:21:07.063 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:07.066 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-11-28T18:21:07.098 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-11-28T18:21:07.167 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-28T18:21:07.302 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-28T18:21:07.477 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-11-28T18:21:07.492 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-11-28T18:21:07.536 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-11-28T18:21:07.865 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-11-28T18:21:07.880 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-11-28T18:21:07.923 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-11-28T18:21:08.175 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-11-28T18:21:08.190 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-11-28T18:21:08.234 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-11-28T18:21:08.503 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-11-28T18:21:08.553 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libfl2:amd64. 2023-11-28T18:21:08.569 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-11-28T18:21:08.612 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-11-28T18:21:08.873 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-11-28T18:21:08.888 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-11-28T18:21:08.931 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-11-28T18:21:09.164 INFO:teuthology.orchestra.run.smithi193.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-28T18:21:09.166 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libjson-perl. 2023-11-28T18:21:09.168 DEBUG:teuthology.orchestra.run.smithi193:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-11-28T18:21:09.183 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-11-28T18:21:09.217 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-11-28T18:21:09.242 INFO:teuthology.orchestra.run.smithi193.stdout:Reading package lists... 2023-11-28T18:21:09.409 INFO:teuthology.orchestra.run.smithi193.stdout:Building dependency tree... 2023-11-28T18:21:09.410 INFO:teuthology.orchestra.run.smithi193.stdout:Reading state information... 2023-11-28T18:21:09.553 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libllvm14:amd64. 2023-11-28T18:21:09.559 INFO:teuthology.orchestra.run.smithi193.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-11-28T18:21:09.559 INFO:teuthology.orchestra.run.smithi193.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-11-28T18:21:09.559 INFO:teuthology.orchestra.run.smithi193.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-11-28T18:21:09.560 INFO:teuthology.orchestra.run.smithi193.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-11-28T18:21:09.560 INFO:teuthology.orchestra.run.smithi193.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-11-28T18:21:09.560 INFO:teuthology.orchestra.run.smithi193.stdout:The following packages were automatically installed and are no longer required: 2023-11-28T18:21:09.560 INFO:teuthology.orchestra.run.smithi193.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-28T18:21:09.560 INFO:teuthology.orchestra.run.smithi193.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-28T18:21:09.560 INFO:teuthology.orchestra.run.smithi193.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-28T18:21:09.560 INFO:teuthology.orchestra.run.smithi193.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-28T18:21:09.560 INFO:teuthology.orchestra.run.smithi193.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-28T18:21:09.561 INFO:teuthology.orchestra.run.smithi193.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-28T18:21:09.562 INFO:teuthology.orchestra.run.smithi193.stdout:The following additional packages will be installed: 2023-11-28T18:21:09.562 INFO:teuthology.orchestra.run.smithi193.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-11-28T18:21:09.563 INFO:teuthology.orchestra.run.smithi193.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-11-28T18:21:09.563 INFO:teuthology.orchestra.run.smithi193.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-11-28T18:21:09.563 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-11-28T18:21:09.563 INFO:teuthology.orchestra.run.smithi193.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-11-28T18:21:09.563 INFO:teuthology.orchestra.run.smithi193.stdout:Suggested packages: 2023-11-28T18:21:09.563 INFO:teuthology.orchestra.run.smithi193.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-11-28T18:21:09.564 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-11-28T18:21:09.564 INFO:teuthology.orchestra.run.smithi193.stdout: libengine-pkcs11-openssl 2023-11-28T18:21:09.569 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-11-28T18:21:09.611 INFO:teuthology.orchestra.run.smithi193.stdout:The following NEW packages will be installed: 2023-11-28T18:21:09.611 INFO:teuthology.orchestra.run.smithi193.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-11-28T18:21:09.611 INFO:teuthology.orchestra.run.smithi193.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-11-28T18:21:09.612 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-11-28T18:21:09.612 INFO:teuthology.orchestra.run.smithi193.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-11-28T18:21:09.612 INFO:teuthology.orchestra.run.smithi193.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-11-28T18:21:09.613 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-11-28T18:21:09.613 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-11-28T18:21:09.613 INFO:teuthology.orchestra.run.smithi193.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-11-28T18:21:09.613 INFO:teuthology.orchestra.run.smithi193.stdout:The following packages will be upgraded: 2023-11-28T18:21:09.614 INFO:teuthology.orchestra.run.smithi193.stdout: libssl3 2023-11-28T18:21:09.798 INFO:teuthology.orchestra.run.smithi193.stdout:1 upgraded, 39 newly installed, 0 to remove and 263 not upgraded. 2023-11-28T18:21:09.799 INFO:teuthology.orchestra.run.smithi193.stdout:Need to get 54.6 MB of archives. 2023-11-28T18:21:09.799 INFO:teuthology.orchestra.run.smithi193.stdout:After this operation, 202 MB of additional disk space will be used. 2023-11-28T18:21:09.799 INFO:teuthology.orchestra.run.smithi193.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-11-28T18:21:10.295 INFO:teuthology.orchestra.run.smithi193.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-11-28T18:21:10.553 INFO:teuthology.orchestra.run.smithi193.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-11-28T18:21:10.587 INFO:teuthology.orchestra.run.smithi193.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-11-28T18:21:10.589 INFO:teuthology.orchestra.run.smithi193.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-11-28T18:21:10.594 INFO:teuthology.orchestra.run.smithi193.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-11-28T18:21:10.598 INFO:teuthology.orchestra.run.smithi193.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-11-28T18:21:10.598 INFO:teuthology.orchestra.run.smithi193.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-11-28T18:21:10.601 INFO:teuthology.orchestra.run.smithi193.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-11-28T18:21:10.603 INFO:teuthology.orchestra.run.smithi193.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-11-28T18:21:10.604 INFO:teuthology.orchestra.run.smithi193.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-11-28T18:21:10.768 INFO:teuthology.orchestra.run.smithi193.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-11-28T18:21:10.990 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-11-28T18:21:11.006 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-11-28T18:21:11.048 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-11-28T18:21:11.104 INFO:teuthology.orchestra.run.smithi193.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-11-28T18:21:11.300 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libmbim-proxy. 2023-11-28T18:21:11.317 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-11-28T18:21:11.358 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-11-28T18:21:11.636 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-11-28T18:21:11.653 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-11-28T18:21:11.695 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-11-28T18:21:12.073 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libndp0:amd64. 2023-11-28T18:21:12.097 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-11-28T18:21:12.137 INFO:teuthology.orchestra.run.smithi193.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-11-28T18:21:12.139 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-11-28T18:21:12.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-11-28T18:21:12.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-11-28T18:21:12.165 INFO:teuthology.orchestra.run.smithi193.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-11-28T18:21:12.166 INFO:teuthology.orchestra.run.smithi193.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-11-28T18:21:12.170 INFO:teuthology.orchestra.run.smithi193.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-11-28T18:21:12.172 INFO:teuthology.orchestra.run.smithi193.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-11-28T18:21:12.201 INFO:teuthology.orchestra.run.smithi193.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-11-28T18:21:12.201 INFO:teuthology.orchestra.run.smithi193.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-11-28T18:21:12.419 INFO:teuthology.orchestra.run.smithi193.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-11-28T18:21:12.492 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libnm0:amd64. 2023-11-28T18:21:12.501 INFO:teuthology.orchestra.run.smithi193.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-11-28T18:21:12.508 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-11-28T18:21:12.567 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-11-28T18:21:12.862 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libpq5:amd64. 2023-11-28T18:21:12.878 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:12.912 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:12.982 INFO:teuthology.orchestra.run.smithi193.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-11-28T18:21:13.115 INFO:teuthology.orchestra.run.smithi193.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-11-28T18:21:13.126 INFO:teuthology.orchestra.run.smithi193.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-11-28T18:21:13.126 INFO:teuthology.orchestra.run.smithi193.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-11-28T18:21:13.127 INFO:teuthology.orchestra.run.smithi193.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-11-28T18:21:13.128 INFO:teuthology.orchestra.run.smithi193.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-11-28T18:21:13.162 INFO:teuthology.orchestra.run.smithi193.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-11-28T18:21:13.165 INFO:teuthology.orchestra.run.smithi193.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-11-28T18:21:13.265 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-11-28T18:21:13.281 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:13.323 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-11-28T18:21:13.600 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqmi-proxy. 2023-11-28T18:21:13.612 INFO:teuthology.orchestra.run.smithi193.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-11-28T18:21:13.616 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:13.659 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-11-28T18:21:13.776 INFO:teuthology.orchestra.run.smithi193.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-11-28T18:21:13.859 INFO:teuthology.orchestra.run.smithi193.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-11-28T18:21:13.859 INFO:teuthology.orchestra.run.smithi193.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-11-28T18:21:13.894 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-11-28T18:21:13.910 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-11-28T18:21:13.944 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-11-28T18:21:13.945 INFO:teuthology.orchestra.run.smithi193.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-11-28T18:21:13.952 INFO:teuthology.orchestra.run.smithi193.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-11-28T18:21:14.418 INFO:teuthology.orchestra.run.smithi193.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-11-28T18:21:14.418 INFO:teuthology.orchestra.run.smithi193.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-11-28T18:21:14.473 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-11-28T18:21:14.489 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-11-28T18:21:14.556 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-11-28T18:21:14.810 INFO:teuthology.orchestra.run.smithi193.stdout:Preconfiguring packages ... 2023-11-28T18:21:14.909 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package modemmanager. 2023-11-28T18:21:14.926 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-11-28T18:21:14.968 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-11-28T18:21:14.974 INFO:teuthology.orchestra.run.smithi193.stdout:Fetched 54.6 MB in 5s (11.4 MB/s) 2023-11-28T18:21:15.117 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package flex. 2023-11-28T18:21:15.190 INFO:teuthology.orchestra.run.smithi193.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140566 files and directories currently installed.) 2023-11-28T18:21:15.194 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-11-28T18:21:15.241 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking flex (2.6.4-8build2) ... 2023-11-28T18:21:15.591 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package network-manager. 2023-11-28T18:21:15.607 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-11-28T18:21:15.658 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-11-28T18:21:15.735 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-11-28T18:21:15.880 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-11-28T18:21:16.248 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ppp. 2023-11-28T18:21:16.264 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-11-28T18:21:16.278 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-11-28T18:21:16.620 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-11-28T18:21:16.821 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package bison. 2023-11-28T18:21:16.893 INFO:teuthology.orchestra.run.smithi193.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140639 files and directories currently installed.) 2023-11-28T18:21:16.897 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-11-28T18:21:16.940 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-11-28T18:21:16.986 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package pptp-linux. 2023-11-28T18:21:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-11-28T18:21:17.036 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-11-28T18:21:17.288 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package network-manager-pptp. 2023-11-28T18:21:17.304 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-11-28T18:21:17.347 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-11-28T18:21:17.391 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-11-28T18:21:17.406 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-11-28T18:21:17.441 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-11-28T18:21:17.650 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-client-common. 2023-11-28T18:21:17.665 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-11-28T18:21:17.726 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-client-common (238) ... 2023-11-28T18:21:17.744 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package dump. 2023-11-28T18:21:17.759 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-11-28T18:21:17.802 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking dump (0.4b47-3) ... 2023-11-28T18:21:18.013 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-client-14. 2023-11-28T18:21:18.028 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:18.063 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:18.080 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package indent. 2023-11-28T18:21:18.095 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:18.139 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-11-28T18:21:18.400 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-11-28T18:21:18.415 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-11-28T18:21:18.458 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-11-28T18:21:18.601 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-common. 2023-11-28T18:21:18.616 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-11-28T18:21:18.678 INFO:teuthology.orchestra.run.smithi007.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-11-28T18:21:18.709 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-common (238) ... 2023-11-28T18:21:18.778 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-11-28T18:21:18.794 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-11-28T18:21:18.828 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-11-28T18:21:19.081 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-11-28T18:21:19.097 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-11-28T18:21:19.139 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-11-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-14. 2023-11-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:19.475 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libfl2:amd64. 2023-11-28T18:21:19.491 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-11-28T18:21:19.534 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-11-28T18:21:19.811 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-11-28T18:21:19.827 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-11-28T18:21:19.869 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-11-28T18:21:20.097 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libjson-perl. 2023-11-28T18:21:20.113 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-11-28T18:21:20.147 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-11-28T18:21:20.329 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql. 2023-11-28T18:21:20.344 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-11-28T18:21:20.387 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql (14+238) ... 2023-11-28T18:21:20.433 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libllvm14:amd64. 2023-11-28T18:21:20.449 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-11-28T18:21:20.491 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-11-28T18:21:20.614 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-client. 2023-11-28T18:21:20.717 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-11-28T18:21:20.717 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-client (14+238) ... 2023-11-28T18:21:21.069 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-contrib. 2023-11-28T18:21:21.084 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-11-28T18:21:21.127 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-contrib (14+238) ... 2023-11-28T18:21:21.446 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package tcl8.6. 2023-11-28T18:21:21.462 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-11-28T18:21:21.504 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-11-28T18:21:21.766 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package tcl. 2023-11-28T18:21:21.782 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-11-28T18:21:21.821 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-11-28T18:21:21.877 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-11-28T18:21:21.893 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-11-28T18:21:21.935 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-11-28T18:21:22.201 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package wpasupplicant. 2023-11-28T18:21:22.218 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-11-28T18:21:22.221 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libmbim-proxy. 2023-11-28T18:21:22.237 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-11-28T18:21:22.260 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-11-28T18:21:22.279 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-11-28T18:21:22.565 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-11-28T18:21:22.581 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-11-28T18:21:22.615 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-11-28T18:21:22.615 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-11-28T18:21:22.631 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-11-28T18:21:22.673 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-11-28T18:21:22.959 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libndp0:amd64. 2023-11-28T18:21:22.975 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-11-28T18:21:23.018 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-11-28T18:21:23.354 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libnm0:amd64. 2023-11-28T18:21:23.369 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-11-28T18:21:23.431 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-11-28T18:21:23.587 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package usb-modeswitch. 2023-11-28T18:21:23.604 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-11-28T18:21:23.654 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-11-28T18:21:23.739 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libpq5:amd64. 2023-11-28T18:21:23.755 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:23.798 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:23.940 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-client-common (238) ... 2023-11-28T18:21:24.126 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-11-28T18:21:24.142 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:24.160 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up flex (2.6.4-8build2) ... 2023-11-28T18:21:24.184 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-11-28T18:21:24.286 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-11-28T18:21:24.395 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:24.497 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-11-28T18:21:24.520 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libqmi-proxy. 2023-11-28T18:21:24.536 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:24.578 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-11-28T18:21:24.622 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-11-28T18:21:24.773 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:24.822 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-11-28T18:21:24.837 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-11-28T18:21:24.880 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-11-28T18:21:25.317 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-11-28T18:21:25.332 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-11-28T18:21:25.366 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-11-28T18:21:25.644 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package modemmanager. 2023-11-28T18:21:25.660 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-11-28T18:21:25.711 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-11-28T18:21:25.797 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-11-28T18:21:25.888 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-11-28T18:21:26.299 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-11-28T18:21:26.299 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-11-28T18:21:26.350 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package network-manager. 2023-11-28T18:21:26.366 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-11-28T18:21:26.417 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-11-28T18:21:26.999 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-11-28T18:21:27.049 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package ppp. 2023-11-28T18:21:27.065 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-11-28T18:21:27.125 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-11-28T18:21:27.259 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-11-28T18:21:27.415 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-11-28T18:21:27.419 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-11-28T18:21:27.528 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-11-28T18:21:27.646 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-11-28T18:21:27.695 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package pptp-linux. 2023-11-28T18:21:27.709 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-11-28T18:21:27.753 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-11-28T18:21:27.755 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-11-28T18:21:27.947 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package network-manager-pptp. 2023-11-28T18:21:27.961 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-11-28T18:21:27.997 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-11-28T18:21:28.342 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package postgresql-client-common. 2023-11-28T18:21:28.356 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-11-28T18:21:28.400 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking postgresql-client-common (238) ... 2023-11-28T18:21:28.444 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-11-28T18:21:28.562 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up dump (0.4b47-3) ... 2023-11-28T18:21:28.666 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-11-28T18:21:28.703 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package postgresql-client-14. 2023-11-28T18:21:28.718 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:28.722 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-11-28T18:21:28.762 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:28.848 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-11-28T18:21:28.974 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up tcl (8.6.11+1build2) ... 2023-11-28T18:21:29.108 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libjson-perl (4.04000-1) ... 2023-11-28T18:21:29.234 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-11-28T18:21:29.324 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-11-28T18:21:29.366 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package postgresql-common. 2023-11-28T18:21:29.381 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-11-28T18:21:29.394 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-11-28T18:21:29.451 INFO:teuthology.orchestra.run.smithi193.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-11-28T18:21:29.482 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking postgresql-common (238) ... 2023-11-28T18:21:29.512 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-11-28T18:21:29.663 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-11-28T18:21:29.781 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-11-28T18:21:29.932 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package postgresql-14. 2023-11-28T18:21:29.940 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-11-28T18:21:29.946 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-11-28T18:21:29.984 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:30.109 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-client (14+238) ... 2023-11-28T18:21:30.235 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-11-28T18:21:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:30.841 INFO:teuthology.orchestra.run.smithi007.stdout:The following network interfaces were found in /etc/network/interfaces 2023-11-28T18:21:30.841 INFO:teuthology.orchestra.run.smithi007.stdout:which means they are currently configured by ifupdown: 2023-11-28T18:21:30.841 INFO:teuthology.orchestra.run.smithi007.stdout:- ens1f0 2023-11-28T18:21:30.841 INFO:teuthology.orchestra.run.smithi007.stdout:If you want to manage those interfaces with NetworkManager instead 2023-11-28T18:21:30.841 INFO:teuthology.orchestra.run.smithi007.stdout:remove their configuration from /etc/network/interfaces. 2023-11-28T18:21:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:30.983 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-11-28T18:21:31.176 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package postgresql. 2023-11-28T18:21:31.191 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-11-28T18:21:31.234 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking postgresql (14+238) ... 2023-11-28T18:21:31.393 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-11-28T18:21:31.470 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package postgresql-client. 2023-11-28T18:21:31.484 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-11-28T18:21:31.520 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking postgresql-client (14+238) ... 2023-11-28T18:21:31.738 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package postgresql-contrib. 2023-11-28T18:21:31.753 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-11-28T18:21:31.796 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking postgresql-contrib (14+238) ... 2023-11-28T18:21:31.824 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-11-28T18:21:32.141 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package tcl8.6. 2023-11-28T18:21:32.155 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-11-28T18:21:32.199 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-11-28T18:21:32.493 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package tcl. 2023-11-28T18:21:32.508 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-11-28T18:21:32.556 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-11-28T18:21:32.622 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-11-28T18:21:32.757 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-11-28T18:21:32.879 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package wpasupplicant. 2023-11-28T18:21:32.883 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-11-28T18:21:32.895 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-11-28T18:21:33.009 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-common (238) ... 2023-11-28T18:21:33.279 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-11-28T18:21:33.602 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-11-28T18:21:33.618 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-11-28T18:21:33.661 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-11-28T18:21:34.039 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package usb-modeswitch. 2023-11-28T18:21:34.055 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-11-28T18:21:34.105 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-11-28T18:21:34.243 INFO:teuthology.orchestra.run.smithi007.stdout:Adding user postgres to group ssl-cert 2023-11-28T18:21:34.373 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up postgresql-client-common (238) ... 2023-11-28T18:21:34.513 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:34.513 INFO:teuthology.orchestra.run.smithi007.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-11-28T18:21:34.586 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up flex (2.6.4-8build2) ... 2023-11-28T18:21:34.712 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-11-28T18:21:34.795 INFO:teuthology.orchestra.run.smithi007.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-11-28T18:21:34.795 INFO:teuthology.orchestra.run.smithi007.stdout:Removing obsolete dictionary files: 2023-11-28T18:21:34.838 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:34.964 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-11-28T18:21:35.090 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-11-28T18:21:35.216 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:35.565 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-11-28T18:21:36.045 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:36.211 INFO:teuthology.orchestra.run.smithi193.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-11-28T18:21:36.320 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-11-28T18:21:36.477 INFO:teuthology.orchestra.run.smithi007.stdout:Creating new PostgreSQL cluster 14/main ... 2023-11-28T18:21:36.500 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout:The files belonging to this database system will be owned by user "postgres". 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout:This user must also own the server process. 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout:The database cluster will be initialized with locale "en_US". 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout:The default text search configuration will be set to "english". 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout:Data page checksums are disabled. 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:36.553 INFO:teuthology.orchestra.run.smithi007.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-11-28T18:21:36.554 INFO:teuthology.orchestra.run.smithi007.stdout:creating subdirectories ... ok 2023-11-28T18:21:36.554 INFO:teuthology.orchestra.run.smithi007.stdout:selecting dynamic shared memory implementation ... posix 2023-11-28T18:21:36.579 INFO:teuthology.orchestra.run.smithi007.stdout:selecting default max_connections ... 100 2023-11-28T18:21:36.617 INFO:teuthology.orchestra.run.smithi007.stdout:selecting default shared_buffers ... 128MB 2023-11-28T18:21:36.631 INFO:teuthology.orchestra.run.smithi007.stdout:selecting default time zone ... Etc/UTC 2023-11-28T18:21:36.632 INFO:teuthology.orchestra.run.smithi007.stdout:creating configuration files ... ok 2023-11-28T18:21:36.700 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-11-28T18:21:36.700 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-11-28T18:21:36.810 INFO:teuthology.orchestra.run.smithi007.stdout:running bootstrap script ... ok 2023-11-28T18:21:37.373 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-11-28T18:21:37.455 INFO:teuthology.orchestra.run.smithi007.stdout:performing post-bootstrap initialization ... ok 2023-11-28T18:21:37.507 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-11-28T18:21:37.617 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-11-28T18:21:37.718 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-11-28T18:21:37.819 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-11-28T18:21:37.945 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-11-28T18:21:38.041 INFO:teuthology.orchestra.run.smithi007.stdout:syncing data to disk ... ok 2023-11-28T18:21:38.096 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-11-28T18:21:38.577 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-11-28T18:21:38.835 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-11-28T18:21:38.961 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up dump (0.4b47-3) ... 2023-11-28T18:21:39.056 INFO:teuthology.orchestra.run.smithi193.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-11-28T18:21:39.121 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-11-28T18:21:39.280 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-11-28T18:21:39.382 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up tcl (8.6.11+1build2) ... 2023-11-28T18:21:39.482 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libjson-perl (4.04000-1) ... 2023-11-28T18:21:39.600 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-11-28T18:21:39.690 INFO:teuthology.orchestra.run.smithi193.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-11-28T18:21:39.760 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-11-28T18:21:39.869 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-11-28T18:21:40.046 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-11-28T18:21:40.172 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-11-28T18:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-11-28T18:21:40.427 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up postgresql-client (14+238) ... 2023-11-28T18:21:40.550 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-11-28T18:21:41.111 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:41.111 INFO:teuthology.orchestra.run.smithi193.stdout:The following network interfaces were found in /etc/network/interfaces 2023-11-28T18:21:41.111 INFO:teuthology.orchestra.run.smithi193.stdout:which means they are currently configured by ifupdown: 2023-11-28T18:21:41.111 INFO:teuthology.orchestra.run.smithi193.stdout:- enp3s0f1 2023-11-28T18:21:41.111 INFO:teuthology.orchestra.run.smithi193.stdout:If you want to manage those interfaces with NetworkManager instead 2023-11-28T18:21:41.111 INFO:teuthology.orchestra.run.smithi193.stdout:remove their configuration from /etc/network/interfaces. 2023-11-28T18:21:41.111 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:41.243 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-11-28T18:21:41.552 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-11-28T18:21:41.630 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-11-28T18:21:41.653 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-contrib (14+238) ... 2023-11-28T18:21:41.771 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql (14+238) ... 2023-11-28T18:21:41.897 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-11-28T18:21:42.023 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-11-28T18:21:42.039 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-11-28T18:21:42.294 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-11-28T18:21:42.294 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-11-28T18:21:42.728 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-11-28T18:21:42.913 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-11-28T18:21:43.026 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-11-28T18:21:43.039 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-11-28T18:21:43.165 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up postgresql-common (238) ... 2023-11-28T18:21:43.361 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-28T18:21:43.513 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-28T18:21:44.283 INFO:teuthology.orchestra.run.smithi193.stdout:Adding user postgres to group ssl-cert 2023-11-28T18:21:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:44.538 INFO:teuthology.orchestra.run.smithi193.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-11-28T18:21:44.622 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-11-28T18:21:44.714 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-11-28T18:21:44.794 INFO:teuthology.orchestra.run.smithi193.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-11-28T18:21:44.794 INFO:teuthology.orchestra.run.smithi193.stdout:Removing obsolete dictionary files: 2023-11-28T18:21:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-28T18:21:45.250 DEBUG:teuthology.parallel:result is None 2023-11-28T18:21:45.517 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-11-28T18:21:46.126 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-11-28T18:21:46.464 INFO:teuthology.orchestra.run.smithi193.stdout:Creating new PostgreSQL cluster 14/main ... 2023-11-28T18:21:46.481 INFO:teuthology.orchestra.run.smithi193.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout:The files belonging to this database system will be owned by user "postgres". 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout:This user must also own the server process. 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout:The database cluster will be initialized with locale "en_US". 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout:The default text search configuration will be set to "english". 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout:Data page checksums are disabled. 2023-11-28T18:21:46.548 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:46.549 INFO:teuthology.orchestra.run.smithi193.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-11-28T18:21:46.549 INFO:teuthology.orchestra.run.smithi193.stdout:creating subdirectories ... ok 2023-11-28T18:21:46.549 INFO:teuthology.orchestra.run.smithi193.stdout:selecting dynamic shared memory implementation ... posix 2023-11-28T18:21:46.573 INFO:teuthology.orchestra.run.smithi193.stdout:selecting default max_connections ... 100 2023-11-28T18:21:46.609 INFO:teuthology.orchestra.run.smithi193.stdout:selecting default shared_buffers ... 128MB 2023-11-28T18:21:46.624 INFO:teuthology.orchestra.run.smithi193.stdout:selecting default time zone ... Etc/UTC 2023-11-28T18:21:46.625 INFO:teuthology.orchestra.run.smithi193.stdout:creating configuration files ... ok 2023-11-28T18:21:46.795 INFO:teuthology.orchestra.run.smithi193.stdout:running bootstrap script ... ok 2023-11-28T18:21:47.435 INFO:teuthology.orchestra.run.smithi193.stdout:performing post-bootstrap initialization ... ok 2023-11-28T18:21:47.978 INFO:teuthology.orchestra.run.smithi193.stdout:syncing data to disk ... ok 2023-11-28T18:21:48.501 INFO:teuthology.orchestra.run.smithi193.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-11-28T18:21:51.473 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-11-28T18:21:51.599 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up postgresql-contrib (14+238) ... 2023-11-28T18:21:51.725 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up postgresql (14+238) ... 2023-11-28T18:21:51.852 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-11-28T18:21:51.969 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-11-28T18:21:52.212 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-11-28T18:21:52.213 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-11-28T18:21:52.938 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-11-28T18:21:53.240 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-28T18:21:53.391 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-28T18:21:54.786 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-11-28T18:21:54.903 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-11-28T18:21:55.441 INFO:teuthology.orchestra.run.smithi193.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-28T18:21:55.445 DEBUG:teuthology.parallel:result is None 2023-11-28T18:21:55.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T18:21:55.592 DEBUG:teuthology.orchestra.run.smithi007:> dpkg-query -W -f '${Version}' ceph 2023-11-28T18:21:55.618 INFO:teuthology.orchestra.run.smithi007.stdout:18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:21:55.619 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:21:55.619 INFO:teuthology.task.install:The correct ceph version 18.0.0-7531-g47c1c547-1jammy is installed. 2023-11-28T18:21:55.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T18:21:55.834 DEBUG:teuthology.orchestra.run.smithi193:> dpkg-query -W -f '${Version}' ceph 2023-11-28T18:21:55.856 INFO:teuthology.orchestra.run.smithi193.stdout:18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:21:55.856 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7531-g47c1c547-1jammy 2023-11-28T18:21:55.857 INFO:teuthology.task.install:The correct ceph version 18.0.0-7531-g47c1c547-1jammy is installed. 2023-11-28T18:21:55.857 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-28T18:21:55.857 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:55.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T18:21:55.880 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:21:55.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T18:21:55.918 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-28T18:21:55.919 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:55.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T18:21:55.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T18:21:56.007 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:21:56.007 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T18:21:56.028 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T18:21:56.094 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-28T18:21:56.094 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:56.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T18:21:56.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T18:21:56.179 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:21:56.179 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T18:21:56.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T18:21:56.261 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-28T18:21:56.262 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:56.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T18:21:56.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T18:21:56.351 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:21:56.351 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T18:21:56.371 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T18:21:56.435 INFO:teuthology.run_tasks:Running task ceph... 2023-11-28T18:21:56.513 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-28T18:21:56.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2023-11-28T18:21:56.519 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /var/log/ceph 2023-11-28T18:21:56.539 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-28T18:21:56.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T18:21:56.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T18:21:56.604 INFO:tasks.ceph:Creating extra log directories... 2023-11-28T18:21:56.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T18:21:56.653 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T18:21:56.680 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-28T18:21:56.680 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-28T18:21:56.680 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-28T18:21:56.681 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:56.681 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T18:21:56.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T18:21:56.827 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:21:56.828 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T18:21:56.836 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T18:21:56.931 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-28T18:21:56.931 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-28T18:21:56.931 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457', 'branch': 'wip-vshankar-testing-20231127.102654', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7469457', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20231127.102654'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20231127.102654', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1098, 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'sleep_before_teardown': 0, 'subset': '854/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20231127.102654', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ6dCvqETJOdN7f1Epo6cniZht7c5goNuYAOZUBrh9thfzQcM2F3QKi5mYaL+zO29hg6o7/fjXX4Y1jFJrsaJyc=', 'smithi193.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBgCbSH1Nj/42ryCnkWgSNJF0gbI6Oe+/+KDqHSBCnArI3dbGEbt4LSUv3/4QJAvpTe6bSYl7wBvSQEb1HWidsk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '5ae09a5cc7064ca2bf994df19f3006b79794789a', 'timestamp': '2023-11-28_06:23:42', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-28T18:21:56.932 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T18:21:56.936 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T18:21:56.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T18:21:56.989 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T18:21:57.010 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:57.010 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T18:21:57.059 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T18:21:57.059 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2023-11-28T18:21:57.111 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T18:21:57.111 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T18:21:57.111 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-11-28T18:21:57.111 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:21:57.111 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-28 18:15:25.304569820 +0000 2023-11-28T18:21:57.111 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-28 18:15:24.884573572 +0000 2023-11-28T18:21:57.112 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-28 18:15:24.884573572 +0000 2023-11-28T18:21:57.112 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-28 18:15:24.884573572 +0000 2023-11-28T18:21:57.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T18:21:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-28T18:21:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-28T18:21:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000254583 s, 2.0 MB/s 2023-11-28T18:21:57.175 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T18:21:57.229 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2023-11-28T18:21:57.279 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T18:21:57.279 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T18:21:57.279 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 919 Links: 1 2023-11-28T18:21:57.279 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:21:57.279 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-28 18:15:25.960563961 +0000 2023-11-28T18:21:57.279 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-28 18:15:25.556567570 +0000 2023-11-28T18:21:57.279 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-28 18:15:25.556567570 +0000 2023-11-28T18:21:57.280 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-28 18:15:25.556567570 +0000 2023-11-28T18:21:57.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T18:21:57.341 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-28T18:21:57.341 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-28T18:21:57.341 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000255096 s, 2.0 MB/s 2023-11-28T18:21:57.343 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T18:21:57.396 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2023-11-28T18:21:57.447 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T18:21:57.447 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T18:21:57.447 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 938 Links: 1 2023-11-28T18:21:57.447 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:21:57.447 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-28 18:15:26.648557816 +0000 2023-11-28T18:21:57.447 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-28 18:15:26.212561710 +0000 2023-11-28T18:21:57.447 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-28 18:15:26.212561710 +0000 2023-11-28T18:21:57.447 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-28 18:15:26.212561710 +0000 2023-11-28T18:21:57.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T18:21:57.513 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-28T18:21:57.514 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-28T18:21:57.514 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000257954 s, 2.0 MB/s 2023-11-28T18:21:57.515 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T18:21:57.569 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2023-11-28T18:21:57.619 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T18:21:57.619 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T18:21:57.619 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 952 Links: 1 2023-11-28T18:21:57.619 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:21:57.619 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-28 18:15:26.916555421 +0000 2023-11-28T18:21:57.619 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-28 18:15:26.912555458 +0000 2023-11-28T18:21:57.619 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-28 18:15:26.912555458 +0000 2023-11-28T18:21:57.619 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-28 18:15:26.912555458 +0000 2023-11-28T18:21:57.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T18:21:57.681 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-28T18:21:57.682 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-28T18:21:57.682 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000266196 s, 1.9 MB/s 2023-11-28T18:21:57.684 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T18:21:57.737 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T18:21:57.737 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:21:57.737 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T18:21:57.747 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T18:21:57.747 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2023-11-28T18:21:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T18:21:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T18:21:57.804 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 904 Links: 1 2023-11-28T18:21:57.804 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:21:57.804 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-28 18:15:21.597087209 +0000 2023-11-28T18:21:57.804 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-28 18:15:21.201090747 +0000 2023-11-28T18:21:57.804 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-28 18:15:21.201090747 +0000 2023-11-28T18:21:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2023-11-28 18:15:21.201090747 +0000 2023-11-28T18:21:57.804 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T18:21:57.869 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-28T18:21:57.870 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-28T18:21:57.870 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000389638 s, 1.3 MB/s 2023-11-28T18:21:57.871 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T18:21:57.926 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2023-11-28T18:21:57.979 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T18:21:57.979 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T18:21:57.979 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 918 Links: 1 2023-11-28T18:21:57.979 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:21:57.979 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-28 18:15:22.293080993 +0000 2023-11-28T18:21:57.979 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-28 18:15:21.869084780 +0000 2023-11-28T18:21:57.980 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-28 18:15:21.869084780 +0000 2023-11-28T18:21:57.980 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2023-11-28 18:15:21.869084780 +0000 2023-11-28T18:21:57.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T18:21:58.042 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-28T18:21:58.042 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-28T18:21:58.042 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000408957 s, 1.3 MB/s 2023-11-28T18:21:58.044 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T18:21:58.101 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2023-11-28T18:21:58.151 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T18:21:58.151 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T18:21:58.151 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 935 Links: 1 2023-11-28T18:21:58.151 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:21:58.151 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-28 18:15:22.933075276 +0000 2023-11-28T18:21:58.151 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-28 18:15:22.561078599 +0000 2023-11-28T18:21:58.151 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-28 18:15:22.561078599 +0000 2023-11-28T18:21:58.151 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2023-11-28 18:15:22.561078599 +0000 2023-11-28T18:21:58.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T18:21:58.214 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-28T18:21:58.214 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-28T18:21:58.214 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.00038922 s, 1.3 MB/s 2023-11-28T18:21:58.216 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T18:21:58.273 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2023-11-28T18:21:58.323 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T18:21:58.323 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T18:21:58.323 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 954 Links: 1 2023-11-28T18:21:58.323 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:21:58.324 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-28 18:15:23.677068631 +0000 2023-11-28T18:21:58.324 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-28 18:15:23.269072274 +0000 2023-11-28T18:21:58.325 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-28 18:15:23.269072274 +0000 2023-11-28T18:21:58.326 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2023-11-28 18:15:23.269072274 +0000 2023-11-28T18:21:58.327 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T18:21:58.390 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-28T18:21:58.390 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-28T18:21:58.390 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000394681 s, 1.3 MB/s 2023-11-28T18:21:58.392 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T18:21:58.445 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T18:21:58.445 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T18:21:58.445 INFO:tasks.ceph:Generating config... 2023-11-28T18:21:58.446 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-28T18:21:58.446 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-11-28T18:21:58.446 INFO:tasks.ceph:[client] debug client = 20 2023-11-28T18:21:58.447 INFO:tasks.ceph:[client] debug ms = 1 2023-11-28T18:21:58.447 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-28T18:21:58.447 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-28T18:21:58.447 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-28T18:21:58.448 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-28T18:21:58.449 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-28T18:21:58.449 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-28T18:21:58.449 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-28T18:21:58.449 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-28T18:21:58.449 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-28T18:21:58.449 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-28T18:21:58.449 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-28T18:21:58.449 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-28T18:21:58.449 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-11-28T18:21:58.450 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-28T18:21:58.450 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-28T18:21:58.450 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-28T18:21:58.450 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] filestore flush min = 0 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-28T18:21:58.451 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-11-28T18:21:58.452 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-11-28T18:21:58.452 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-11-28T18:21:58.452 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-11-28T18:21:58.452 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-11-28T18:21:58.452 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-28T18:21:58.452 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-11-28T18:21:58.452 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-28T18:21:58.452 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-28T18:21:58.453 INFO:tasks.ceph:Setting up mon.a... 2023-11-28T18:21:58.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-28T18:21:58.501 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2023-11-28T18:21:58.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-28T18:21:58.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T18:21:58.590 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7'), ('mon.b', '172.21.15.193'), ('mon.c', '[v2:172.21.15.193:3301,v1:172.21.15.193:6790]')] 2023-11-28T18:21:58.590 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7,172.21.15.193,[v2:172.21.15.193:3301,v1:172.21.15.193:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-28T18:21:58.591 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:58.591 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T18:21:58.642 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --add b 172.21.15.193 --addv c '[v2:172.21.15.193:3301,v1:172.21.15.193:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:21:58.739 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:21:58.739 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid b4396e54-0cbb-426d-8760-05a802ba48a2 2023-11-28T18:21:58.739 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = pacific 2023-11-28T18:21:58.739 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2023-11-28T18:21:58.739 INFO:teuthology.orchestra.run.smithi007.stdout:fsid b4396e54-0cbb-426d-8760-05a802ba48a2 2023-11-28T18:21:58.739 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2023-11-28T18:21:58.739676+0000 2023-11-28T18:21:58.739 INFO:teuthology.orchestra.run.smithi007.stdout:created 2023-11-28T18:21:58.739676+0000 2023-11-28T18:21:58.739 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 16 (pacific) 2023-11-28T18:21:58.740 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2023-11-28T18:21:58.740 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-28T18:21:58.740 INFO:teuthology.orchestra.run.smithi007.stdout:1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.b 2023-11-28T18:21:58.740 INFO:teuthology.orchestra.run.smithi007.stdout:2: [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] mon.c 2023-11-28T18:21:58.740 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-28T18:21:58.742 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T18:21:58.750 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b4396e54-0cbb-426d-8760-05a802ba48a2... 2023-11-28T18:21:58.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T18:21:58.798 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T18:21:58.839 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2023-11-28T18:21:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2023-11-28T18:21:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T18:21:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2023-11-28T18:21:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2023-11-28T18:21:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2023-11-28T18:21:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2023-11-28T18:21:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "172.21.15.7,172.21.15.193,[v2:172.21.15.193:3301,v1:172.21.15.193:6790]" 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = b4396e54-0cbb-426d-8760-05a802ba48a2 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2023-11-28T18:21:58.844 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.845 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: bdev async discard = True 2023-11-28T18:21:58.846 INFO:teuthology.orchestra.run.smithi007.stdout: bdev enable discard = True 2023-11-28T18:21:58.847 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore allocator = bitmap 2023-11-28T18:21:58.847 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2023-11-28T18:21:58.847 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2023-11-28T18:21:58.847 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1/20 2023-11-28T18:21:58.847 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1/20 2023-11-28T18:21:58.847 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-28T18:21:58.847 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2023-11-28T18:21:58.847 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 4/10 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: filestore flush min = 0 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay max = 1 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay probability = 0.005 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay type = client mds 2023-11-28T18:21:58.849 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 2500 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: osd blocked scrub grace period = 3600 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: osd max markdown count = 1000 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: osd op complaint time = 180 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-11-28T18:21:58.850 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2023-11-28T18:21:58.851 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-11-28T18:21:58.852 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2023-11-28T18:21:58.852 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2023-11-28T18:21:58.852 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-11-28T18:21:58.852 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.852 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T18:21:58.853 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T18:21:58.853 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T18:21:58.853 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.853 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T18:21:58.853 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T18:21:58.853 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T18:21:58.854 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.854 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2023-11-28T18:21:58.854 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2023-11-28T18:21:58.854 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.854 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2023-11-28T18:21:58.855 INFO:teuthology.orchestra.run.smithi007.stdout: mon op complaint time = 120 2023-11-28T18:21:58.855 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:21:58.855 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2023-11-28T18:21:58.855 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2023-11-28T18:21:58.855 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2023-11-28T18:21:58.856 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2023-11-28T18:21:58.856 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T18:21:58.856 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T18:21:58.856 INFO:teuthology.orchestra.run.smithi007.stdout: client mount timeout = 600 2023-11-28T18:21:58.856 INFO:teuthology.orchestra.run.smithi007.stdout: client_shutdown_timeout = 120 2023-11-28T18:21:58.856 INFO:teuthology.orchestra.run.smithi007.stdout: debug client = 20 2023-11-28T18:21:58.857 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-28T18:21:58.857 INFO:teuthology.orchestra.run.smithi007.stdout: rados mon op timeout = 900 2023-11-28T18:21:58.857 INFO:teuthology.orchestra.run.smithi007.stdout: rados osd op timeout = 900 2023-11-28T18:21:58.857 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2023-11-28T18:21:58.857 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2023-11-28T18:21:58.858 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2023-11-28T18:21:58.859 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.a] 2023-11-28T18:21:58.859 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.c] 2023-11-28T18:21:58.859 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.e] 2023-11-28T18:21:58.859 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.b] 2023-11-28T18:21:58.859 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.d] 2023-11-28T18:21:58.859 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.f] 2023-11-28T18:21:58.859 INFO:teuthology.orchestra.run.smithi007.stdout:[mds] 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds = 20 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds balancer = 20 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds bal fragment size max = 10000 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds bal merge size = 5 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds bal split bits = 3 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds bal split size = 100 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds debug frag = True 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds debug scatterstat = True 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds oft prefetch dirfrags = True 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds op complaint time = 180 2023-11-28T18:21:58.860 INFO:teuthology.orchestra.run.smithi007.stdout: mds verify scatter = True 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: mds_dir_prefetch = False 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: mds_sleep_rank_change = 5000000.0 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay max = 1 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay probability = 0.005 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay type = client mds osd 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 2500 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: osd op complaint time = 180 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: rados mon op timeout = 900 2023-11-28T18:21:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: rados osd op timeout = 900 2023-11-28T18:21:58.862 INFO:teuthology.orchestra.run.smithi193.stdout:[global] 2023-11-28T18:21:58.862 INFO:teuthology.orchestra.run.smithi193.stdout: chdir = "" 2023-11-28T18:21:58.862 INFO:teuthology.orchestra.run.smithi193.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T18:21:58.862 INFO:teuthology.orchestra.run.smithi193.stdout: auth supported = cephx 2023-11-28T18:21:58.862 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.862 INFO:teuthology.orchestra.run.smithi193.stdout: filestore xattr use omap = true 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: mon clock drift allowed = 1.000 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: osd crush chooseleaf type = 0 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: auth debug = true 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on old message = true 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on bug = true 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: mon pg warn max object skew = 0 2023-11-28T18:21:58.863 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default size = 2 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow primary affinity = true 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow pg remap = true 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on legacy crush tunables = false 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on crush straw calc version zero = false 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on no sortbitwise = false 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on osd down out interval zero = false 2023-11-28T18:21:58.864 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on too few osds = false 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: mon_allow_pool_size_one = true 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: osd default data pool replay window = 5 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: mon allow pool delete = true 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: mon cluster log file level = debug 2023-11-28T18:21:58.865 INFO:teuthology.orchestra.run.smithi193.stdout: debug asserts on shutdown = true 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: mon health detail to clog = false 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: mon host = "172.21.15.7,172.21.15.193,[v2:172.21.15.193:3301,v1:172.21.15.193:6790]" 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: fsid = b4396e54-0cbb-426d-8760-05a802ba48a2 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout:[osd] 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: osd journal size = 100 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub load threshold = 5.0 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub max interval = 600 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: osd mclock profile = high_recovery_ops 2023-11-28T18:21:58.866 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd recover clone overlap = true 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd recovery max chunk = 1048576 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug shutdown = true 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug op order = true 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify stray on activate = true 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug trim objects = true 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd open classes on start = true 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug pg log writeout = true 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.867 INFO:teuthology.orchestra.run.smithi193.stdout: osd deep scrub update digest min age = 30 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: osd map max advance = 10 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: journal zero on create = true 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: filestore ondisk finisher threads = 3 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: filestore apply finisher threads = 3 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: bdev debug aio = true 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug misdirected ops = true 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: bdev async discard = True 2023-11-28T18:21:58.868 INFO:teuthology.orchestra.run.smithi193.stdout: bdev enable discard = True 2023-11-28T18:21:58.869 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore allocator = bitmap 2023-11-28T18:21:58.869 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore block size = 96636764160 2023-11-28T18:21:58.869 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore fsck on mount = True 2023-11-28T18:21:58.869 INFO:teuthology.orchestra.run.smithi193.stdout: debug bluefs = 1/20 2023-11-28T18:21:58.869 INFO:teuthology.orchestra.run.smithi193.stdout: debug bluestore = 1/20 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: debug osd = 20 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: debug rocksdb = 4/10 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: filestore flush min = 0 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: mon client directed command retry = 5 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd full ratio = 0.9 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd nearfull ratio = 0.8 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject delay max = 1 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject delay probability = 0.005 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject delay type = client mds 2023-11-28T18:21:58.870 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject socket failures = 2500 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: osd blocked scrub grace period = 3600 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: osd failsafe full ratio = 0.95 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: osd max markdown count = 1000 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: osd objectstore = bluestore 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: osd op complaint time = 180 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr] 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: debug mgr = 20 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2023-11-28T18:21:58.871 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: mgr/telemetry/nag = false 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout:[mon] 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: debug paxos = 20 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: mon data avail warn = 5 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: mon mgr mkfs grace = 240 2023-11-28T18:21:58.872 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd reporter subtree level = osd 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd prime pg temp = true 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T18:21:58.873 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: # 1m isn't quite enough 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: mon_down_mkfs_grace = 2m 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_filestore_osds = false 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: mon op complaint time = 120 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout:[client] 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: rgw cache enabled = true 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable ops log = true 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable usage log = true 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T18:21:58.874 INFO:teuthology.orchestra.run.smithi193.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout: client mount timeout = 600 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout: client_shutdown_timeout = 120 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout: debug client = 20 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout: rados mon op timeout = 900 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout: rados osd op timeout = 900 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.a] 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.b] 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.c] 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.a] 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.c] 2023-11-28T18:21:58.875 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.e] 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.b] 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.d] 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.f] 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout:[mds] 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout: debug mds = 20 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout: debug mds balancer = 20 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout: mds bal fragment size max = 10000 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout: mds bal merge size = 5 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout: mds bal split bits = 3 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout: mds bal split size = 100 2023-11-28T18:21:58.876 INFO:teuthology.orchestra.run.smithi193.stdout: mds debug frag = True 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: mds debug scatterstat = True 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: mds oft prefetch dirfrags = True 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: mds op complaint time = 180 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: mds verify scatter = True 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: mds_dir_prefetch = False 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: mds_sleep_rank_change = 5000000.0 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: mon client directed command retry = 5 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject delay max = 1 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject delay probability = 0.005 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject delay type = client mds osd 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject socket failures = 2500 2023-11-28T18:21:58.877 INFO:teuthology.orchestra.run.smithi193.stdout: osd op complaint time = 180 2023-11-28T18:21:58.878 INFO:teuthology.orchestra.run.smithi193.stdout: rados mon op timeout = 900 2023-11-28T18:21:58.878 INFO:teuthology.orchestra.run.smithi193.stdout: rados osd op timeout = 900 2023-11-28T18:21:58.879 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-28T18:21:58.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-28T18:21:58.981 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-28T18:21:58.981 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:58.981 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-28T18:21:58.990 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:58.990 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-28T18:21:59.042 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:21:59.043 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:59.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T18:21:59.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T18:21:59.123 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:21:59.123 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:21:59.174 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:21:59.174 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:21:59.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T18:21:59.175 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T18:21:59.211 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:21:59.211 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:21:59.267 INFO:tasks.ceph:Setting up mon nodes... 2023-11-28T18:21:59.267 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-28T18:21:59.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T18:21:59.326 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T18:21:59.331 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T18:21:59.392 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T18:21:59.410 INFO:tasks.ceph:Setting up mds nodes... 2023-11-28T18:21:59.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T18:21:59.469 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T18:21:59.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-28T18:21:59.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T18:21:59.595 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T18:21:59.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-28T18:21:59.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-11-28T18:21:59.723 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-11-28T18:21:59.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-11-28T18:21:59.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T18:21:59.809 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T18:21:59.814 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-28T18:21:59.835 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T18:21:59.938 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T18:21:59.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-28T18:21:59.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-11-28T18:22:00.061 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-11-28T18:22:00.065 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-11-28T18:22:00.085 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-28T18:22:00.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-28T18:22:00.131 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-28T18:22:00.150 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-28T18:22:00.150 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T18:22:00.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-28T18:22:00.214 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T18:22:00.214 INFO:tasks.ceph:role: osd.0 2023-11-28T18:22:00.214 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:22:00.214 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T18:22:00.278 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T18:22:00.278 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T18:22:00.278 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T18:22:00.278 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T18:22:00.278 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T18:22:00.278 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-28T18:22:00.278 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T18:22:00.278 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T18:22:00.279 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T18:22:00.279 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T18:22:00.302 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-28T18:22:00.304 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-28T18:22:00.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-28T18:22:00.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-28T18:22:00.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2023-11-28T18:22:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:22:00.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-28T18:22:00.503 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T18:22:00.503 INFO:tasks.ceph:role: osd.1 2023-11-28T18:22:00.503 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:22:00.503 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T18:22:00.570 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T18:22:00.594 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-28T18:22:00.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-28T18:22:00.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-28T18:22:00.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-28T18:22:00.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2023-11-28T18:22:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:22:00.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-28T18:22:00.798 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T18:22:00.798 INFO:tasks.ceph:role: osd.2 2023-11-28T18:22:00.798 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:22:00.799 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T18:22:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T18:22:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-28T18:22:00.889 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-28T18:22:00.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-28T18:22:00.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-28T18:22:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2023-11-28T18:22:01.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:22:01.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-28T18:22:01.078 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T18:22:01.078 INFO:tasks.ceph:role: osd.3 2023-11-28T18:22:01.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:22:01.078 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T18:22:01.141 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T18:22:01.142 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T18:22:01.171 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-28T18:22:01.171 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-28T18:22:01.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-28T18:22:01.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-28T18:22:01.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2023-11-28T18:22:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:22:01.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:01.434+0000 7efc3dd5a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-28T18:22:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:01.434+0000 7efc3dd5a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-28T18:22:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:01.434+0000 7efc3dd5a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T18:22:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:01.434+0000 7efc3dd5a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T18:22:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:01.434+0000 7efc3dd5a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T18:22:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:01.438+0000 7efc3dd5a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-28T18:22:04.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-28T18:22:04.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:04.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:04.558+0000 7f62f1c69640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-28T18:22:04.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:04.558+0000 7f62f1c69640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-28T18:22:04.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:04.558+0000 7f62f1c69640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T18:22:04.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:04.558+0000 7f62f1c69640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T18:22:04.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:04.558+0000 7f62f1c69640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T18:22:04.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:04.558+0000 7f62f1c69640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-28T18:22:07.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-28T18:22:07.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:07.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:07.674+0000 7f4f72ee6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-28T18:22:07.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:07.674+0000 7f4f72ee6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-28T18:22:07.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:07.674+0000 7f4f72ee6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T18:22:07.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:07.674+0000 7f4f72ee6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T18:22:07.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:07.674+0000 7f4f72ee6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T18:22:07.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:07.674+0000 7f4f72ee6640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-28T18:22:10.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-28T18:22:10.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:10.770+0000 7f0ce5c2d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-28T18:22:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:10.774+0000 7f0ce5c2d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-28T18:22:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:10.774+0000 7f0ce5c2d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T18:22:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:10.774+0000 7f0ce5c2d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T18:22:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:10.774+0000 7f0ce5c2d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T18:22:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:10.774+0000 7f0ce5c2d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-28T18:22:13.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-28T18:22:13.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-28T18:22:13.802 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T18:22:13.802 INFO:tasks.ceph:role: osd.4 2023-11-28T18:22:13.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:22:13.802 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T18:22:13.875 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T18:22:13.876 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T18:22:14.364 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2023-11-28T18:22:14.367 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2023-11-28T18:22:14.367 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-28T18:22:14.541 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-28T18:22:14.560 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: /sbin/restorecon: command not found 2023-11-28T18:22:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:22:14.561 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-28T18:22:14.628 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T18:22:14.628 INFO:tasks.ceph:role: osd.5 2023-11-28T18:22:14.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:22:14.629 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T18:22:14.699 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T18:22:14.700 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T18:22:15.135 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2023-11-28T18:22:15.138 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2023-11-28T18:22:15.138 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-28T18:22:15.246 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-28T18:22:15.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: /sbin/restorecon: command not found 2023-11-28T18:22:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:22:15.265 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-28T18:22:15.331 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T18:22:15.331 INFO:tasks.ceph:role: osd.6 2023-11-28T18:22:15.331 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:22:15.332 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T18:22:15.407 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T18:22:15.408 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T18:22:15.878 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2023-11-28T18:22:15.881 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2023-11-28T18:22:15.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-28T18:22:16.036 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-28T18:22:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:22:16.055 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: /sbin/restorecon: command not found 2023-11-28T18:22:16.055 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-28T18:22:16.120 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T18:22:16.120 INFO:tasks.ceph:role: osd.7 2023-11-28T18:22:16.120 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:22:16.120 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T18:22:16.190 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T18:22:16.191 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T18:22:16.660 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2023-11-28T18:22:16.663 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2023-11-28T18:22:16.663 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-28T18:22:16.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-28T18:22:16.842 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: /sbin/restorecon: command not found 2023-11-28T18:22:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:22:16.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:16.977 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:16.974+0000 7ff48dde8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-28T18:22:17.056 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:16.978+0000 7ff48dde8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-28T18:22:17.056 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:16.978+0000 7ff48dde8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T18:22:17.056 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:16.978+0000 7ff48dde8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T18:22:17.056 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:16.978+0000 7ff48dde8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T18:22:17.057 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:16.978+0000 7ff48dde8640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-28T18:22:19.963 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-28T18:22:19.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:20.082 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:20.078+0000 7fc3e0480640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-28T18:22:20.082 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:20.078+0000 7fc3e0480640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-28T18:22:20.082 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:20.078+0000 7fc3e0480640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T18:22:20.082 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:20.078+0000 7fc3e0480640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T18:22:20.082 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:20.078+0000 7fc3e0480640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T18:22:20.083 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:20.082+0000 7fc3e0480640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-28T18:22:23.086 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-28T18:22:23.108 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:23.203 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:23.202+0000 7fbd32457640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-28T18:22:23.204 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:23.202+0000 7fbd32457640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-28T18:22:23.204 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:23.202+0000 7fbd32457640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T18:22:23.204 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:23.202+0000 7fbd32457640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T18:22:23.204 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:23.202+0000 7fbd32457640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T18:22:23.205 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:23.202+0000 7fbd32457640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-28T18:22:26.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-28T18:22:26.214 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:26.317 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:26.314+0000 7fdd6ff07640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-28T18:22:26.318 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:26.314+0000 7fdd6ff07640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-28T18:22:26.318 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:26.314+0000 7fdd6ff07640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T18:22:26.318 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:26.314+0000 7fdd6ff07640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T18:22:26.318 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:26.314+0000 7fdd6ff07640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T18:22:26.319 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:26.318+0000 7fdd6ff07640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-28T18:22:26.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:22:26.938 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:22:29.305 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-28T18:22:29.326 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-28T18:22:29.326 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:29.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-28T18:22:29.350 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:29.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-28T18:22:29.416 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:29.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-28T18:22:29.484 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:29.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-11-28T18:22:29.548 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:29.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-28T18:22:29.612 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:29.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-28T18:22:29.676 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:29.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-28T18:22:29.739 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:29.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-28T18:22:29.803 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:29.804 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-28T18:22:29.825 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:29.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-28T18:22:29.891 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:29.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-28T18:22:29.955 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:29.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-11-28T18:22:30.019 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:30.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-28T18:22:30.083 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:30.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-28T18:22:30.147 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:30.147 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-28T18:22:30.210 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:30.210 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-28T18:22:30.274 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:30.275 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-28T18:22:30.285 INFO:tasks.ceph:Adding keys to all mons... 2023-11-28T18:22:30.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T18:22:30.333 DEBUG:teuthology.orchestra.run.smithi193:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T18:22:30.350 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.y] 2023-11-28T18:22:30.350 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDHL2ZlLGiDExAAlDAs3BNpFiY6Uu+PIkb2Aw== 2023-11-28T18:22:30.350 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.a] 2023-11-28T18:22:30.350 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDHL2Zl8e38GxAA/3IkvlhzHEeDqzzeRNfngA== 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.c] 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDHL2ZlIlCIIxAAhpeMKr4Kdf/QBzdzZY+ZgQ== 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.e] 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDHL2Zl/NsxKxAAfSrNu8E3zalflE1vasR5SA== 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDJL2Zlfi4aGhAAw2lqiCZ+ev0Q5MN3lX3pVQ== 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDML2Zl1SNgIRAAt+ec6rIDupetHjhbsjHqSA== 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDPL2ZlVSVFKBAAWQ+RNRBgGjmSbbax56e6mQ== 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDSL2ZltnEzLhAAu4/Cw0SCWgDolkhcrv//zw== 2023-11-28T18:22:30.351 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDHL2Zlyp12FxAAHEFL7GvapBDAj3vKsUG7xA== 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.b] 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDHL2Zlk2tPMBAAFRkkFhYirKnJdRpwnt5uUg== 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.d] 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDHL2ZlvbP+NxAAxslhGf9UJW0qldLwm+Zcsw== 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.f] 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDIL2Zl1x27AxAAqO3HOMVcnr5Wwr8wvr0YZg== 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.4] 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDYL2ZlYJRMOhAA6SvIAU/KtUGQ1VSNE7fYQA== 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.5] 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDcL2ZlphjvBBAAelAxfRv2w01QZReUfs1kaA== 2023-11-28T18:22:30.352 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.6] 2023-11-28T18:22:30.353 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDfL2ZlaT4uDBAA/JzegzPu6RabLiRxt44W9w== 2023-11-28T18:22:30.353 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.7] 2023-11-28T18:22:30.353 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDiL2ZlvkD5EhAAIwy0c0SlA2VWzOfb5/+P2w== 2023-11-28T18:22:30.353 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2023-11-28T18:22:30.353 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDIL2Zl9yLbBxAAuKyDgQak90cH4KWNTp80mA== 2023-11-28T18:22:30.354 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.y] 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDHL2ZlLGiDExAAlDAs3BNpFiY6Uu+PIkb2Aw== 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.a] 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDHL2Zl8e38GxAA/3IkvlhzHEeDqzzeRNfngA== 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.c] 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDHL2ZlIlCIIxAAhpeMKr4Kdf/QBzdzZY+ZgQ== 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.e] 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDHL2Zl/NsxKxAAfSrNu8E3zalflE1vasR5SA== 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.0] 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDJL2Zlfi4aGhAAw2lqiCZ+ev0Q5MN3lX3pVQ== 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.1] 2023-11-28T18:22:30.355 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDML2Zl1SNgIRAAt+ec6rIDupetHjhbsjHqSA== 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.2] 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDPL2ZlVSVFKBAAWQ+RNRBgGjmSbbax56e6mQ== 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.3] 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDSL2ZltnEzLhAAu4/Cw0SCWgDolkhcrv//zw== 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.x] 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDHL2Zlyp12FxAAHEFL7GvapBDAj3vKsUG7xA== 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.b] 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDHL2Zlk2tPMBAAFRkkFhYirKnJdRpwnt5uUg== 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.d] 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDHL2ZlvbP+NxAAxslhGf9UJW0qldLwm+Zcsw== 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.f] 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDIL2Zl1x27AxAAqO3HOMVcnr5Wwr8wvr0YZg== 2023-11-28T18:22:30.356 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.4] 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDYL2ZlYJRMOhAA6SvIAU/KtUGQ1VSNE7fYQA== 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.5] 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDcL2ZlphjvBBAAelAxfRv2w01QZReUfs1kaA== 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.6] 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDfL2ZlaT4uDBAA/JzegzPu6RabLiRxt44W9w== 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.7] 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDiL2ZlvkD5EhAAIwy0c0SlA2VWzOfb5/+P2w== 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout:[client.0] 2023-11-28T18:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQDIL2Zl9yLbBxAAuKyDgQak90cH4KWNTp80mA== 2023-11-28T18:22:30.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T18:22:30.402 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T18:22:30.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.525 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:30.647 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:30.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:30.705 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:30.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:30.762 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:30.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:30.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:30.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T18:22:30.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T18:22:30.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.938 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:30.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:31.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:31.060 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T18:22:31.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:31.120 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:31.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:31.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:31.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:31.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:31.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:31.296 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T18:22:31.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T18:22:31.354 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T18:22:31.410 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-28T18:22:31.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-28T18:22:31.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T18:22:31.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-28T18:22:31.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-28T18:22:31.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T18:22:31.707 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-28T18:22:31.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-28T18:22:31.790 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T18:22:31.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-28T18:22:31.922 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:31.926 DEBUG:teuthology.orchestra.run.smithi193:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T18:22:31.974 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-28T18:22:31.974 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T18:22:31.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T18:22:31.981 INFO:tasks.ceph.mon.a:Started 2023-11-28T18:22:31.981 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-28T18:22:31.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-28T18:22:32.017 INFO:tasks.ceph.mon.b:Started 2023-11-28T18:22:32.017 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-28T18:22:32.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-28T18:22:32.021 INFO:tasks.ceph.mon.c:Started 2023-11-28T18:22:32.021 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-28T18:22:32.021 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-28T18:22:32.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-28T18:22:32.025 INFO:tasks.ceph.mgr.y:Started 2023-11-28T18:22:32.025 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-28T18:22:32.025 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-28T18:22:32.029 INFO:tasks.ceph.mgr.x:Started 2023-11-28T18:22:32.029 INFO:tasks.ceph:Setting crush tunables to default 2023-11-28T18:22:32.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2023-11-28T18:22:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.169+0000 7f7c4d65a640 1 Processor -- start 2023-11-28T18:22:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.169+0000 7f7c4d65a640 1 -- start start 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.169+0000 7f7c4d65a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c4d65a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48157f60 con 0x7f7c4806e7c0 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c4d65a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48158240 con 0x7f7c48141030 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c4d65a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48158520 con 0x7f7c4806e3a0 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:51880/0 (socket says 172.21.15.7:51880) 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 learned_addr learned my addr 172.21.15.7:0/2585555705 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:32.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.173+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:22:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.377+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.377+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:22:32.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:32.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.781+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:32.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.781+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:32.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:32.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:32.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:22:33.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:33.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:33.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:33.585+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:33.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:33.585+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:33.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:33.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:33.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:33.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:22:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 msgr2=0x7f7c480c4b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:22:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c4806e3a0 0x7f7c480c4b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 >> v1:172.21.15.193:6789/0 conn(0x7f7c48141030 legacy=0x7f7c4815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 >> v1:172.21.15.7:6789/0 conn(0x7f7c4806e7c0 legacy=0x7f7c48072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c48158520 con 0x7f7c28003720 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c28008d90 con 0x7f7c28006260 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:22:35.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.173+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:22:35.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:22:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:22:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:35.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:35.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:35.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:22:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:22:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:36.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:36.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:36.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:36.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:36.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:22:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:22:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 msgr2=0x7f7c28005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:22:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c28003720 0x7f7c28005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c477fe640 1 -- 172.21.15.7:0/2585555705 reap_dead start 2023-11-28T18:22:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48158800 con 0x7f7c2800a8e0 2023-11-28T18:22:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2800f560 con 0x7f7c28003b70 2023-11-28T18:22:38.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c48158520 con 0x7f7c28006260 2023-11-28T18:22:38.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:38.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:38.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:38.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:38.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:22:38.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:38.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.377+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:38.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.377+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:38.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:38.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:22:38.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:38.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.781+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:38.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.781+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:38.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:38.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:38.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:22:39.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:39.585+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:39.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:39.585+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:39.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:39.585+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:39.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:39.585+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:39.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:39.585+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:22:41.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 msgr2=0x7f7c28008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:22:41.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.193:3300/0 conn(0x7f7c28006260 0x7f7c28008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 >> v1:172.21.15.193:6789/0 conn(0x7f7c28003b70 legacy=0x7f7c280091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 >> v1:172.21.15.7:6789/0 conn(0x7f7c2800a8e0 legacy=0x7f7c2800ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c457fa640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c48158520 con 0x7f7c2800e510 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c457fa640 1 -- 172.21.15.7:0/2585555705 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c28008d90 con 0x7f7c2800f870 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 msgr2=0x7f7c2800fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 msgr2=0x7f7c2800e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 msgr2=0x7f7c2800fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 msgr2=0x7f7c2800e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:41.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:41.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:41.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:22:41.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.173+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:22:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 msgr2=0x7f7c2800e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 msgr2=0x7f7c2800e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:22:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 msgr2=0x7f7c2800fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 msgr2=0x7f7c2800fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:41.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.377+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:22:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 msgr2=0x7f7c2800e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T18:22:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 msgr2=0x7f7c2800e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:22:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 msgr2=0x7f7c2800fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T18:22:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c46ffd640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 msgr2=0x7f7c2800fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:22:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:22:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:41.781+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:22:42.568 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:42.565+0000 7fd851af0180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T18:22:42.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> v2:172.21.15.7:3300/0 conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:42.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:42.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 msgr2=0x7f7c2800fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:42.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c2800f870 0x7f7c2800fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:42.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c3ffff640 1 -- 172.21.15.7:0/2585555705 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c280107b0 con 0x7f7c2800e510 2023-11-28T18:22:42.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2800e510 0x7f7c2800e9c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009100 tx=0x7f7c30007390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445cc2863a5c1974 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:42.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c467fc640 1 -- 172.21.15.7:0/2585555705 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c300ad670 con 0x7f7c2800e510 2023-11-28T18:22:42.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c467fc640 1 -- 172.21.15.7:0/2585555705 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3000c040 con 0x7f7c2800e510 2023-11-28T18:22:42.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.585+0000 7f7c467fc640 1 -- 172.21.15.7:0/2585555705 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c300ada10 con 0x7f7c2800e510 2023-11-28T18:22:42.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- 172.21.15.7:0/2585555705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2800e510 msgr2=0x7f7c2800e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:42.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2800e510 0x7f7c2800e9c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009100 tx=0x7f7c30007390 comp rx=0 tx=0).stop 2023-11-28T18:22:42.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c477fe640 1 -- 172.21.15.7:0/2585555705 reap_dead start 2023-11-28T18:22:42.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- 172.21.15.7:0/2585555705 shutdown_connections 2023-11-28T18:22:42.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 --2- 172.21.15.7:0/2585555705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2800e510 0x7f7c2800e9c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:42.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- 172.21.15.7:0/2585555705 >> 172.21.15.7:0/2585555705 conn(0x7f7c48073f00 msgr2=0x7f7c480662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:42.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- 172.21.15.7:0/2585555705 shutdown_connections 2023-11-28T18:22:42.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- 172.21.15.7:0/2585555705 wait complete. 2023-11-28T18:22:42.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 Processor -- start 2023-11-28T18:22:42.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- start start 2023-11-28T18:22:42.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c2800e510 0x7f7c4813d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:42.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c480c2590 0x7f7c48139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c48152d50 0x7f7c4813a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c481ee300 con 0x7f7c480c2590 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4813cb60 con 0x7f7c2800e510 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.589+0000 7f7c4d65a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c481ee7f0 con 0x7f7c48152d50 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c480c2590 0x7f7c48139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c480c2590 0x7f7c48139ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53400/0 (socket says 172.21.15.7:53400) 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3ffff640 1 -- 172.21.15.7:0/1041290581 learned_addr learned my addr 172.21.15.7:0/1041290581 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c2800e510 0x7f7c4813d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c477fe640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c48152d50 0x7f7c4813a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c46ffd640 1 -- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c48152d50 msgr2=0x7f7c4813a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c48152d50 0x7f7c4813a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c46ffd640 1 -- 172.21.15.7:0/1041290581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c480c2590 msgr2=0x7f7c48139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c480c2590 0x7f7c48139ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c46ffd640 1 -- 172.21.15.7:0/1041290581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3000e040 con 0x7f7c2800e510 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3ffff640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c480c2590 0x7f7c48139ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c46ffd640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c2800e510 0x7f7c4813d4b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7c38003f50 tx=0x7f7c380178c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3f7fe640 1 -- 172.21.15.7:0/1041290581 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38001570 con 0x7f7c2800e510 2023-11-28T18:22:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3f7fe640 1 -- 172.21.15.7:0/1041290581 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3801d040 con 0x7f7c2800e510 2023-11-28T18:22:42.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3f7fe640 1 -- 172.21.15.7:0/1041290581 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c380018d0 con 0x7f7c2800e510 2023-11-28T18:22:42.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c48152870 con 0x7f7c2800e510 2023-11-28T18:22:42.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c481ee510 con 0x7f7c2800e510 2023-11-28T18:22:42.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3f7fe640 1 -- 172.21.15.7:0/1041290581 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7c3802a400 con 0x7f7c2800e510 2023-11-28T18:22:42.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c3f7fe640 1 -- 172.21.15.7:0/1041290581 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7c38021440 con 0x7f7c2800e510 2023-11-28T18:22:42.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.593+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4804e890 con 0x7f7c2800e510 2023-11-28T18:22:42.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.597+0000 7f7c3f7fe640 1 -- 172.21.15.7:0/1041290581 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7c38025020 con 0x7f7c2800e510 2023-11-28T18:22:42.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.661+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7c480c6030 con 0x7f7c2800e510 2023-11-28T18:22:42.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.665+0000 7f7c3f7fe640 1 -- 172.21.15.7:0/1041290581 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7c3802a8e0 con 0x7f7c2800e510 2023-11-28T18:22:42.670 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2023-11-28T18:22:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c2800e510 msgr2=0x7f7c4813d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c2800e510 0x7f7c4813d4b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7c38003f50 tx=0x7f7c380178c0 comp rx=0 tx=0).stop 2023-11-28T18:22:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 shutdown_connections 2023-11-28T18:22:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c48152d50 0x7f7c4813a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c480c2590 0x7f7c48139ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 --2- 172.21.15.7:0/1041290581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c2800e510 0x7f7c4813d4b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 >> 172.21.15.7:0/1041290581 conn(0x7f7c48073f00 msgr2=0x7f7c480c9280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:42.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 shutdown_connections 2023-11-28T18:22:42.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:42.669+0000 7f7c4d65a640 1 -- 172.21.15.7:0/1041290581 wait complete. 2023-11-28T18:22:42.690 INFO:tasks.ceph:check_enable_crimson: False 2023-11-28T18:22:42.690 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-28T18:22:42.691 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:42.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-28T18:22:42.713 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:42.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-28T18:22:42.752 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:42.749+0000 7fd4c8831180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T18:22:42.776 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:42.776 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-28T18:22:42.842 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:22:42.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-28T18:22:42.907 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:42.907 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-28T18:22:42.928 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:42.928 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-28T18:22:42.991 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:42.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-28T18:22:43.055 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-28T18:22:43.055 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-28T18:22:43.119 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 5e137b39-c9c2-4354-b748-63bb98afe2c2 0 2023-11-28T18:22:43.169 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:43.165+0000 7fd851af0180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T18:22:43.296 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:43.293+0000 7fd4c8831180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T18:22:43.298 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.293+0000 7f7f998fe640 1 Processor -- start 2023-11-28T18:22:43.298 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.293+0000 7f7f998fe640 1 -- start start 2023-11-28T18:22:43.300 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f998fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f94154130 0x7f7f94159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.300 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f998fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7f940c4c80 0x7f7f9415de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.300 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f998fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f940caeb0 con 0x7f7f940ca530 2023-11-28T18:22:43.300 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f998fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9415a3f0 con 0x7f7f940c4c80 2023-11-28T18:22:43.301 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f998fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9415d9b0 con 0x7f7f94154130 2023-11-28T18:22:43.301 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f92ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f94154130 0x7f7f94159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.301 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f8bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7f940c4c80 0x7f7f9415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.301 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f92ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f94154130 0x7f7f94159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:41568/0 (socket says 172.21.15.193:41568) 2023-11-28T18:22:43.301 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f92ffd640 1 -- 172.21.15.193:0/4156085568 learned_addr learned my addr 172.21.15.193:0/4156085568 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:43.302 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f927fc640 1 -- 172.21.15.193:0/4156085568 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137101915 0 0) 0x7f7f940caeb0 con 0x7f7f940ca530 2023-11-28T18:22:43.302 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f927fc640 1 -- 172.21.15.193:0/4156085568 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f80003530 con 0x7f7f940ca530 2023-11-28T18:22:43.302 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f92ffd640 1 -- 172.21.15.193:0/4156085568 >> v2:172.21.15.193:3300/0 conn(0x7f7f940c4c80 msgr2=0x7f7f9415de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:43.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f92ffd640 1 --2- 172.21.15.193:0/4156085568 >> v2:172.21.15.193:3300/0 conn(0x7f7f940c4c80 0x7f7f9415de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f92ffd640 1 -- 172.21.15.193:0/4156085568 >> v1:172.21.15.193:6789/0 conn(0x7f7f940ca530 legacy=0x7f7f940cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.297+0000 7f7f92ffd640 1 -- 172.21.15.193:0/4156085568 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9415e380 con 0x7f7f94154130 2023-11-28T18:22:43.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f92ffd640 1 --2- 172.21.15.193:0/4156085568 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f94154130 0x7f7f94159eb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009870 tx=0x7f7f8401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2749bf19981bf578 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:43.304 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f927fc640 1 -- 172.21.15.193:0/4156085568 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f840a8a30 con 0x7f7f94154130 2023-11-28T18:22:43.304 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f927fc640 1 -- 172.21.15.193:0/4156085568 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f840a8b90 con 0x7f7f94154130 2023-11-28T18:22:43.304 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f927fc640 1 -- 172.21.15.193:0/4156085568 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f840b0440 con 0x7f7f94154130 2023-11-28T18:22:43.305 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 -- 172.21.15.193:0/4156085568 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f94154130 msgr2=0x7f7f94159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.305 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 --2- 172.21.15.193:0/4156085568 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f94154130 0x7f7f94159eb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009870 tx=0x7f7f8401ed30 comp rx=0 tx=0).stop 2023-11-28T18:22:43.305 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 -- 172.21.15.193:0/4156085568 shutdown_connections 2023-11-28T18:22:43.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 --2- 172.21.15.193:0/4156085568 >> v2:172.21.15.193:3300/0 conn(0x7f7f940c4c80 0x7f7f9415de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 --2- 172.21.15.193:0/4156085568 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f94154130 0x7f7f94159eb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 -- 172.21.15.193:0/4156085568 >> 172.21.15.193:0/4156085568 conn(0x7f7f94073ca0 msgr2=0x7f7f94071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:43.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 -- 172.21.15.193:0/4156085568 shutdown_connections 2023-11-28T18:22:43.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 -- 172.21.15.193:0/4156085568 wait complete. 2023-11-28T18:22:43.307 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 Processor -- start 2023-11-28T18:22:43.307 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.301+0000 7f7f998fe640 1 -- start start 2023-11-28T18:22:43.308 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f998fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f940c4c80 0x7f7f941463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.308 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f998fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f940ca530 0x7f7f941f3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f998fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f94154130 0x7f7f940b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f998fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940c9a20 con 0x7f7f940c4c80 2023-11-28T18:22:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f998fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f941f2f60 con 0x7f7f94154130 2023-11-28T18:22:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f998fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940c9740 con 0x7f7f940ca530 2023-11-28T18:22:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f940ca530 0x7f7f941f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f937fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f94154130 0x7f7f940b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f940ca530 0x7f7f941f3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:41580/0 (socket says 172.21.15.193:41580) 2023-11-28T18:22:43.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 -- 172.21.15.193:0/2887695170 learned_addr learned my addr 172.21.15.193:0/2887695170 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:43.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f92ffd640 1 --2- 172.21.15.193:0/2887695170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f940c4c80 0x7f7f941463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 -- 172.21.15.193:0/2887695170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f94154130 msgr2=0x7f7f940b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 --2- 172.21.15.193:0/2887695170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f94154130 0x7f7f940b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 -- 172.21.15.193:0/2887695170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f940c4c80 msgr2=0x7f7f941463c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 --2- 172.21.15.193:0/2887695170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f940c4c80 0x7f7f941463c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.311 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 -- 172.21.15.193:0/2887695170 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f840094d0 con 0x7f7f940ca530 2023-11-28T18:22:43.311 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f8bfff640 1 --2- 172.21.15.193:0/2887695170 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f940ca530 0x7f7f941f3720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00e960 tx=0x7f7f7c00ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:43.312 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f90ff9640 1 -- 172.21.15.193:0/2887695170 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c009690 con 0x7f7f940ca530 2023-11-28T18:22:43.312 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f90ff9640 1 -- 172.21.15.193:0/2887695170 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c00ca00 con 0x7f7f940ca530 2023-11-28T18:22:43.312 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f90ff9640 1 -- 172.21.15.193:0/2887695170 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00cd70 con 0x7f7f940ca530 2023-11-28T18:22:43.312 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f941f0530 con 0x7f7f940ca530 2023-11-28T18:22:43.312 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f940b52f0 con 0x7f7f940ca530 2023-11-28T18:22:43.313 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f90ff9640 1 -- 172.21.15.193:0/2887695170 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7f7c005ca0 con 0x7f7f940ca530 2023-11-28T18:22:43.313 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.305+0000 7f7f90ff9640 1 -- 172.21.15.193:0/2887695170 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7f7c013840 con 0x7f7f940ca530 2023-11-28T18:22:43.313 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.309+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9404e890 con 0x7f7f940ca530 2023-11-28T18:22:43.329 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.325+0000 7f7f90ff9640 1 -- 172.21.15.193:0/2887695170 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7f7c021060 con 0x7f7f940ca530 2023-11-28T18:22:43.385 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:43.381+0000 7fd851af0180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T18:22:43.398 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.393+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5e137b39-c9c2-4354-b748-63bb98afe2c2", "id": 0} v 0) v1 -- 0x7f7f941f33d0 con 0x7f7f940ca530 2023-11-28T18:22:43.406 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.401+0000 7f7f90ff9640 1 -- 172.21.15.193:0/2887695170 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5e137b39-c9c2-4354-b748-63bb98afe2c2", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7f7c017070 con 0x7f7f940ca530 2023-11-28T18:22:43.407 INFO:teuthology.orchestra.run.smithi193.stdout:0 2023-11-28T18:22:43.408 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f940ca530 msgr2=0x7f7f941f3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.408 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 --2- 172.21.15.193:0/2887695170 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f940ca530 0x7f7f941f3720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00e960 tx=0x7f7f7c00ee20 comp rx=0 tx=0).stop 2023-11-28T18:22:43.408 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 shutdown_connections 2023-11-28T18:22:43.408 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 --2- 172.21.15.193:0/2887695170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f94154130 0x7f7f940b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.408 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 --2- 172.21.15.193:0/2887695170 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f940ca530 0x7f7f941f3720 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.408 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 --2- 172.21.15.193:0/2887695170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f940c4c80 0x7f7f941463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.409 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 >> 172.21.15.193:0/2887695170 conn(0x7f7f94073ca0 msgr2=0x7f7f94149040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:43.409 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 shutdown_connections 2023-11-28T18:22:43.409 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.405+0000 7f7f998fe640 1 -- 172.21.15.193:0/2887695170 wait complete. 2023-11-28T18:22:43.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 95148a41-c6a1-4b3d-a5cd-76a64e27aa52 1 2023-11-28T18:22:43.505 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:43.501+0000 7fd4c8831180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T18:22:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.557+0000 7f5fdba46640 1 Processor -- start 2023-11-28T18:22:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fdba46640 1 -- start start 2023-11-28T18:22:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fdba46640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5fd4154130 0x7f5fd4159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fdba46640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5fd40c4c80 0x7f5fd415de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fdba46640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd40ca530 0x7f5fd40cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fdba46640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd415a3f0 con 0x7f5fd4154130 2023-11-28T18:22:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fdba46640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd40caeb0 con 0x7f5fd40c4c80 2023-11-28T18:22:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fdba46640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd415d9b0 con 0x7f5fd40ca530 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd9fe3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd40ca530 0x7f5fd40cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd8fe1640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5fd40c4c80 0x7f5fd415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd9fe3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd40ca530 0x7f5fd40cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:41594/0 (socket says 172.21.15.193:41594) 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd8fe1640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5fd40c4c80 0x7f5fd415de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50524/0 (socket says 172.21.15.193:50524) 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd9fe3640 1 -- 172.21.15.193:0/3317156031 learned_addr learned my addr 172.21.15.193:0/3317156031 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd97e2640 1 --2- 172.21.15.193:0/3317156031 >> v2:172.21.15.7:3300/0 conn(0x7f5fd4154130 0x7f5fd4159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd9fe3640 1 -- 172.21.15.193:0/3317156031 >> v2:172.21.15.193:3300/0 conn(0x7f5fd40c4c80 msgr2=0x7f5fd415de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd9fe3640 1 --2- 172.21.15.193:0/3317156031 >> v2:172.21.15.193:3300/0 conn(0x7f5fd40c4c80 0x7f5fd415de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd9fe3640 1 -- 172.21.15.193:0/3317156031 >> v2:172.21.15.7:3300/0 conn(0x7f5fd4154130 msgr2=0x7f5fd4159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd9fe3640 1 --2- 172.21.15.193:0/3317156031 >> v2:172.21.15.7:3300/0 conn(0x7f5fd4154130 0x7f5fd4159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.561+0000 7f5fd9fe3640 1 -- 172.21.15.193:0/3317156031 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd415e380 con 0x7f5fd40ca530 2023-11-28T18:22:43.568 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fd9fe3640 1 --2- 172.21.15.193:0/3317156031 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd40ca530 0x7f5fd40cb2b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40c9e60 tx=0x7f5fcc01f1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=205972171f72bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:43.568 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fcbfff640 1 -- 172.21.15.193:0/3317156031 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0af400 con 0x7f5fd40ca530 2023-11-28T18:22:43.568 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fcbfff640 1 -- 172.21.15.193:0/3317156031 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc0ab050 con 0x7f5fd40ca530 2023-11-28T18:22:43.569 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fcbfff640 1 -- 172.21.15.193:0/3317156031 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0af780 con 0x7f5fd40ca530 2023-11-28T18:22:43.569 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 -- 172.21.15.193:0/3317156031 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd40ca530 msgr2=0x7f5fd40cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.569 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 --2- 172.21.15.193:0/3317156031 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd40ca530 0x7f5fd40cb2b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40c9e60 tx=0x7f5fcc01f1c0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.570 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 -- 172.21.15.193:0/3317156031 shutdown_connections 2023-11-28T18:22:43.570 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 --2- 172.21.15.193:0/3317156031 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd40ca530 0x7f5fd40cb2b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.570 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 --2- 172.21.15.193:0/3317156031 >> v2:172.21.15.193:3300/0 conn(0x7f5fd40c4c80 0x7f5fd415de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.570 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 --2- 172.21.15.193:0/3317156031 >> v2:172.21.15.7:3300/0 conn(0x7f5fd4154130 0x7f5fd4159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.570 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 -- 172.21.15.193:0/3317156031 >> 172.21.15.193:0/3317156031 conn(0x7f5fd4073ca0 msgr2=0x7f5fd4071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:43.570 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 -- 172.21.15.193:0/3317156031 shutdown_connections 2023-11-28T18:22:43.571 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.565+0000 7f5fdba46640 1 -- 172.21.15.193:0/3317156031 wait complete. 2023-11-28T18:22:43.572 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fdba46640 1 Processor -- start 2023-11-28T18:22:43.572 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fdba46640 1 -- start start 2023-11-28T18:22:43.574 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fdba46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd40c4c80 0x7f5fd40b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fdba46640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fd40ca530 0x7f5fd415cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fdba46640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd4154130 0x7f5fd41474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fdba46640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd415c630 con 0x7f5fd40c4c80 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fdba46640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4159310 con 0x7f5fd40ca530 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fdba46640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd415b7d0 con 0x7f5fd4154130 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd8fe1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fd40ca530 0x7f5fd415cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd9fe3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd4154130 0x7f5fd41474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd8fe1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fd40ca530 0x7f5fd415cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50534/0 (socket says 172.21.15.193:50534) 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd8fe1640 1 -- 172.21.15.193:0/1899389737 learned_addr learned my addr 172.21.15.193:0/1899389737 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd97e2640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd40c4c80 0x7f5fd40b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd8fe1640 1 -- 172.21.15.193:0/1899389737 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd4154130 msgr2=0x7f5fd41474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.575 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd8fe1640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd4154130 0x7f5fd41474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.576 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd8fe1640 1 -- 172.21.15.193:0/1899389737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd40c4c80 msgr2=0x7f5fd40b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.576 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd8fe1640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd40c4c80 0x7f5fd40b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.576 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd8fe1640 1 -- 172.21.15.193:0/1899389737 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc009b30 con 0x7f5fd40ca530 2023-11-28T18:22:43.576 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.569+0000 7f5fd9fe3640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd4154130 0x7f5fd41474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:22:43.576 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fd8fe1640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fd40ca530 0x7f5fd415cff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009f00 tx=0x7f5fc40174a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fc9ffb640 1 -- 172.21.15.193:0/1899389737 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4017ee0 con 0x7f5fd40ca530 2023-11-28T18:22:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fc9ffb640 1 -- 172.21.15.193:0/1899389737 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4019040 con 0x7f5fd40ca530 2023-11-28T18:22:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd40b3e20 con 0x7f5fd40ca530 2023-11-28T18:22:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fc9ffb640 1 -- 172.21.15.193:0/1899389737 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4012d20 con 0x7f5fd40ca530 2023-11-28T18:22:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd40b3b40 con 0x7f5fd40ca530 2023-11-28T18:22:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fc9ffb640 1 -- 172.21.15.193:0/1899389737 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5fc4005ca0 con 0x7f5fd40ca530 2023-11-28T18:22:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fc9ffb640 1 -- 172.21.15.193:0/1899389737 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f5fc4006410 con 0x7f5fd40ca530 2023-11-28T18:22:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.573+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd404e890 con 0x7f5fd40ca530 2023-11-28T18:22:43.579 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.577+0000 7f5fc9ffb640 1 -- 172.21.15.193:0/1899389737 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f5fc40065d0 con 0x7f5fd40ca530 2023-11-28T18:22:43.646 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.641+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "95148a41-c6a1-4b3d-a5cd-76a64e27aa52", "id": 1} v 0) v1 -- 0x7f5fd40cb090 con 0x7f5fd40ca530 2023-11-28T18:22:43.653 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fc9ffb640 1 -- 172.21.15.193:0/1899389737 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "95148a41-c6a1-4b3d-a5cd-76a64e27aa52", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5fc4006770 con 0x7f5fd40ca530 2023-11-28T18:22:43.653 INFO:teuthology.orchestra.run.smithi193.stdout:1 2023-11-28T18:22:43.654 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fd40ca530 msgr2=0x7f5fd415cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.654 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fd40ca530 0x7f5fd415cff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009f00 tx=0x7f5fc40174a0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.654 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 shutdown_connections 2023-11-28T18:22:43.654 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fd4154130 0x7f5fd41474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.654 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fd40ca530 0x7f5fd415cff0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.654 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 --2- 172.21.15.193:0/1899389737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd40c4c80 0x7f5fd40b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.654 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 >> 172.21.15.193:0/1899389737 conn(0x7f5fd4073ca0 msgr2=0x7f5fd4145880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:43.655 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 shutdown_connections 2023-11-28T18:22:43.655 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.649+0000 7f5fdba46640 1 -- 172.21.15.193:0/1899389737 wait complete. 2023-11-28T18:22:43.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new a436ddaf-975a-4147-8e22-a17992126b07 2 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc9236ea640 1 Processor -- start 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc9236ea640 1 -- start start 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc9236ea640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc91c153f10 0x7fc91c15d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc9236ea640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91c0ca120 con 0x7fc91c0ca2e0 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc9236ea640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91c15ad20 con 0x7fc91c159d30 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc9236ea640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91c15a1c0 con 0x7fc91c153f10 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc920c85640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc91c159d30 0x7fc91c0cf630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:36460/0 (socket says 172.21.15.193:36460) 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc920c85640 1 -- 172.21.15.193:0/2596858909 learned_addr learned my addr 172.21.15.193:0/2596858909 (peer_addr_for_me v1:172.21.15.193:0/0) 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc921486640 1 --2- 172.21.15.193:0/2596858909 >> v2:172.21.15.7:3300/0 conn(0x7fc91c153f10 0x7fc91c15d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.811 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc913fff640 1 -- 172.21.15.193:0/2596858909 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089975216 0 0) 0x7fc91c15ad20 con 0x7fc91c159d30 2023-11-28T18:22:43.812 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc913fff640 1 -- 172.21.15.193:0/2596858909 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc908003530 con 0x7fc91c159d30 2023-11-28T18:22:43.812 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc913fff640 1 -- 172.21.15.193:0/2596858909 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981227744 0 0) 0x7fc91c0ca120 con 0x7fc91c0ca2e0 2023-11-28T18:22:43.812 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc913fff640 1 -- 172.21.15.193:0/2596858909 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc91c15ad20 con 0x7fc91c0ca2e0 2023-11-28T18:22:43.812 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc921486640 1 -- 172.21.15.193:0/2596858909 >> v1:172.21.15.193:6789/0 conn(0x7fc91c159d30 legacy=0x7fc91c0cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.812 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc913fff640 1 -- 172.21.15.193:0/2596858909 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4270472529 0 0) 0x7fc908003530 con 0x7fc91c159d30 2023-11-28T18:22:43.812 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc921486640 1 -- 172.21.15.193:0/2596858909 >> v1:172.21.15.7:6789/0 conn(0x7fc91c0ca2e0 legacy=0x7fc91c1646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.812 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.805+0000 7fc921486640 1 -- 172.21.15.193:0/2596858909 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91c15a480 con 0x7fc91c153f10 2023-11-28T18:22:43.814 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc921486640 1 --2- 172.21.15.193:0/2596858909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c153f10 0x7fc91c15d970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc918009770 tx=0x7fc91801c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75989909946dda1d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:43.814 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc913fff640 1 -- 172.21.15.193:0/2596858909 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c001680 con 0x7fc91c153f10 2023-11-28T18:22:43.814 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc913fff640 1 -- 172.21.15.193:0/2596858909 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9180a7050 con 0x7fc91c153f10 2023-11-28T18:22:43.814 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc913fff640 1 -- 172.21.15.193:0/2596858909 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9180ab620 con 0x7fc91c153f10 2023-11-28T18:22:43.815 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc9236ea640 1 -- 172.21.15.193:0/2596858909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c153f10 msgr2=0x7fc91c15d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.815 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc9236ea640 1 --2- 172.21.15.193:0/2596858909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c153f10 0x7fc91c15d970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc918009770 tx=0x7fc91801c310 comp rx=0 tx=0).stop 2023-11-28T18:22:43.815 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc9236ea640 1 -- 172.21.15.193:0/2596858909 shutdown_connections 2023-11-28T18:22:43.815 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc9236ea640 1 --2- 172.21.15.193:0/2596858909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c153f10 0x7fc91c15d970 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.815 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.809+0000 7fc9236ea640 1 -- 172.21.15.193:0/2596858909 >> 172.21.15.193:0/2596858909 conn(0x7fc91c0662e0 msgr2=0x7fc91c0666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:43.815 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 -- 172.21.15.193:0/2596858909 shutdown_connections 2023-11-28T18:22:43.815 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 -- 172.21.15.193:0/2596858909 wait complete. 2023-11-28T18:22:43.816 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 Processor -- start 2023-11-28T18:22:43.816 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 -- start start 2023-11-28T18:22:43.818 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c0ca2e0 0x7fc91c0c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.818 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc91c153f10 0x7fc91c147df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.818 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc91c0c7450 0x7fc91c142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:43.818 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c1470c0 con 0x7fc91c0ca2e0 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c147960 con 0x7fc91c153f10 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c149940 con 0x7fc91c0c7450 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc91c153f10 0x7fc91c147df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc921486640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c0ca2e0 0x7fc91c0c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc91c153f10 0x7fc91c147df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50544/0 (socket says 172.21.15.193:50544) 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 -- 172.21.15.193:0/2258106903 learned_addr learned my addr 172.21.15.193:0/2258106903 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc921c87640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc91c0c7450 0x7fc91c142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 -- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc91c0c7450 msgr2=0x7fc91c142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc91c0c7450 0x7fc91c142de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 -- 172.21.15.193:0/2258106903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c0ca2e0 msgr2=0x7fc91c0c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c0ca2e0 0x7fc91c0c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 -- 172.21.15.193:0/2258106903 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc918009420 con 0x7fc91c153f10 2023-11-28T18:22:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc921c87640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc91c0c7450 0x7fc91c142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:22:43.820 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc920c85640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc91c153f10 0x7fc91c147df0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00c760 tx=0x7fc90c00cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:43.820 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc911ffb640 1 -- 172.21.15.193:0/2258106903 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c0072d0 con 0x7fc91c153f10 2023-11-28T18:22:43.820 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc911ffb640 1 -- 172.21.15.193:0/2258106903 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90c007450 con 0x7fc91c153f10 2023-11-28T18:22:43.821 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.813+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91c149bb0 con 0x7fc91c153f10 2023-11-28T18:22:43.821 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.817+0000 7fc911ffb640 1 -- 172.21.15.193:0/2258106903 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c0047f0 con 0x7fc91c153f10 2023-11-28T18:22:43.821 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.817+0000 7fc911ffb640 1 -- 172.21.15.193:0/2258106903 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc90c004c50 con 0x7fc91c153f10 2023-11-28T18:22:43.821 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.817+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91c147310 con 0x7fc91c153f10 2023-11-28T18:22:43.821 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.817+0000 7fc911ffb640 1 -- 172.21.15.193:0/2258106903 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fc90c00aa60 con 0x7fc91c153f10 2023-11-28T18:22:43.821 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.817+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91c04e890 con 0x7fc91c153f10 2023-11-28T18:22:43.822 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.817+0000 7fc911ffb640 1 -- 172.21.15.193:0/2258106903 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc90c018070 con 0x7fc91c153f10 2023-11-28T18:22:43.889 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.885+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a436ddaf-975a-4147-8e22-a17992126b07", "id": 2} v 0) v1 -- 0x7fc91c147aa0 con 0x7fc91c153f10 2023-11-28T18:22:43.894 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.889+0000 7fc911ffb640 1 -- 172.21.15.193:0/2258106903 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a436ddaf-975a-4147-8e22-a17992126b07", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc90c00a640 con 0x7fc91c153f10 2023-11-28T18:22:43.894 INFO:teuthology.orchestra.run.smithi193.stdout:2 2023-11-28T18:22:43.895 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.889+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc91c153f10 msgr2=0x7fc91c147df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:43.895 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.889+0000 7fc9236ea640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc91c153f10 0x7fc91c147df0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00c760 tx=0x7fc90c00cc20 comp rx=0 tx=0).stop 2023-11-28T18:22:43.895 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.893+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 shutdown_connections 2023-11-28T18:22:43.895 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.893+0000 7fc9236ea640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc91c0c7450 0x7fc91c142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.895 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.893+0000 7fc9236ea640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc91c153f10 0x7fc91c147df0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.895 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.893+0000 7fc9236ea640 1 --2- 172.21.15.193:0/2258106903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc91c0ca2e0 0x7fc91c0c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:43.895 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.893+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 >> 172.21.15.193:0/2258106903 conn(0x7fc91c0662e0 msgr2=0x7fc91c0c55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:43.896 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.893+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 shutdown_connections 2023-11-28T18:22:43.896 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:43.893+0000 7fc9236ea640 1 -- 172.21.15.193:0/2258106903 wait complete. 2023-11-28T18:22:43.914 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 857e014e-444b-407f-a223-62e7384e471c 3 2023-11-28T18:22:44.040 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.037+0000 7fd851af0180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T18:22:44.049 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.045+0000 7efef46ea640 1 Processor -- start 2023-11-28T18:22:44.050 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.045+0000 7efef46ea640 1 -- start start 2023-11-28T18:22:44.051 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.045+0000 7efef46ea640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7efeec154130 0x7efeec159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.051 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.045+0000 7efef46ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec15de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.051 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.045+0000 7efef46ea640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec0caeb0 con 0x7efeec0ca530 2023-11-28T18:22:44.051 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.045+0000 7efef46ea640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec15a3f0 con 0x7efeec154130 2023-11-28T18:22:44.051 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef46ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec15d9b0 con 0x7efeec0c4c80 2023-11-28T18:22:44.052 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef2c87640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efeec0ca530 0x7efeec0cb2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.193:48610/0 (socket says 172.21.15.193:48610) 2023-11-28T18:22:44.052 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef2c87640 1 -- 172.21.15.193:0/1277759374 learned_addr learned my addr 172.21.15.193:0/1277759374 (peer_addr_for_me v1:172.21.15.193:0/0) 2023-11-28T18:22:44.052 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef1c85640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.052 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef2486640 1 --2- 172.21.15.193:0/1277759374 >> v2:172.21.15.193:3300/0 conn(0x7efeec154130 0x7efeec159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.053 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef1484640 1 -- 172.21.15.193:0/1277759374 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650130465 0 0) 0x7efeec0caeb0 con 0x7efeec0ca530 2023-11-28T18:22:44.053 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef1484640 1 -- 172.21.15.193:0/1277759374 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed8003530 con 0x7efeec0ca530 2023-11-28T18:22:44.053 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef1c85640 1 -- 172.21.15.193:0/1277759374 >> v2:172.21.15.193:3300/0 conn(0x7efeec154130 msgr2=0x7efeec159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:44.053 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef1c85640 1 --2- 172.21.15.193:0/1277759374 >> v2:172.21.15.193:3300/0 conn(0x7efeec154130 0x7efeec159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.053 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef1c85640 1 -- 172.21.15.193:0/1277759374 >> v1:172.21.15.7:6789/0 conn(0x7efeec0ca530 legacy=0x7efeec0cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.053 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef1c85640 1 -- 172.21.15.193:0/1277759374 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec15e380 con 0x7efeec0c4c80 2023-11-28T18:22:44.053 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.049+0000 7efef2486640 1 --2- 172.21.15.193:0/1277759374 >> v2:172.21.15.193:3300/0 conn(0x7efeec154130 0x7efeec159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:22:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef1c85640 1 --2- 172.21.15.193:0/1277759374 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec15de40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efedc0097c0 tx=0x7efedc01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8979f161a3c7033e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef1484640 1 -- 172.21.15.193:0/1277759374 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc0a8970 con 0x7efeec0c4c80 2023-11-28T18:22:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef1484640 1 -- 172.21.15.193:0/1277759374 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc0a8ad0 con 0x7efeec0c4c80 2023-11-28T18:22:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef1484640 1 -- 172.21.15.193:0/1277759374 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc0a8e60 con 0x7efeec0c4c80 2023-11-28T18:22:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef46ea640 1 -- 172.21.15.193:0/1277759374 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 msgr2=0x7efeec15de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef46ea640 1 --2- 172.21.15.193:0/1277759374 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec15de40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efedc0097c0 tx=0x7efedc01ec70 comp rx=0 tx=0).stop 2023-11-28T18:22:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef46ea640 1 -- 172.21.15.193:0/1277759374 shutdown_connections 2023-11-28T18:22:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef46ea640 1 --2- 172.21.15.193:0/1277759374 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec15de40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef46ea640 1 --2- 172.21.15.193:0/1277759374 >> v2:172.21.15.193:3300/0 conn(0x7efeec154130 0x7efeec159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef46ea640 1 -- 172.21.15.193:0/1277759374 >> 172.21.15.193:0/1277759374 conn(0x7efeec073ca0 msgr2=0x7efeec071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef46ea640 1 -- 172.21.15.193:0/1277759374 shutdown_connections 2023-11-28T18:22:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.053+0000 7efef46ea640 1 -- 172.21.15.193:0/1277759374 wait complete. 2023-11-28T18:22:44.060 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 Processor -- start 2023-11-28T18:22:44.060 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 -- start start 2023-11-28T18:22:44.061 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efeec0ca530 0x7efeec0c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeec154130 0x7efeec149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec1e7160 con 0x7efeec154130 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0c97d0 con 0x7efeec0ca530 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0c74f0 con 0x7efeec0c4c80 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef2486640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef1c85640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efeec0ca530 0x7efeec0c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef2486640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:41636/0 (socket says 172.21.15.193:41636) 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef2486640 1 -- 172.21.15.193:0/1505315926 learned_addr learned my addr 172.21.15.193:0/1505315926 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef2c87640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeec154130 0x7efeec149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef1c85640 1 -- 172.21.15.193:0/1505315926 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 msgr2=0x7efeec159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.062 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef1c85640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.063 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef1c85640 1 -- 172.21.15.193:0/1505315926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeec154130 msgr2=0x7efeec149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.063 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef1c85640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeec154130 0x7efeec149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.063 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef1c85640 1 -- 172.21.15.193:0/1505315926 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8009420 con 0x7efeec0ca530 2023-11-28T18:22:44.063 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef2486640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef1c85640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efeec0ca530 0x7efeec0c7f40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efedc0af000 tx=0x7efedc01ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efee2ffd640 1 -- 172.21.15.193:0/1505315926 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc0bd070 con 0x7efeec0ca530 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efee2ffd640 1 -- 172.21.15.193:0/1505315926 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc0b0ba0 con 0x7efeec0ca530 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efedc009420 con 0x7efeec0ca530 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efee2ffd640 1 -- 172.21.15.193:0/1505315926 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc0b9430 con 0x7efeec0ca530 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec0c9d00 con 0x7efeec0ca530 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efee2ffd640 1 -- 172.21.15.193:0/1505315926 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efedc0b7040 con 0x7efeec0ca530 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.057+0000 7efee2ffd640 1 -- 172.21.15.193:0/1505315926 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7efedc0b9d20 con 0x7efeec0ca530 2023-11-28T18:22:44.064 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.061+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec04e890 con 0x7efeec0ca530 2023-11-28T18:22:44.067 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.065+0000 7efee2ffd640 1 -- 172.21.15.193:0/1505315926 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7efedc0b5070 con 0x7efeec0ca530 2023-11-28T18:22:44.105 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.101+0000 7fd851af0180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T18:22:44.134 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.129+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "857e014e-444b-407f-a223-62e7384e471c", "id": 3} v 0) v1 -- 0x7efeec15dfd0 con 0x7efeec0ca530 2023-11-28T18:22:44.140 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efee2ffd640 1 -- 172.21.15.193:0/1505315926 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "857e014e-444b-407f-a223-62e7384e471c", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7efedc0b9890 con 0x7efeec0ca530 2023-11-28T18:22:44.140 INFO:teuthology.orchestra.run.smithi193.stdout:3 2023-11-28T18:22:44.141 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efeec0ca530 msgr2=0x7efeec0c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.141 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efeec0ca530 0x7efeec0c7f40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efedc0af000 tx=0x7efedc01ba10 comp rx=0 tx=0).stop 2023-11-28T18:22:44.141 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 shutdown_connections 2023-11-28T18:22:44.141 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeec154130 0x7efeec149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.141 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efeec0ca530 0x7efeec0c7f40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.141 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 --2- 172.21.15.193:0/1505315926 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efeec0c4c80 0x7efeec159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.142 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 >> 172.21.15.193:0/1505315926 conn(0x7efeec073ca0 msgr2=0x7efeec152590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:44.142 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 shutdown_connections 2023-11-28T18:22:44.142 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.137+0000 7efef46ea640 1 -- 172.21.15.193:0/1505315926 wait complete. 2023-11-28T18:22:44.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new b6d20668-0212-4e4f-84fd-968ea13370b7 4 2023-11-28T18:22:44.169 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.165+0000 7fd851af0180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T18:22:44.170 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.165+0000 7fd4c8831180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T18:22:44.234 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.229+0000 7fd4c8831180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T18:22:44.262 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.257+0000 7fd851af0180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T18:22:44.296 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.293+0000 7fd4c8831180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T18:22:44.297 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3e73e640 1 Processor -- start 2023-11-28T18:22:44.297 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3e73e640 1 -- start start 2023-11-28T18:22:44.298 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3e73e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6a38153f10 0x7f6a3815d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.298 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3e73e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6a380ca2e0 0x7f6a3815deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3e73e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a380ca120 con 0x7f6a38159d30 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3e73e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3815ad20 con 0x7f6a38153f10 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3e73e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3815a1c0 con 0x7f6a380ca2e0 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3ccdb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6a380ca2e0 0x7f6a3815deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3ccdb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6a380ca2e0 0x7f6a3815deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50578/0 (socket says 172.21.15.193:50578) 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3ccdb640 1 -- 172.21.15.193:0/913445286 learned_addr learned my addr 172.21.15.193:0/913445286 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a37fff640 1 --2- 172.21.15.193:0/913445286 >> v2:172.21.15.7:3300/0 conn(0x7f6a38153f10 0x7f6a3815d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3ccdb640 1 -- 172.21.15.193:0/913445286 >> v2:172.21.15.7:3300/0 conn(0x7f6a38153f10 msgr2=0x7f6a3815d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3ccdb640 1 --2- 172.21.15.193:0/913445286 >> v2:172.21.15.7:3300/0 conn(0x7f6a38153f10 0x7f6a3815d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3ccdb640 1 -- 172.21.15.193:0/913445286 >> v1:172.21.15.7:6789/0 conn(0x7f6a38159d30 legacy=0x7f6a380cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.299 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.293+0000 7f6a3ccdb640 1 -- 172.21.15.193:0/913445286 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3815a480 con 0x7f6a380ca2e0 2023-11-28T18:22:44.301 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3ccdb640 1 --2- 172.21.15.193:0/913445286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a380ca2e0 0x7f6a3815deb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6a380b2bc0 tx=0x7f6a3001c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce4db9156833b70 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:44.301 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a36ffd640 1 -- 172.21.15.193:0/913445286 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a300a5770 con 0x7f6a380ca2e0 2023-11-28T18:22:44.302 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a36ffd640 1 -- 172.21.15.193:0/913445286 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a300a58d0 con 0x7f6a380ca2e0 2023-11-28T18:22:44.302 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a36ffd640 1 -- 172.21.15.193:0/913445286 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a300a5c60 con 0x7f6a380ca2e0 2023-11-28T18:22:44.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3e73e640 1 -- 172.21.15.193:0/913445286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a380ca2e0 msgr2=0x7f6a3815deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3e73e640 1 --2- 172.21.15.193:0/913445286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a380ca2e0 0x7f6a3815deb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6a380b2bc0 tx=0x7f6a3001c9a0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3e73e640 1 -- 172.21.15.193:0/913445286 shutdown_connections 2023-11-28T18:22:44.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3e73e640 1 --2- 172.21.15.193:0/913445286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a380ca2e0 0x7f6a3815deb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3e73e640 1 --2- 172.21.15.193:0/913445286 >> v2:172.21.15.7:3300/0 conn(0x7f6a38153f10 0x7f6a3815d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3e73e640 1 -- 172.21.15.193:0/913445286 >> 172.21.15.193:0/913445286 conn(0x7f6a380662e0 msgr2=0x7f6a380666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:44.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3e73e640 1 -- 172.21.15.193:0/913445286 shutdown_connections 2023-11-28T18:22:44.303 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.297+0000 7f6a3e73e640 1 -- 172.21.15.193:0/913445286 wait complete. 2023-11-28T18:22:44.304 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 Processor -- start 2023-11-28T18:22:44.304 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 -- start start 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a380ca2e0 0x7f6a3815c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a38153f10 0x7f6a381ecf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a381f3e20 0x7f6a381ed470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a381f42f0 con 0x7f6a380ca2e0 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a381ebe90 con 0x7f6a38153f10 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a381ec170 con 0x7f6a381f3e20 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a377fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a38153f10 0x7f6a381ecf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3ccdb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a381f3e20 0x7f6a381ed470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a380ca2e0 0x7f6a3815c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.306 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a380ca2e0 0x7f6a3815c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.193:52346/0 (socket says 172.21.15.193:52346) 2023-11-28T18:22:44.307 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3ccdb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a381f3e20 0x7f6a381ed470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:41644/0 (socket says 172.21.15.193:41644) 2023-11-28T18:22:44.307 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a377fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a38153f10 0x7f6a381ecf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50584/0 (socket says 172.21.15.193:50584) 2023-11-28T18:22:44.307 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a37fff640 1 -- 172.21.15.193:0/3486256586 learned_addr learned my addr 172.21.15.193:0/3486256586 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a377fe640 1 -- 172.21.15.193:0/3486256586 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a381f3e20 msgr2=0x7f6a381ed470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a377fe640 1 --2- 172.21.15.193:0/3486256586 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a381f3e20 0x7f6a381ed470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a377fe640 1 -- 172.21.15.193:0/3486256586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a380ca2e0 msgr2=0x7f6a3815c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a377fe640 1 --2- 172.21.15.193:0/3486256586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a380ca2e0 0x7f6a3815c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a377fe640 1 -- 172.21.15.193:0/3486256586 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a30009b30 con 0x7f6a38153f10 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3ccdb640 1 --2- 172.21.15.193:0/3486256586 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a381f3e20 0x7f6a381ed470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a377fe640 1 --2- 172.21.15.193:0/3486256586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a38153f10 0x7f6a381ecf30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c000f70 tx=0x7f6a2c00c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a34ff9640 1 -- 172.21.15.193:0/3486256586 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c0052d0 con 0x7f6a38153f10 2023-11-28T18:22:44.309 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a34ff9640 1 -- 172.21.15.193:0/3486256586 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c005430 con 0x7f6a38153f10 2023-11-28T18:22:44.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a34ff9640 1 -- 172.21.15.193:0/3486256586 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c003840 con 0x7f6a38153f10 2023-11-28T18:22:44.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.301+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a381ec450 con 0x7f6a38153f10 2023-11-28T18:22:44.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.305+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a381eca10 con 0x7f6a38153f10 2023-11-28T18:22:44.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.305+0000 7f6a34ff9640 1 -- 172.21.15.193:0/3486256586 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6a2c003ca0 con 0x7f6a38153f10 2023-11-28T18:22:44.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.305+0000 7f6a34ff9640 1 -- 172.21.15.193:0/3486256586 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f6a2c012930 con 0x7f6a38153f10 2023-11-28T18:22:44.310 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.305+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a381ecc30 con 0x7f6a38153f10 2023-11-28T18:22:44.311 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.305+0000 7f6a34ff9640 1 -- 172.21.15.193:0/3486256586 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f6a2c017070 con 0x7f6a38153f10 2023-11-28T18:22:44.325 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.321+0000 7fd851af0180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T18:22:44.380 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.377+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b6d20668-0212-4e4f-84fd-968ea13370b7", "id": 4} v 0) v1 -- 0x7f6a3804e890 con 0x7f6a38153f10 2023-11-28T18:22:44.386 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.381+0000 7f6a34ff9640 1 -- 172.21.15.193:0/3486256586 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6d20668-0212-4e4f-84fd-968ea13370b7", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6a2c012bc0 con 0x7f6a38153f10 2023-11-28T18:22:44.386 INFO:teuthology.orchestra.run.smithi193.stdout:4 2023-11-28T18:22:44.387 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.381+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a38153f10 msgr2=0x7f6a381ecf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.387 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.381+0000 7f6a3e73e640 1 --2- 172.21.15.193:0/3486256586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a38153f10 0x7f6a381ecf30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c000f70 tx=0x7f6a2c00c820 comp rx=0 tx=0).stop 2023-11-28T18:22:44.387 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.381+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 shutdown_connections 2023-11-28T18:22:44.388 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.381+0000 7f6a3e73e640 1 --2- 172.21.15.193:0/3486256586 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a381f3e20 0x7f6a381ed470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.389 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.381+0000 7f6a3e73e640 1 --2- 172.21.15.193:0/3486256586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a38153f10 0x7f6a381ecf30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.389 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.381+0000 7f6a3e73e640 1 --2- 172.21.15.193:0/3486256586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a380ca2e0 0x7f6a3815c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.389 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.381+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 >> 172.21.15.193:0/3486256586 conn(0x7f6a380662e0 msgr2=0x7f6a380666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:44.389 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.385+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 shutdown_connections 2023-11-28T18:22:44.389 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.385+0000 7f6a3e73e640 1 -- 172.21.15.193:0/3486256586 wait complete. 2023-11-28T18:22:44.391 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.389+0000 7fd4c8831180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T18:22:44.406 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new d01156aa-e11d-4fec-9f46-8fcd16feb3bb 5 2023-11-28T18:22:44.420 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.417+0000 7fd851af0180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T18:22:44.455 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.449+0000 7fd4c8831180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T18:22:44.482 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.481+0000 7fd851af0180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T18:22:44.545 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.541+0000 7fd4c8831180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T18:22:44.588 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d837c4640 1 Processor -- start 2023-11-28T18:22:44.588 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d837c4640 1 -- start start 2023-11-28T18:22:44.590 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d837c4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9d7c154130 0x7f9d7c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.590 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d837c4640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9d7c0ca530 0x7f9d7c0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.590 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d837c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7c0caeb0 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.590 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d837c4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c15a3f0 con 0x7f9d7c154130 2023-11-28T18:22:44.590 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d837c4640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c15d9b0 con 0x7f9d7c0ca530 2023-11-28T18:22:44.590 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d81d61640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9d7c0ca530 0x7f9d7c0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.591 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d81d61640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9d7c0ca530 0x7f9d7c0cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50586/0 (socket says 172.21.15.193:50586) 2023-11-28T18:22:44.591 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d81d61640 1 -- 172.21.15.193:0/568533723 learned_addr learned my addr 172.21.15.193:0/568533723 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:44.591 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d81560640 1 --2- 172.21.15.193:0/568533723 >> v2:172.21.15.7:3300/0 conn(0x7f9d7c154130 0x7f9d7c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.591 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d81d61640 1 -- 172.21.15.193:0/568533723 >> v2:172.21.15.7:3300/0 conn(0x7f9d7c154130 msgr2=0x7f9d7c159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:44.592 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d81d61640 1 --2- 172.21.15.193:0/568533723 >> v2:172.21.15.7:3300/0 conn(0x7f9d7c154130 0x7f9d7c159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.592 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d81d61640 1 -- 172.21.15.193:0/568533723 >> v1:172.21.15.7:6789/0 conn(0x7f9d7c0c4c80 legacy=0x7f9d7c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.592 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.585+0000 7f9d81d61640 1 -- 172.21.15.193:0/568533723 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c15a5d0 con 0x7f9d7c0ca530 2023-11-28T18:22:44.593 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d81d61640 1 --2- 172.21.15.193:0/568533723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c0ca530 0x7f9d7c0cb2b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0cd650 tx=0x7f9d7401f1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93909ad5a70a651f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:44.594 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d73fff640 1 -- 172.21.15.193:0/568533723 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740af400 con 0x7f9d7c0ca530 2023-11-28T18:22:44.594 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d73fff640 1 -- 172.21.15.193:0/568533723 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d740ab050 con 0x7f9d7c0ca530 2023-11-28T18:22:44.594 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d73fff640 1 -- 172.21.15.193:0/568533723 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740af780 con 0x7f9d7c0ca530 2023-11-28T18:22:44.595 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d837c4640 1 -- 172.21.15.193:0/568533723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c0ca530 msgr2=0x7f9d7c0cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.595 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d837c4640 1 --2- 172.21.15.193:0/568533723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c0ca530 0x7f9d7c0cb2b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0cd650 tx=0x7f9d7401f1c0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.595 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d837c4640 1 -- 172.21.15.193:0/568533723 shutdown_connections 2023-11-28T18:22:44.595 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d837c4640 1 --2- 172.21.15.193:0/568533723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c0ca530 0x7f9d7c0cb2b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.595 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d837c4640 1 --2- 172.21.15.193:0/568533723 >> v2:172.21.15.7:3300/0 conn(0x7f9d7c154130 0x7f9d7c159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.595 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d837c4640 1 -- 172.21.15.193:0/568533723 >> 172.21.15.193:0/568533723 conn(0x7f9d7c073ca0 msgr2=0x7f9d7c071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:44.595 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d837c4640 1 -- 172.21.15.193:0/568533723 shutdown_connections 2023-11-28T18:22:44.595 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.589+0000 7f9d837c4640 1 -- 172.21.15.193:0/568533723 wait complete. 2023-11-28T18:22:44.596 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 Processor -- start 2023-11-28T18:22:44.596 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 -- start start 2023-11-28T18:22:44.598 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d7c0c4c80 0x7f9d7c1e6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.599 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d7c0ca530 0x7f9d7c0cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.599 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c154130 0x7f9d7c14a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.599 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0cd740 con 0x7f9d7c0ca530 2023-11-28T18:22:44.599 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c1e5c60 con 0x7f9d7c154130 2023-11-28T18:22:44.599 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c1e3f60 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.599 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d7c0c4c80 0x7f9d7c1e6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d7c0c4c80 0x7f9d7c1e6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:41660/0 (socket says 172.21.15.193:41660) 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 -- 172.21.15.193:0/789591655 learned_addr learned my addr 172.21.15.193:0/789591655 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d80d5f640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d7c0ca530 0x7f9d7c0cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81d61640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c154130 0x7f9d7c14a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 -- 172.21.15.193:0/789591655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c154130 msgr2=0x7f9d7c14a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c154130 0x7f9d7c14a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 -- 172.21.15.193:0/789591655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d7c0ca530 msgr2=0x7f9d7c0cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d7c0ca530 0x7f9d7c0cfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 -- 172.21.15.193:0/789591655 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d74009b30 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.600 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d81560640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d7c0c4c80 0x7f9d7c1e6b00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800e8a0 tx=0x7f9d7800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:44.601 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d71ffb640 1 -- 172.21.15.193:0/789591655 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d78009510 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.601 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d71ffb640 1 -- 172.21.15.193:0/789591655 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7800c940 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.601 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d71ffb640 1 -- 172.21.15.193:0/789591655 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7800ccb0 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.601 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.593+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c1e4240 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.601 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.597+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c1e32c0 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.602 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.597+0000 7f9d71ffb640 1 -- 172.21.15.193:0/789591655 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9d78005ca0 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.602 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.597+0000 7f9d71ffb640 1 -- 172.21.15.193:0/789591655 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f9d7800ce10 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.602 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.597+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c0cd9b0 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.602 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.597+0000 7f9d71ffb640 1 -- 172.21.15.193:0/789591655 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9d78021060 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.608 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.605+0000 7fd4c8831180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T18:22:44.647 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.645+0000 7fd851af0180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T18:22:44.668 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.665+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d01156aa-e11d-4fec-9f46-8fcd16feb3bb", "id": 5} v 0) v1 -- 0x7f9d7c04e890 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.673 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d71ffb640 1 -- 172.21.15.193:0/789591655 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d01156aa-e11d-4fec-9f46-8fcd16feb3bb", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9d78029020 con 0x7f9d7c0c4c80 2023-11-28T18:22:44.673 INFO:teuthology.orchestra.run.smithi193.stdout:5 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d7c0c4c80 msgr2=0x7f9d7c1e6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d7c0c4c80 0x7f9d7c1e6b00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800e8a0 tx=0x7f9d7800eca0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 shutdown_connections 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d7c154130 0x7f9d7c14a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d7c0ca530 0x7f9d7c0cfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 --2- 172.21.15.193:0/789591655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d7c0c4c80 0x7f9d7c1e6b00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 >> 172.21.15.193:0/789591655 conn(0x7f9d7c073ca0 msgr2=0x7f9d7c151960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 shutdown_connections 2023-11-28T18:22:44.675 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.669+0000 7f9d837c4640 1 -- 172.21.15.193:0/789591655 wait complete. 2023-11-28T18:22:44.693 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 2846c4d0-bec5-4f62-a048-e8a0c163134d 6 2023-11-28T18:22:44.719 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:44.717+0000 7fd851af0180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T18:22:44.770 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.765+0000 7fd4c8831180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T18:22:44.829 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa0aebb640 1 Processor -- start 2023-11-28T18:22:44.829 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa0aebb640 1 -- start start 2023-11-28T18:22:44.830 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa0aebb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040ca2e0 0x7ffa040cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.830 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa0aebb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa040ca120 con 0x7ffa04153f10 2023-11-28T18:22:44.831 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa0aebb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0415ad20 con 0x7ffa04159d30 2023-11-28T18:22:44.831 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa0aebb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0415a1c0 con 0x7ffa040ca2e0 2023-11-28T18:22:44.831 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa03fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ffa04159d30 0x7ffa040cf630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:36476/0 (socket says 172.21.15.193:36476) 2023-11-28T18:22:44.831 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa03fff640 1 -- 172.21.15.193:0/4052502564 learned_addr learned my addr 172.21.15.193:0/4052502564 (peer_addr_for_me v1:172.21.15.193:0/0) 2023-11-28T18:22:44.831 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa09458640 1 --2- 172.21.15.193:0/4052502564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040ca2e0 0x7ffa040cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.832 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.825+0000 7ffa037fe640 1 -- 172.21.15.193:0/4052502564 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855191631 0 0) 0x7ffa0415ad20 con 0x7ffa04159d30 2023-11-28T18:22:44.832 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.829+0000 7ffa037fe640 1 -- 172.21.15.193:0/4052502564 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f4003530 con 0x7ffa04159d30 2023-11-28T18:22:44.832 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.829+0000 7ffa037fe640 1 -- 172.21.15.193:0/4052502564 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707190704 0 0) 0x7ffa040ca120 con 0x7ffa04153f10 2023-11-28T18:22:44.833 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.829+0000 7ffa037fe640 1 -- 172.21.15.193:0/4052502564 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa0415ad20 con 0x7ffa04153f10 2023-11-28T18:22:44.833 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.829+0000 7ffa09458640 1 -- 172.21.15.193:0/4052502564 >> v1:172.21.15.193:6789/0 conn(0x7ffa04159d30 legacy=0x7ffa040cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.833 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.829+0000 7ffa09458640 1 -- 172.21.15.193:0/4052502564 >> v1:172.21.15.7:6789/0 conn(0x7ffa04153f10 legacy=0x7ffa0415d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.833 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.829+0000 7ffa09458640 1 -- 172.21.15.193:0/4052502564 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0415a480 con 0x7ffa040ca2e0 2023-11-28T18:22:44.836 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa09458640 1 --2- 172.21.15.193:0/4052502564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040ca2e0 0x7ffa040cfd40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009dc0 tx=0x7ff9fc01c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34b02c0c6177da75 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:44.836 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa037fe640 1 -- 172.21.15.193:0/4052502564 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0a5a00 con 0x7ffa040ca2e0 2023-11-28T18:22:44.836 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa037fe640 1 -- 172.21.15.193:0/4052502564 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9fc0a5b60 con 0x7ffa040ca2e0 2023-11-28T18:22:44.836 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa037fe640 1 -- 172.21.15.193:0/4052502564 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0ad440 con 0x7ffa040ca2e0 2023-11-28T18:22:44.837 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 -- 172.21.15.193:0/4052502564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040ca2e0 msgr2=0x7ffa040cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.837 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 --2- 172.21.15.193:0/4052502564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040ca2e0 0x7ffa040cfd40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009dc0 tx=0x7ff9fc01c940 comp rx=0 tx=0).stop 2023-11-28T18:22:44.837 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 -- 172.21.15.193:0/4052502564 shutdown_connections 2023-11-28T18:22:44.837 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 --2- 172.21.15.193:0/4052502564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040ca2e0 0x7ffa040cfd40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.837 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 -- 172.21.15.193:0/4052502564 >> 172.21.15.193:0/4052502564 conn(0x7ffa040662e0 msgr2=0x7ffa040666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:44.838 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 -- 172.21.15.193:0/4052502564 shutdown_connections 2023-11-28T18:22:44.838 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 -- 172.21.15.193:0/4052502564 wait complete. 2023-11-28T18:22:44.839 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 Processor -- start 2023-11-28T18:22:44.839 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.833+0000 7ffa0aebb640 1 -- start start 2023-11-28T18:22:44.841 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa040ca2e0 0x7ffa040c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.841 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa04153f10 0x7ffa04146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.841 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040c8860 0x7ffa04147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:44.841 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa040c7a50 con 0x7ffa040ca2e0 2023-11-28T18:22:44.842 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0414e1c0 con 0x7ffa04153f10 2023-11-28T18:22:44.842 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0414f0b0 con 0x7ffa040c8860 2023-11-28T18:22:44.842 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa09458640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040c8860 0x7ffa04147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.843 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa03fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa04153f10 0x7ffa04146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.843 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa09458640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040c8860 0x7ffa04147460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:41686/0 (socket says 172.21.15.193:41686) 2023-11-28T18:22:44.843 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa09458640 1 -- 172.21.15.193:0/2696723467 learned_addr learned my addr 172.21.15.193:0/2696723467 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:44.843 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa08c57640 1 --2- 172.21.15.193:0/2696723467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa040ca2e0 0x7ffa040c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:44.844 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa03fff640 1 -- 172.21.15.193:0/2696723467 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040c8860 msgr2=0x7ffa04147460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.844 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa03fff640 1 --2- 172.21.15.193:0/2696723467 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040c8860 0x7ffa04147460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.844 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa03fff640 1 -- 172.21.15.193:0/2696723467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa040ca2e0 msgr2=0x7ffa040c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.844 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa03fff640 1 --2- 172.21.15.193:0/2696723467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa040ca2e0 0x7ffa040c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.844 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa03fff640 1 -- 172.21.15.193:0/2696723467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9fc009a70 con 0x7ffa04153f10 2023-11-28T18:22:44.845 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa03fff640 1 --2- 172.21.15.193:0/2696723467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa04153f10 0x7ffa04146f20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c390 tx=0x7ff9f000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:44.846 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa017fa640 1 -- 172.21.15.193:0/2696723467 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0005cf0 con 0x7ffa04153f10 2023-11-28T18:22:44.846 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa017fa640 1 -- 172.21.15.193:0/2696723467 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f0005e50 con 0x7ffa04153f10 2023-11-28T18:22:44.846 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa017fa640 1 -- 172.21.15.193:0/2696723467 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0007310 con 0x7ffa04153f10 2023-11-28T18:22:44.846 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0414f390 con 0x7ffa04153f10 2023-11-28T18:22:44.846 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0414e3d0 con 0x7ffa04153f10 2023-11-28T18:22:44.846 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa017fa640 1 -- 172.21.15.193:0/2696723467 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff9f001c440 con 0x7ffa04153f10 2023-11-28T18:22:44.846 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa017fa640 1 -- 172.21.15.193:0/2696723467 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ff9f0012a40 con 0x7ffa04153f10 2023-11-28T18:22:44.847 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.837+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0404e890 con 0x7ffa04153f10 2023-11-28T18:22:44.847 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:44.841+0000 7fd4c8831180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T18:22:44.847 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.841+0000 7ffa017fa640 1 -- 172.21.15.193:0/2696723467 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff9f0017070 con 0x7ffa04153f10 2023-11-28T18:22:44.911 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.909+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2846c4d0-bec5-4f62-a048-e8a0c163134d", "id": 6} v 0) v1 -- 0x7ffa040c4790 con 0x7ffa04153f10 2023-11-28T18:22:44.917 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa017fa640 1 -- 172.21.15.193:0/2696723467 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2846c4d0-bec5-4f62-a048-e8a0c163134d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff9f0012520 con 0x7ffa04153f10 2023-11-28T18:22:44.917 INFO:teuthology.orchestra.run.smithi193.stdout:6 2023-11-28T18:22:44.918 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa04153f10 msgr2=0x7ffa04146f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:44.919 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 --2- 172.21.15.193:0/2696723467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa04153f10 0x7ffa04146f20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c390 tx=0x7ff9f000c850 comp rx=0 tx=0).stop 2023-11-28T18:22:44.919 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 shutdown_connections 2023-11-28T18:22:44.919 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 --2- 172.21.15.193:0/2696723467 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa040c8860 0x7ffa04147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.919 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 --2- 172.21.15.193:0/2696723467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa04153f10 0x7ffa04146f20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.919 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 --2- 172.21.15.193:0/2696723467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa040ca2e0 0x7ffa040c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:44.919 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 >> 172.21.15.193:0/2696723467 conn(0x7ffa040662e0 msgr2=0x7ffa040c52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:44.919 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 shutdown_connections 2023-11-28T18:22:44.919 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:44.913+0000 7ffa0aebb640 1 -- 172.21.15.193:0/2696723467 wait complete. 2023-11-28T18:22:44.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5 7 2023-11-28T18:22:45.072 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7eac89a640 1 Processor -- start 2023-11-28T18:22:45.073 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7eac89a640 1 -- start start 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7eac89a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ea81540d0 0x7f7ea8072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7eac89a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7ea815db70 0x7f7ea815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7eac89a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea815ae90 con 0x7f7ea80cd560 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7eac89a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea815e4c0 con 0x7f7ea81540d0 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7eac89a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea815a370 con 0x7f7ea815db70 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7e9dd9b640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7ea815db70 0x7f7ea815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7e9dd9b640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7ea815db70 0x7f7ea815df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50610/0 (socket says 172.21.15.193:50610) 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7e9dd9b640 1 -- 172.21.15.193:0/2793991905 learned_addr learned my addr 172.21.15.193:0/2793991905 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7ea659c640 1 --2- 172.21.15.193:0/2793991905 >> v2:172.21.15.7:3300/0 conn(0x7f7ea81540d0 0x7f7ea8072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.076 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7e9dd9b640 1 -- 172.21.15.193:0/2793991905 >> v2:172.21.15.7:3300/0 conn(0x7f7ea81540d0 msgr2=0x7f7ea8072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:45.077 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7e9dd9b640 1 --2- 172.21.15.193:0/2793991905 >> v2:172.21.15.7:3300/0 conn(0x7f7ea81540d0 0x7f7ea8072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.077 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7e9dd9b640 1 -- 172.21.15.193:0/2793991905 >> v1:172.21.15.7:6789/0 conn(0x7f7ea80cd560 legacy=0x7f7ea80ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.077 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7e9dd9b640 1 -- 172.21.15.193:0/2793991905 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea815a610 con 0x7f7ea815db70 2023-11-28T18:22:45.077 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.069+0000 7f7ea659c640 1 --2- 172.21.15.193:0/2793991905 >> v2:172.21.15.7:3300/0 conn(0x7f7ea81540d0 0x7f7ea8072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7e9dd9b640 1 --2- 172.21.15.193:0/2793991905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea815db70 0x7f7ea815df80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7ea815a800 tx=0x7f7e9001c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96f95c893e7182a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7ea5d9b640 1 -- 172.21.15.193:0/2793991905 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e900ab5f0 con 0x7f7ea815db70 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7ea5d9b640 1 -- 172.21.15.193:0/2793991905 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e900ab750 con 0x7f7ea815db70 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7ea5d9b640 1 -- 172.21.15.193:0/2793991905 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e900abae0 con 0x7f7ea815db70 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7eac89a640 1 -- 172.21.15.193:0/2793991905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea815db70 msgr2=0x7f7ea815df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7eac89a640 1 --2- 172.21.15.193:0/2793991905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea815db70 0x7f7ea815df80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7ea815a800 tx=0x7f7e9001c500 comp rx=0 tx=0).stop 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7eac89a640 1 -- 172.21.15.193:0/2793991905 shutdown_connections 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7eac89a640 1 --2- 172.21.15.193:0/2793991905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea815db70 0x7f7ea815df80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7eac89a640 1 --2- 172.21.15.193:0/2793991905 >> v2:172.21.15.7:3300/0 conn(0x7f7ea81540d0 0x7f7ea8072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7eac89a640 1 -- 172.21.15.193:0/2793991905 >> 172.21.15.193:0/2793991905 conn(0x7f7ea80662e0 msgr2=0x7f7ea80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:45.080 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7eac89a640 1 -- 172.21.15.193:0/2793991905 shutdown_connections 2023-11-28T18:22:45.080 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.073+0000 7f7eac89a640 1 -- 172.21.15.193:0/2793991905 wait complete. 2023-11-28T18:22:45.080 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 Processor -- start 2023-11-28T18:22:45.080 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 -- start start 2023-11-28T18:22:45.082 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea80cd560 0x7f7ea80cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.083 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ea81540d0 0x7f7ea80c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.083 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ea815db70 0x7f7ea80c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.083 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea81473e0 con 0x7f7ea81540d0 2023-11-28T18:22:45.083 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80c5850 con 0x7f7ea80cd560 2023-11-28T18:22:45.083 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea81464f0 con 0x7f7ea815db70 2023-11-28T18:22:45.083 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea80cd560 0x7f7ea80cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.084 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea80cd560 0x7f7ea80cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50620/0 (socket says 172.21.15.193:50620) 2023-11-28T18:22:45.084 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 -- 172.21.15.193:0/2215744534 learned_addr learned my addr 172.21.15.193:0/2215744534 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-11-28T18:22:45.084 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea6d9d640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ea815db70 0x7f7ea80c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.084 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7e9dd9b640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ea81540d0 0x7f7ea80c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.084 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 -- 172.21.15.193:0/2215744534 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ea815db70 msgr2=0x7f7ea80c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.084 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ea815db70 0x7f7ea80c72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 -- 172.21.15.193:0/2215744534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ea81540d0 msgr2=0x7f7ea80c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ea81540d0 0x7f7ea80c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 -- 172.21.15.193:0/2215744534 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e900094d0 con 0x7f7ea80cd560 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7e9dd9b640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ea81540d0 0x7f7ea80c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7ea659c640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea80cd560 0x7f7ea80cc6a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009fc0 tx=0x7f7e9800d560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7e9f7fe640 1 -- 172.21.15.193:0/2215744534 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98003ba0 con 0x7f7ea80cd560 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7e9f7fe640 1 -- 172.21.15.193:0/2215744534 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9800f040 con 0x7f7ea80cd560 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea81467d0 con 0x7f7ea80cd560 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7e9f7fe640 1 -- 172.21.15.193:0/2215744534 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98004030 con 0x7f7ea80cd560 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea8146d90 con 0x7f7ea80cd560 2023-11-28T18:22:45.085 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7e9f7fe640 1 -- 172.21.15.193:0/2215744534 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7e98004490 con 0x7f7ea80cd560 2023-11-28T18:22:45.086 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.077+0000 7f7e9f7fe640 1 -- 172.21.15.193:0/2215744534 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7e98004740 con 0x7f7ea80cd560 2023-11-28T18:22:45.086 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.081+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea8147590 con 0x7f7ea80cd560 2023-11-28T18:22:45.086 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.081+0000 7f7e9f7fe640 1 -- 172.21.15.193:0/2215744534 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7e98024060 con 0x7f7ea80cd560 2023-11-28T18:22:45.160 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.157+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5", "id": 7} v 0) v1 -- 0x7f7ea804e890 con 0x7f7ea80cd560 2023-11-28T18:22:45.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7e9f7fe640 1 -- 172.21.15.193:0/2215744534 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7e9801a070 con 0x7f7ea80cd560 2023-11-28T18:22:45.170 INFO:teuthology.orchestra.run.smithi193.stdout:7 2023-11-28T18:22:45.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea80cd560 msgr2=0x7f7ea80cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea80cd560 0x7f7ea80cc6a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009fc0 tx=0x7f7e9800d560 comp rx=0 tx=0).stop 2023-11-28T18:22:45.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 shutdown_connections 2023-11-28T18:22:45.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ea815db70 0x7f7ea80c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ea81540d0 0x7f7ea80c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 --2- 172.21.15.193:0/2215744534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ea80cd560 0x7f7ea80cc6a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 >> 172.21.15.193:0/2215744534 conn(0x7f7ea80662e0 msgr2=0x7f7ea81525b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:45.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 shutdown_connections 2023-11-28T18:22:45.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-11-28T18:22:45.165+0000 7f7eac89a640 1 -- 172.21.15.193:0/2215744534 wait complete. 2023-11-28T18:22:45.201 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-28T18:22:45.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-28T18:22:45.206 INFO:tasks.ceph.osd.0:Started 2023-11-28T18:22:45.206 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-28T18:22:45.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-28T18:22:45.211 INFO:tasks.ceph.osd.1:Started 2023-11-28T18:22:45.211 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-28T18:22:45.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-28T18:22:45.214 INFO:tasks.ceph.osd.2:Started 2023-11-28T18:22:45.215 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-28T18:22:45.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-28T18:22:45.218 INFO:tasks.ceph.osd.3:Started 2023-11-28T18:22:45.219 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-28T18:22:45.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-28T18:22:45.222 INFO:tasks.ceph.osd.4:Started 2023-11-28T18:22:45.223 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-28T18:22:45.223 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-28T18:22:45.226 INFO:tasks.ceph.osd.5:Started 2023-11-28T18:22:45.226 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-28T18:22:45.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-28T18:22:45.232 INFO:tasks.ceph.osd.6:Started 2023-11-28T18:22:45.233 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-28T18:22:45.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-28T18:22:45.237 INFO:tasks.ceph.osd.7:Started 2023-11-28T18:22:45.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:22:45.329 INFO:tasks.ceph.mgr.y.smithi007.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-11-28T18:22:45.329 INFO:tasks.ceph.mgr.y.smithi007.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-11-28T18:22:45.330 INFO:tasks.ceph.mgr.y.smithi007.stderr: from numpy import show_config as show_numpy_config 2023-11-28T18:22:45.384 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:45.381+0000 7fd851af0180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T18:22:45.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.401+0000 7f3241479640 1 Processor -- start 2023-11-28T18:22:45.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.401+0000 7f3241479640 1 -- start start 2023-11-28T18:22:45.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f3241479640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f323c151cd0 0x7f323c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f3241479640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f323c157b90 0x7f323c0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f3241479640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c071e60 con 0x7f323c06e560 2023-11-28T18:22:45.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f3241479640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323c070730 con 0x7f323c151cd0 2023-11-28T18:22:45.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f3241479640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323c070930 con 0x7f323c157b90 2023-11-28T18:22:45.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f323affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f323c151cd0 0x7f323c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f323affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f323c151cd0 0x7f323c15bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38288/0 (socket says 172.21.15.7:38288) 2023-11-28T18:22:45.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f323affd640 1 -- 172.21.15.7:0/2331083147 learned_addr learned my addr 172.21.15.7:0/2331083147 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:22:45.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f323b7fe640 1 --2- 172.21.15.7:0/2331083147 >> v2:172.21.15.193:3300/0 conn(0x7f323c157b90 0x7f323c0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f323affd640 1 -- 172.21.15.7:0/2331083147 >> v2:172.21.15.193:3300/0 conn(0x7f323c157b90 msgr2=0x7f323c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:45.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f323affd640 1 --2- 172.21.15.7:0/2331083147 >> v2:172.21.15.193:3300/0 conn(0x7f323c157b90 0x7f323c0701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f323affd640 1 -- 172.21.15.7:0/2331083147 >> v1:172.21.15.193:6789/0 conn(0x7f323c06e560 legacy=0x7f323c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:45.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.405+0000 7f323affd640 1 -- 172.21.15.7:0/2331083147 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c157fa0 con 0x7f323c151cd0 2023-11-28T18:22:45.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f323affd640 1 --2- 172.21.15.7:0/2331083147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c15bd00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f323c0c5d70 tx=0x7f322c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e429ebbf68efe3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:45.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3239ffb640 1 -- 172.21.15.7:0/2331083147 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c0ad660 con 0x7f323c151cd0 2023-11-28T18:22:45.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3239ffb640 1 -- 172.21.15.7:0/2331083147 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c0ad7c0 con 0x7f323c151cd0 2023-11-28T18:22:45.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3239ffb640 1 -- 172.21.15.7:0/2331083147 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c0adb50 con 0x7f323c151cd0 2023-11-28T18:22:45.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 -- 172.21.15.7:0/2331083147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 msgr2=0x7f323c15bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 --2- 172.21.15.7:0/2331083147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c15bd00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f323c0c5d70 tx=0x7f322c01c470 comp rx=0 tx=0).stop 2023-11-28T18:22:45.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 -- 172.21.15.7:0/2331083147 shutdown_connections 2023-11-28T18:22:45.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 --2- 172.21.15.7:0/2331083147 >> v2:172.21.15.193:3300/0 conn(0x7f323c157b90 0x7f323c0701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 --2- 172.21.15.7:0/2331083147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c15bd00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 -- 172.21.15.7:0/2331083147 >> 172.21.15.7:0/2331083147 conn(0x7f323c0662e0 msgr2=0x7f323c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:45.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 -- 172.21.15.7:0/2331083147 shutdown_connections 2023-11-28T18:22:45.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 -- 172.21.15.7:0/2331083147 wait complete. 2023-11-28T18:22:45.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.409+0000 7f3241479640 1 Processor -- start 2023-11-28T18:22:45.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 -- start start 2023-11-28T18:22:45.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f323c06e560 0x7f323c0c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f323c157b90 0x7f323c0c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:45.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c1f25d0 con 0x7f323c151cd0 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0c7540 con 0x7f323c157b90 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0c4b30 con 0x7f323c06e560 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c156c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38300/0 (socket says 172.21.15.7:38300) 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 -- 172.21.15.7:0/2173358070 learned_addr learned my addr 172.21.15.7:0/2173358070 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323affd640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f323c06e560 0x7f323c0c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323b7fe640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f323c157b90 0x7f323c0c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 -- 172.21.15.7:0/2173358070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f323c06e560 msgr2=0x7f323c0c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f323c06e560 0x7f323c0c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 -- 172.21.15.7:0/2173358070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f323c157b90 msgr2=0x7f323c0c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f323c157b90 0x7f323c0c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 -- 172.21.15.7:0/2173358070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0094d0 con 0x7f323c151cd0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f323a7fc640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c156c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f323000c530 tx=0x7f323000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f32237fe640 1 -- 172.21.15.7:0/2173358070 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230005220 con 0x7f323c151cd0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f32237fe640 1 -- 172.21.15.7:0/2173358070 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3230005380 con 0x7f323c151cd0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c1f27e0 con 0x7f323c151cd0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f32237fe640 1 -- 172.21.15.7:0/2173358070 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230007270 con 0x7f323c151cd0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c1f2d60 con 0x7f323c151cd0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f32237fe640 1 -- 172.21.15.7:0/2173358070 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f32300067e0 con 0x7f323c151cd0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.413+0000 7f32237fe640 1 -- 172.21.15.7:0/2173358070 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f323001bad0 con 0x7f323c151cd0 2023-11-28T18:22:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.417+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c04e890 con 0x7f323c151cd0 2023-11-28T18:22:45.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.433+0000 7f32237fe640 1 -- 172.21.15.7:0/2173358070 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3230016020 con 0x7f323c151cd0 2023-11-28T18:22:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f323c1555b0 con 0x7f323c151cd0 2023-11-28T18:22:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f32237fe640 1 -- 172.21.15.7:0/2173358070 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f323002a090 con 0x7f323c151cd0 2023-11-28T18:22:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:22:45.497 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:22:45.160970+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:22:45.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 msgr2=0x7f323c156c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:45.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c156c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f323000c530 tx=0x7f323000c9f0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 shutdown_connections 2023-11-28T18:22:45.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f323c157b90 0x7f323c0c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f323c151cd0 0x7f323c156c00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 --2- 172.21.15.7:0/2173358070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f323c06e560 0x7f323c0c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 >> 172.21.15.7:0/2173358070 conn(0x7f323c0662e0 msgr2=0x7f323c1606e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 shutdown_connections 2023-11-28T18:22:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:22:45.493+0000 7f3241479640 1 -- 172.21.15.7:0/2173358070 wait complete. 2023-11-28T18:22:45.518 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-28T18:22:45.518 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-28T18:22:45.526 INFO:tasks.ceph.mgr.x.smithi193.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-11-28T18:22:45.527 INFO:tasks.ceph.mgr.x.smithi193.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-11-28T18:22:45.527 INFO:tasks.ceph.mgr.x.smithi193.stderr: from numpy import show_config as show_numpy_config 2023-11-28T18:22:45.535 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:45.533+0000 7fd851af0180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T18:22:45.588 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:45.585+0000 7fd4c8831180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T18:22:45.711 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:45.709+0000 7fd851af0180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T18:22:45.723 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:45.717+0000 7fd4c8831180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T18:22:45.787 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:45.785+0000 7fd851af0180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T18:22:45.872 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-28T18:22:45.869+0000 7fb426e2b640 -1 Falling back to public interface 2023-11-28T18:22:45.875 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-28T18:22:45.873+0000 7f8154b6e640 -1 Falling back to public interface 2023-11-28T18:22:45.876 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-28T18:22:45.873+0000 7fe7c022f640 -1 Falling back to public interface 2023-11-28T18:22:45.888 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-11-28T18:22:45.885+0000 7fe23c657640 -1 Falling back to public interface 2023-11-28T18:22:45.888 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-11-28T18:22:45.885+0000 7f30582f5640 -1 Falling back to public interface 2023-11-28T18:22:45.891 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-28T18:22:45.889+0000 7f4f47196640 -1 Falling back to public interface 2023-11-28T18:22:45.896 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-11-28T18:22:45.893+0000 7f8a09720640 -1 Falling back to public interface 2023-11-28T18:22:45.897 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-11-28T18:22:45.893+0000 7f745cc2b640 -1 Falling back to public interface 2023-11-28T18:22:45.903 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:45.897+0000 7fd4c8831180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T18:22:45.935 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:45.933+0000 7fd851af0180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T18:22:45.976 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:45.973+0000 7fd4c8831180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T18:22:46.112 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:46.109+0000 7fd4c8831180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T18:22:46.324 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T18:22:46.458 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.453+0000 7fee16b1e640 1 Processor -- start 2023-11-28T18:22:46.459 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee16b1e640 1 -- start start 2023-11-28T18:22:46.460 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee16b1e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee1015db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:46.460 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee16b1e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fee1015e070 0x7fee100cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee16b1e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1015d3c0 con 0x7fee10159dd0 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee16b1e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1015d6a0 con 0x7fee1015e070 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee16b1e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee100cad40 con 0x7fee10153fe0 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee1531b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fee1015e070 0x7fee100cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee1531b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fee1015e070 0x7fee100cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38304/0 (socket says 172.21.15.7:38304) 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee1531b640 1 -- 172.21.15.7:0/2268212186 learned_addr learned my addr 172.21.15.7:0/2268212186 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee15b1c640 1 --2- 172.21.15.7:0/2268212186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee1015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee1531b640 1 -- 172.21.15.7:0/2268212186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 msgr2=0x7fee1015db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee1531b640 1 --2- 172.21.15.7:0/2268212186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee1015db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee1531b640 1 -- 172.21.15.7:0/2268212186 >> v1:172.21.15.193:6789/0 conn(0x7fee10159dd0 legacy=0x7fee100cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:46.461 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.457+0000 7fee1531b640 1 -- 172.21.15.7:0/2268212186 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee100c4930 con 0x7fee1015e070 2023-11-28T18:22:46.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee1531b640 1 --2- 172.21.15.7:0/2268212186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee1015e070 0x7fee100cb170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fee00009870 tx=0x7fee0001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af65c22045c38000 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:46.464 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee14b1a640 1 -- 172.21.15.7:0/2268212186 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee000a8a30 con 0x7fee1015e070 2023-11-28T18:22:46.464 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee14b1a640 1 -- 172.21.15.7:0/2268212186 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee000a8b90 con 0x7fee1015e070 2023-11-28T18:22:46.464 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee14b1a640 1 -- 172.21.15.7:0/2268212186 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee000b0440 con 0x7fee1015e070 2023-11-28T18:22:46.465 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee16b1e640 1 -- 172.21.15.7:0/2268212186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee1015e070 msgr2=0x7fee100cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:46.465 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee16b1e640 1 --2- 172.21.15.7:0/2268212186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee1015e070 0x7fee100cb170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fee00009870 tx=0x7fee0001ed30 comp rx=0 tx=0).stop 2023-11-28T18:22:46.465 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee16b1e640 1 -- 172.21.15.7:0/2268212186 shutdown_connections 2023-11-28T18:22:46.465 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee16b1e640 1 --2- 172.21.15.7:0/2268212186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee1015e070 0x7fee100cb170 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.465 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee16b1e640 1 --2- 172.21.15.7:0/2268212186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee1015db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.465 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee16b1e640 1 -- 172.21.15.7:0/2268212186 >> 172.21.15.7:0/2268212186 conn(0x7fee100712c0 msgr2=0x7fee10066450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:46.465 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee16b1e640 1 -- 172.21.15.7:0/2268212186 shutdown_connections 2023-11-28T18:22:46.465 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.461+0000 7fee16b1e640 1 -- 172.21.15.7:0/2268212186 wait complete. 2023-11-28T18:22:46.466 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 Processor -- start 2023-11-28T18:22:46.467 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 -- start start 2023-11-28T18:22:46.468 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee100c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:46.468 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee10159dd0 0x7fee10153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:46.468 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee1015e070 0x7fee101ef710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:46.469 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101f2c80 con 0x7fee10159dd0 2023-11-28T18:22:46.469 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee100c70d0 con 0x7fee1015e070 2023-11-28T18:22:46.469 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee100c9980 con 0x7fee10153fe0 2023-11-28T18:22:46.469 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee100c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:46.469 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee1531b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee10159dd0 0x7fee10153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:46.470 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee100c6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:46716/0 (socket says 172.21.15.7:46716) 2023-11-28T18:22:46.470 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 -- 172.21.15.7:0/2786514851 learned_addr learned my addr 172.21.15.7:0/2786514851 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:22:46.470 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee1631d640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee1015e070 0x7fee101ef710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:46.470 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 -- 172.21.15.7:0/2786514851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee1015e070 msgr2=0x7fee101ef710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:46.470 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee1015e070 0x7fee101ef710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.471 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 -- 172.21.15.7:0/2786514851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee10159dd0 msgr2=0x7fee10153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:46.472 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee10159dd0 0x7fee10153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.472 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 -- 172.21.15.7:0/2786514851 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee000094d0 con 0x7fee10153fe0 2023-11-28T18:22:46.472 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee1531b640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee10159dd0 0x7fee10153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:22:46.474 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee15b1c640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee100c6b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e930 tx=0x7fee0c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:46.474 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee067fc640 1 -- 172.21.15.7:0/2786514851 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c009690 con 0x7fee10153fe0 2023-11-28T18:22:46.474 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee067fc640 1 -- 172.21.15.7:0/2786514851 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c00ca00 con 0x7fee10153fe0 2023-11-28T18:22:46.474 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee101f2eb0 con 0x7fee10153fe0 2023-11-28T18:22:46.474 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee067fc640 1 -- 172.21.15.7:0/2786514851 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c00cd70 con 0x7fee10153fe0 2023-11-28T18:22:46.474 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee101f3450 con 0x7fee10153fe0 2023-11-28T18:22:46.474 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee067fc640 1 -- 172.21.15.7:0/2786514851 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fee0c005ca0 con 0x7fee10153fe0 2023-11-28T18:22:46.474 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee067fc640 1 -- 172.21.15.7:0/2786514851 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fee0c013470 con 0x7fee10153fe0 2023-11-28T18:22:46.475 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.465+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee10075020 con 0x7fee10153fe0 2023-11-28T18:22:46.475 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.469+0000 7fee067fc640 1 -- 172.21.15.7:0/2786514851 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fee0c021060 con 0x7fee10153fe0 2023-11-28T18:22:46.533 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.529+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee100caf50 con 0x7fee10153fe0 2023-11-28T18:22:46.534 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.529+0000 7fee067fc640 1 -- 172.21.15.7:0/2786514851 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fee0c029020 con 0x7fee10153fe0 2023-11-28T18:22:46.534 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-28T18:22:46.534 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:22:45.160970+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:22:46.535 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 msgr2=0x7fee100c6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:46.536 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee100c6b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e930 tx=0x7fee0c00edf0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.536 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 shutdown_connections 2023-11-28T18:22:46.536 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee1015e070 0x7fee101ef710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.536 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee10159dd0 0x7fee10153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.536 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 --2- 172.21.15.7:0/2786514851 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fee10153fe0 0x7fee100c6b90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:46.536 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 >> 172.21.15.7:0/2786514851 conn(0x7fee100712c0 msgr2=0x7fee100cb170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:46.536 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 shutdown_connections 2023-11-28T18:22:46.536 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:46.533+0000 7fee16b1e640 1 -- 172.21.15.7:0/2786514851 wait complete. 2023-11-28T18:22:46.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-28T18:22:46.643 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:46.641+0000 7fd851af0180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T18:22:46.734 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:46.729+0000 7fd851af0180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T18:22:46.782 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:46.777+0000 7fd4c8831180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T18:22:46.869 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:46.865+0000 7fd4c8831180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T18:22:46.935 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:46.933+0000 7fd851af0180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T18:22:47.024 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:47.021+0000 7fd851af0180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T18:22:47.058 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:47.053+0000 7fd4c8831180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T18:22:47.144 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:47.141+0000 7fd4c8831180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T18:22:47.370 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:47.369+0000 7fd851af0180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T18:22:47.442 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:47.437+0000 7fd851af0180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T18:22:47.471 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:47.469+0000 7fd4c8831180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T18:22:47.542 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-11-28T18:22:47.537+0000 7fd4c8831180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T18:22:48.465 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:48.461+0000 7fd7f9274640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-28T18:22:48.466 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:22:48.461+0000 7fd7f9274640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-28T18:22:49.757 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-28T18:22:49.753+0000 7fb426e2b640 -1 osd.3 0 log_to_monitors true 2023-11-28T18:22:49.760 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-28T18:22:49.757+0000 7f8154b6e640 -1 osd.2 0 log_to_monitors true 2023-11-28T18:22:49.771 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-11-28T18:22:49.765+0000 7f30582f5640 -1 osd.7 0 log_to_monitors true 2023-11-28T18:22:49.828 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-11-28T18:22:49.825+0000 7f745cc2b640 -1 osd.4 0 log_to_monitors true 2023-11-28T18:22:49.891 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-11-28T18:22:49.885+0000 7f8a09720640 -1 osd.6 0 log_to_monitors true 2023-11-28T18:22:49.904 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-28T18:22:49.901+0000 7fe7c022f640 -1 osd.0 0 log_to_monitors true 2023-11-28T18:22:49.909 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-11-28T18:22:49.905+0000 7fe23c657640 -1 osd.5 0 log_to_monitors true 2023-11-28T18:22:49.918 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-28T18:22:49.913+0000 7f4f47196640 -1 osd.1 0 log_to_monitors true 2023-11-28T18:22:53.323 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-28T18:22:53.321+0000 7fe7bb9b7640 -1 osd.0 0 waiting for initial osdmap 2023-11-28T18:22:53.328 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-28T18:22:53.325+0000 7f4f4291e640 -1 osd.1 0 waiting for initial osdmap 2023-11-28T18:22:53.352 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-28T18:22:53.349+0000 7fe7b4004640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T18:22:53.357 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-28T18:22:53.353+0000 7fb421a97640 -1 osd.3 0 waiting for initial osdmap 2023-11-28T18:22:53.359 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T18:22:53.364 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-28T18:22:53.361+0000 7f4f3af6b640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T18:22:53.370 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-28T18:22:53.365+0000 7f81502f6640 -1 osd.2 0 waiting for initial osdmap 2023-11-28T18:22:53.388 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-28T18:22:53.385+0000 7fb41a0e4640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T18:22:53.396 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-28T18:22:53.393+0000 7f8148943640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T18:22:53.496 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8ed4e3640 1 Processor -- start 2023-11-28T18:22:53.496 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8ed4e3640 1 -- start start 2023-11-28T18:22:53.496 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8ed4e3640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa8e8153f90 0x7fa8e80cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:53.496 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8ed4e3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e8154370 0x7fa8e80ca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:53.497 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8ed4e3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e815d440 con 0x7fa8e80ca690 2023-11-28T18:22:53.497 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8ed4e3640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e815d670 con 0x7fa8e8153f90 2023-11-28T18:22:53.497 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8ed4e3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e8159e50 con 0x7fa8e8154370 2023-11-28T18:22:53.497 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8e77fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa8e80ca690 0x7fa8e80722f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:52784/0 (socket says 172.21.15.7:52784) 2023-11-28T18:22:53.497 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8e77fe640 1 -- 172.21.15.7:0/3460207585 learned_addr learned my addr 172.21.15.7:0/3460207585 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:22:53.497 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8e6ffd640 1 --2- 172.21.15.7:0/3460207585 >> v2:172.21.15.193:3300/0 conn(0x7fa8e8153f90 0x7fa8e80cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:53.498 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8dffff640 1 --2- 172.21.15.7:0/3460207585 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e8154370 0x7fa8e80ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:53.498 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8dffff640 1 -- 172.21.15.7:0/3460207585 >> v2:172.21.15.193:3300/0 conn(0x7fa8e8153f90 msgr2=0x7fa8e80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:53.499 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8dffff640 1 --2- 172.21.15.7:0/3460207585 >> v2:172.21.15.193:3300/0 conn(0x7fa8e8153f90 0x7fa8e80cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.499 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8dffff640 1 -- 172.21.15.7:0/3460207585 >> v1:172.21.15.193:6789/0 conn(0x7fa8e80ca690 legacy=0x7fa8e80722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:22:53.499 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.493+0000 7fa8dffff640 1 -- 172.21.15.7:0/3460207585 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e815d930 con 0x7fa8e8154370 2023-11-28T18:22:53.502 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8dffff640 1 --2- 172.21.15.7:0/3460207585 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e8154370 0x7fa8e80ca150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00097f0 tx=0x7fa8d001ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd3dcd76d4051f4c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:53.503 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8e67fc640 1 -- 172.21.15.7:0/3460207585 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d00a88f0 con 0x7fa8e8154370 2023-11-28T18:22:53.503 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8e67fc640 1 -- 172.21.15.7:0/3460207585 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d00a8a50 con 0x7fa8e8154370 2023-11-28T18:22:53.503 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8e67fc640 1 -- 172.21.15.7:0/3460207585 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d00a8de0 con 0x7fa8e8154370 2023-11-28T18:22:53.503 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/3460207585 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e8154370 msgr2=0x7fa8e80ca150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:53.503 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/3460207585 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e8154370 0x7fa8e80ca150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00097f0 tx=0x7fa8d001ec30 comp rx=0 tx=0).stop 2023-11-28T18:22:53.503 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/3460207585 shutdown_connections 2023-11-28T18:22:53.503 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/3460207585 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e8154370 0x7fa8e80ca150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.503 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/3460207585 >> v2:172.21.15.193:3300/0 conn(0x7fa8e8153f90 0x7fa8e80cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.504 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/3460207585 >> 172.21.15.7:0/3460207585 conn(0x7fa8e8070e10 msgr2=0x7fa8e8071220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:53.504 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/3460207585 shutdown_connections 2023-11-28T18:22:53.504 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.497+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/3460207585 wait complete. 2023-11-28T18:22:53.504 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8ed4e3640 1 Processor -- start 2023-11-28T18:22:53.504 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8ed4e3640 1 -- start start 2023-11-28T18:22:53.505 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8ed4e3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8e80ca690 0x7fa8e81f0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8ed4e3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8e8153f90 0x7fa8e80ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8ed4e3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e81471d0 0x7fa8e81f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8ed4e3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e814d070 con 0x7fa8e80ca690 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8ed4e3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e81f0da0 con 0x7fa8e8153f90 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8ed4e3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e814d2a0 con 0x7fa8e81471d0 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8e80ca690 0x7fa8e81f0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8e8153f90 0x7fa8e80ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8e80ca690 0x7fa8e81f0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46104/0 (socket says 172.21.15.7:46104) 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 -- 172.21.15.7:0/1747974990 learned_addr learned my addr 172.21.15.7:0/1747974990 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:22:53.506 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e77fe640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e81471d0 0x7fa8e81f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:53.507 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 -- 172.21.15.7:0/1747974990 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e81471d0 msgr2=0x7fa8e81f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:53.507 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e81471d0 0x7fa8e81f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.508 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 -- 172.21.15.7:0/1747974990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8e8153f90 msgr2=0x7fa8e80ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:53.508 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8e8153f90 0x7fa8e80ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.508 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 -- 172.21.15.7:0/1747974990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d0009420 con 0x7fa8e80ca690 2023-11-28T18:22:53.508 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8e6ffd640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8e80ca690 0x7fa8e81f0860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800e930 tx=0x7fa8d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:53.508 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8df7fe640 1 -- 172.21.15.7:0/1747974990 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8009690 con 0x7fa8e80ca690 2023-11-28T18:22:53.508 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8df7fe640 1 -- 172.21.15.7:0/1747974990 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d800ca00 con 0x7fa8e80ca690 2023-11-28T18:22:53.508 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.501+0000 7fa8df7fe640 1 -- 172.21.15.7:0/1747974990 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d800cd70 con 0x7fa8e80ca690 2023-11-28T18:22:53.509 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.505+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e814d580 con 0x7fa8e80ca690 2023-11-28T18:22:53.509 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.505+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e814c250 con 0x7fa8e80ca690 2023-11-28T18:22:53.509 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.505+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e804e890 con 0x7fa8e80ca690 2023-11-28T18:22:53.510 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.505+0000 7fa8df7fe640 1 -- 172.21.15.7:0/1747974990 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 86114+0+0 (secure 0 0 0) 0x7fa8d8013470 con 0x7fa8e80ca690 2023-11-28T18:22:53.511 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.505+0000 7fa8df7fe640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8c8066870 0x7fa8c8068d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:22:53.511 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.505+0000 7fa8df7fe640 1 -- 172.21.15.7:0/1747974990 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fa8d8083770 con 0x7fa8e80ca690 2023-11-28T18:22:53.511 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.505+0000 7fa8dffff640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8c8066870 0x7fa8c8068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:22:53.511 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.509+0000 7fa8dffff640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8c8066870 0x7fa8c8068d30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00b2000 tx=0x7fa8d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:22:53.515 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.513+0000 7fa8df7fe640 1 -- 172.21.15.7:0/1747974990 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa8d80518f0 con 0x7fa8e80ca690 2023-11-28T18:22:53.675 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.668+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8e80d06a0 con 0x7fa8e80ca690 2023-11-28T18:22:53.675 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8df7fe640 1 -- 172.21.15.7:0/1747974990 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa8d8021090 con 0x7fa8e80ca690 2023-11-28T18:22:53.676 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-28T18:22:53.677 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":12,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:22:51.464854+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:22:53.681 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8c8066870 msgr2=0x7fa8c8068d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:53.681 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8c8066870 0x7fa8c8068d30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00b2000 tx=0x7fa8d0008040 comp rx=0 tx=0).stop 2023-11-28T18:22:53.682 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8e80ca690 msgr2=0x7fa8e81f0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:22:53.683 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8e80ca690 0x7fa8e81f0860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800e930 tx=0x7fa8d800edf0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.683 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 shutdown_connections 2023-11-28T18:22:53.683 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8c8066870 0x7fa8c8068d30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8e81471d0 0x7fa8e81f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8e8153f90 0x7fa8e80ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 --2- 172.21.15.7:0/1747974990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8e80ca690 0x7fa8e81f0860 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:22:53.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 >> 172.21.15.7:0/1747974990 conn(0x7fa8e8070e10 msgr2=0x7fa8e80caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:22:53.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 shutdown_connections 2023-11-28T18:22:53.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:22:53.672+0000 7fa8ed4e3640 1 -- 172.21.15.7:0/1747974990 wait complete. 2023-11-28T18:22:53.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-28T18:22:53.852 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-11-28T18:22:53.849+0000 7f3053a7b640 -1 osd.7 0 waiting for initial osdmap 2023-11-28T18:22:53.858 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-11-28T18:22:53.853+0000 7f7457990640 -1 osd.4 0 waiting for initial osdmap 2023-11-28T18:22:53.877 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-11-28T18:22:53.873+0000 7f304c0c8640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T18:22:53.889 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-11-28T18:22:53.885+0000 7f744ffdd640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T18:22:53.907 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-11-28T18:22:53.901+0000 7f8a04ea6640 -1 osd.6 0 waiting for initial osdmap 2023-11-28T18:22:53.945 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-11-28T18:22:53.941+0000 7f89fd4f3640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T18:22:53.964 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-11-28T18:22:53.961+0000 7fe239613640 -1 osd.5 0 waiting for initial osdmap 2023-11-28T18:22:53.993 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-11-28T18:22:53.989+0000 7fe23042a640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T18:22:57.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:22:57.013 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:23:00.502 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:00.635 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cf69a640 1 Processor -- start 2023-11-28T18:23:00.636 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cf69a640 1 -- start start 2023-11-28T18:23:00.637 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cf69a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f73c81541b0 0x7f73c80c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:00.637 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cf69a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f73c8070280 0x7f73c80c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:00.637 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cf69a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c815d530 con 0x7f73c815df80 2023-11-28T18:23:00.637 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cf69a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c815db60 con 0x7f73c81541b0 2023-11-28T18:23:00.637 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cf69a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c815e3f0 con 0x7f73c8070280 2023-11-28T18:23:00.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cd436640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f73c81541b0 0x7f73c80c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:00.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cd436640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f73c81541b0 0x7f73c80c97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46140/0 (socket says 172.21.15.7:46140) 2023-11-28T18:23:00.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cd436640 1 -- 172.21.15.7:0/2043710135 learned_addr learned my addr 172.21.15.7:0/2043710135 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:00.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cdc37640 1 --2- 172.21.15.7:0/2043710135 >> v2:172.21.15.193:3300/0 conn(0x7f73c8070280 0x7f73c80c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:00.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cd436640 1 -- 172.21.15.7:0/2043710135 >> v2:172.21.15.193:3300/0 conn(0x7f73c8070280 msgr2=0x7f73c80c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:00.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cd436640 1 --2- 172.21.15.7:0/2043710135 >> v2:172.21.15.193:3300/0 conn(0x7f73c8070280 0x7f73c80c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cd436640 1 -- 172.21.15.7:0/2043710135 >> v1:172.21.15.193:6789/0 conn(0x7f73c815df80 legacy=0x7f73c8159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:00.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73bffff640 1 -- 172.21.15.7:0/2043710135 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654632863 0 0) 0x7f73c815d530 con 0x7f73c815df80 2023-11-28T18:23:00.639 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.632+0000 7f73cd436640 1 -- 172.21.15.7:0/2043710135 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c815a740 con 0x7f73c81541b0 2023-11-28T18:23:00.640 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cd436640 1 --2- 172.21.15.7:0/2043710135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c81541b0 0x7f73c80c97c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f73c806d9f0 tx=0x7f73c401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6aca09fc4442f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:00.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73bffff640 1 -- 172.21.15.7:0/2043710135 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c40ab600 con 0x7f73c81541b0 2023-11-28T18:23:00.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73bffff640 1 -- 172.21.15.7:0/2043710135 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c40a7050 con 0x7f73c81541b0 2023-11-28T18:23:00.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73bffff640 1 -- 172.21.15.7:0/2043710135 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c40ab980 con 0x7f73c81541b0 2023-11-28T18:23:00.642 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cf69a640 1 -- 172.21.15.7:0/2043710135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c81541b0 msgr2=0x7f73c80c97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:00.642 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cf69a640 1 --2- 172.21.15.7:0/2043710135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c81541b0 0x7f73c80c97c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f73c806d9f0 tx=0x7f73c401c3e0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.642 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cf69a640 1 -- 172.21.15.7:0/2043710135 shutdown_connections 2023-11-28T18:23:00.642 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cf69a640 1 --2- 172.21.15.7:0/2043710135 >> v2:172.21.15.193:3300/0 conn(0x7f73c8070280 0x7f73c80c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.642 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cf69a640 1 --2- 172.21.15.7:0/2043710135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c81541b0 0x7f73c80c97c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.642 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cf69a640 1 -- 172.21.15.7:0/2043710135 >> 172.21.15.7:0/2043710135 conn(0x7f73c8066450 msgr2=0x7f73c8066860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:00.642 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cf69a640 1 -- 172.21.15.7:0/2043710135 shutdown_connections 2023-11-28T18:23:00.642 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.636+0000 7f73cf69a640 1 -- 172.21.15.7:0/2043710135 wait complete. 2023-11-28T18:23:00.643 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 Processor -- start 2023-11-28T18:23:00.643 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 -- start start 2023-11-28T18:23:00.644 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73c8070280 0x7f73c81569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73c81541b0 0x7f73c8156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c815df80 0x7f73c8146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c814a720 con 0x7f73c815df80 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8156540 con 0x7f73c81541b0 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c814a4b0 con 0x7f73c8070280 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c815df80 0x7f73c8146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c815df80 0x7f73c8146710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46156/0 (socket says 172.21.15.7:46156) 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 -- 172.21.15.7:0/3213275321 learned_addr learned my addr 172.21.15.7:0/3213275321 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cd436640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73c8070280 0x7f73c81569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73ccc35640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73c81541b0 0x7f73c8156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:00.645 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 -- 172.21.15.7:0/3213275321 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73c8070280 msgr2=0x7f73c81569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:00.646 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73c8070280 0x7f73c81569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.646 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 -- 172.21.15.7:0/3213275321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73c81541b0 msgr2=0x7f73c8156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:00.646 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73c81541b0 0x7f73c8156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.646 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 -- 172.21.15.7:0/3213275321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c40094d0 con 0x7f73c815df80 2023-11-28T18:23:00.647 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cdc37640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c815df80 0x7f73c8146710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f73c000ec10 tx=0x7f73c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:00.647 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73bdffb640 1 -- 172.21.15.7:0/3213275321 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c000cd20 con 0x7f73c815df80 2023-11-28T18:23:00.647 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73bdffb640 1 -- 172.21.15.7:0/3213275321 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c000f040 con 0x7f73c815df80 2023-11-28T18:23:00.647 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73bdffb640 1 -- 172.21.15.7:0/3213275321 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c00146c0 con 0x7f73c815df80 2023-11-28T18:23:00.647 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c80cb700 con 0x7f73c815df80 2023-11-28T18:23:00.647 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c8144d60 con 0x7f73c815df80 2023-11-28T18:23:00.647 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.640+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c81566b0 con 0x7f73c815df80 2023-11-28T18:23:00.647 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.644+0000 7f73bdffb640 1 -- 172.21.15.7:0/3213275321 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f73c0014a30 con 0x7f73c815df80 2023-11-28T18:23:00.648 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.644+0000 7f73bdffb640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73ac066900 0x7f73ac068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:00.649 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.644+0000 7f73bdffb640 1 -- 172.21.15.7:0/3213275321 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f73c0084ee0 con 0x7f73c815df80 2023-11-28T18:23:00.649 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.644+0000 7f73cd436640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73ac066900 0x7f73ac068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:00.649 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.644+0000 7f73cd436640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73ac066900 0x7f73ac068dc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f73c40ad000 tx=0x7f73c40a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:00.653 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.648+0000 7f73bdffb640 1 -- 172.21.15.7:0/3213275321 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f73c0052480 con 0x7f73c815df80 2023-11-28T18:23:00.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.812+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c80cbbb0 con 0x7f73c815df80 2023-11-28T18:23:00.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.812+0000 7f73bdffb640 1 -- 172.21.15.7:0/3213275321 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12790 (secure 0 0 0) 0x7f73c0055aa0 con 0x7f73c815df80 2023-11-28T18:23:00.817 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-28T18:23:00.817 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":16,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:22:57.605286+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:00.823 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73ac066900 msgr2=0x7f73ac068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:00.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73ac066900 0x7f73ac068dc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f73c40ad000 tx=0x7f73c40a6000 comp rx=0 tx=0).stop 2023-11-28T18:23:00.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c815df80 msgr2=0x7f73c8146710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:00.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c815df80 0x7f73c8146710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f73c000ec10 tx=0x7f73c000c490 comp rx=0 tx=0).stop 2023-11-28T18:23:00.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 shutdown_connections 2023-11-28T18:23:00.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73ac066900 0x7f73ac068dc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73c815df80 0x7f73c8146710 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73c81541b0 0x7f73c8156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.825 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 --2- 172.21.15.7:0/3213275321 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73c8070280 0x7f73c81569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:00.825 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 >> 172.21.15.7:0/3213275321 conn(0x7f73c8066450 msgr2=0x7f73c806a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:00.825 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 shutdown_connections 2023-11-28T18:23:00.826 INFO:teuthology.misc.health.smithi007.stderr:2023-11-28T18:23:00.816+0000 7f73cf69a640 1 -- 172.21.15.7:0/3213275321 wait complete. 2023-11-28T18:23:00.837 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-28T18:23:00.837 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-28T18:23:00.838 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T18:23:00.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T18:23:00.841 INFO:tasks.ceph.mds.a:Started 2023-11-28T18:23:00.841 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T18:23:00.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T18:23:00.845 INFO:tasks.ceph.mds.c:Started 2023-11-28T18:23:00.845 INFO:tasks.ceph.mds.e:Restarting daemon 2023-11-28T18:23:00.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-11-28T18:23:00.848 INFO:tasks.ceph.mds.e:Started 2023-11-28T18:23:00.848 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T18:23:00.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T18:23:00.852 INFO:tasks.ceph.mds.b:Started 2023-11-28T18:23:00.852 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T18:23:00.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T18:23:00.856 INFO:tasks.ceph.mds.d:Started 2023-11-28T18:23:00.856 INFO:tasks.ceph.mds.f:Restarting daemon 2023-11-28T18:23:00.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-11-28T18:23:00.859 INFO:tasks.ceph.mds.f:Started 2023-11-28T18:23:00.860 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-28T18:23:00.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:00.947 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2023-11-28T18:23:00.956 INFO:tasks.ceph.mds.f.smithi193.stdout:starting mds.f at 2023-11-28T18:23:00.958 INFO:tasks.ceph.mds.b.smithi193.stdout:starting mds.b at 2023-11-28T18:23:00.958 INFO:tasks.ceph.mds.d.smithi193.stdout:starting mds.d at 2023-11-28T18:23:00.962 INFO:tasks.ceph.mds.c.smithi007.stdout:starting mds.c at 2023-11-28T18:23:00.963 INFO:tasks.ceph.mds.e.smithi007.stdout:starting mds.e at 2023-11-28T18:23:01.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f833de9a640 1 Processor -- start 2023-11-28T18:23:01.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f833de9a640 1 -- start start 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f833de9a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f83381542e0 0x7f833815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f833de9a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8338072450 0x7f8338072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f833de9a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83380ca920 con 0x7f83380cbb70 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f833de9a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8338071d50 con 0x7f8338072450 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f833de9a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8338072030 con 0x7f83381542e0 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f8336ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f83380cbb70 0x7f83380cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60696/0 (socket says 172.21.15.7:60696) 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f8336ffd640 1 -- 172.21.15.7:0/1704546288 learned_addr learned my addr 172.21.15.7:0/1704546288 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.008+0000 7f8337fff640 1 --2- 172.21.15.7:0/1704546288 >> v2:172.21.15.7:3300/0 conn(0x7f8338072450 0x7f8338072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f83377fe640 1 --2- 172.21.15.7:0/1704546288 >> v2:172.21.15.193:3300/0 conn(0x7f83381542e0 0x7f833815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f83367fc640 1 -- 172.21.15.7:0/1704546288 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505164903 0 0) 0x7f83380ca920 con 0x7f83380cbb70 2023-11-28T18:23:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f83367fc640 1 -- 172.21.15.7:0/1704546288 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8320003530 con 0x7f83380cbb70 2023-11-28T18:23:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f8337fff640 1 -- 172.21.15.7:0/1704546288 >> v2:172.21.15.193:3300/0 conn(0x7f83381542e0 msgr2=0x7f833815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f8337fff640 1 --2- 172.21.15.7:0/1704546288 >> v2:172.21.15.193:3300/0 conn(0x7f83381542e0 0x7f833815e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f8337fff640 1 -- 172.21.15.7:0/1704546288 >> v1:172.21.15.7:6789/0 conn(0x7f83380cbb70 legacy=0x7f83380cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f8337fff640 1 -- 172.21.15.7:0/1704546288 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833815e550 con 0x7f8338072450 2023-11-28T18:23:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f8337fff640 1 --2- 172.21.15.7:0/1704546288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8338072450 0x7f8338072870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8338071d20 tx=0x7f833001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15adbc3fab8d8264 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f83367fc640 1 -- 172.21.15.7:0/1704546288 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83300a58e0 con 0x7f8338072450 2023-11-28T18:23:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f83367fc640 1 -- 172.21.15.7:0/1704546288 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83300a5a40 con 0x7f8338072450 2023-11-28T18:23:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.012+0000 7f83367fc640 1 -- 172.21.15.7:0/1704546288 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83300a5db0 con 0x7f8338072450 2023-11-28T18:23:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- 172.21.15.7:0/1704546288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8338072450 msgr2=0x7f8338072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 --2- 172.21.15.7:0/1704546288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8338072450 0x7f8338072870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8338071d20 tx=0x7f833001c910 comp rx=0 tx=0).stop 2023-11-28T18:23:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- 172.21.15.7:0/1704546288 shutdown_connections 2023-11-28T18:23:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 --2- 172.21.15.7:0/1704546288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8338072450 0x7f8338072870 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 --2- 172.21.15.7:0/1704546288 >> v2:172.21.15.193:3300/0 conn(0x7f83381542e0 0x7f833815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- 172.21.15.7:0/1704546288 >> 172.21.15.7:0/1704546288 conn(0x7f83380717e0 msgr2=0x7f83380662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- 172.21.15.7:0/1704546288 shutdown_connections 2023-11-28T18:23:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- 172.21.15.7:0/1704546288 wait complete. 2023-11-28T18:23:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 Processor -- start 2023-11-28T18:23:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- start start 2023-11-28T18:23:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8338072450 0x7f8338162c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83380cbb70 0x7f8338160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83381542e0 0x7f8338161500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338152840 con 0x7f83380cbb70 2023-11-28T18:23:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83381f59f0 con 0x7f83381542e0 2023-11-28T18:23:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f833de9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83381f5cd0 con 0x7f8338072450 2023-11-28T18:23:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8336ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83380cbb70 0x7f8338160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f83377fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8338072450 0x7f8338162c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8336ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83380cbb70 0x7f8338160fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46198/0 (socket says 172.21.15.7:46198) 2023-11-28T18:23:01.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8336ffd640 1 -- 172.21.15.7:0/1828261600 learned_addr learned my addr 172.21.15.7:0/1828261600 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:01.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8337fff640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83381542e0 0x7f8338161500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8336ffd640 1 -- 172.21.15.7:0/1828261600 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8338072450 msgr2=0x7f8338162c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8336ffd640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8338072450 0x7f8338162c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8336ffd640 1 -- 172.21.15.7:0/1828261600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83381542e0 msgr2=0x7f8338161500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8336ffd640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83381542e0 0x7f8338161500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.016+0000 7f8336ffd640 1 -- 172.21.15.7:0/1828261600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8330009a70 con 0x7f83380cbb70 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f8336ffd640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83380cbb70 0x7f8338160fc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f832c00c450 tx=0x7f832c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f831ffff640 1 -- 172.21.15.7:0/1828261600 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c005cf0 con 0x7f83380cbb70 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f831ffff640 1 -- 172.21.15.7:0/1828261600 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c005e50 con 0x7f83380cbb70 2023-11-28T18:23:01.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f831ffff640 1 -- 172.21.15.7:0/1828261600 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c007290 con 0x7f83380cbb70 2023-11-28T18:23:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83381f5fb0 con 0x7f83380cbb70 2023-11-28T18:23:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83380667b0 con 0x7f83380cbb70 2023-11-28T18:23:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833806c670 con 0x7f83380cbb70 2023-11-28T18:23:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f831ffff640 1 -- 172.21.15.7:0/1828261600 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f832c0073f0 con 0x7f83380cbb70 2023-11-28T18:23:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f831ffff640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83240668d0 0x7f8324068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f831ffff640 1 -- 172.21.15.7:0/1828261600 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f832c0836b0 con 0x7f83380cbb70 2023-11-28T18:23:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.020+0000 7f83377fe640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83240668d0 0x7f8324068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.024+0000 7f83377fe640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83240668d0 0x7f8324068d90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8328009aa0 tx=0x7f8328009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.024+0000 7f831ffff640 1 -- 172.21.15.7:0/1828261600 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f832c050c50 con 0x7f83380cbb70 2023-11-28T18:23:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83381f67b0 con 0x7f83380cbb70 2023-11-28T18:23:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f831ffff640 1 -- 172.21.15.7:0/1828261600 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12790 (secure 0 0 0) 0x7f832c017020 con 0x7f83380cbb70 2023-11-28T18:23:01.188 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:01.188 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:22:57.605286+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83240668d0 msgr2=0x7f8324068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83240668d0 0x7f8324068d90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8328009aa0 tx=0x7f8328009190 comp rx=0 tx=0).stop 2023-11-28T18:23:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83380cbb70 msgr2=0x7f8338160fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83380cbb70 0x7f8338160fc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f832c00c450 tx=0x7f832c00c910 comp rx=0 tx=0).stop 2023-11-28T18:23:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 shutdown_connections 2023-11-28T18:23:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83240668d0 0x7f8324068d90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83381542e0 0x7f8338161500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83380cbb70 0x7f8338160fc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 --2- 172.21.15.7:0/1828261600 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8338072450 0x7f8338162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 >> 172.21.15.7:0/1828261600 conn(0x7f83380717e0 msgr2=0x7f83380b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 shutdown_connections 2023-11-28T18:23:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.184+0000 7f833de9a640 1 -- 172.21.15.7:0/1828261600 wait complete. 2023-11-28T18:23:01.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:23:01.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:23:01.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fecd6640 1 Processor -- start 2023-11-28T18:23:01.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fecd6640 1 -- start start 2023-11-28T18:23:01.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fecd6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f815be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fecd6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f815c390 con 0x7f14f806e7c0 2023-11-28T18:23:01.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fecd6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f8157f60 con 0x7f14f806e3a0 2023-11-28T18:23:01.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fecd6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f8158240 con 0x7f14f8141030 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fd273640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f14f806e7c0 0x7f14f8072880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60708/0 (socket says 172.21.15.7:60708) 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fd273640 1 -- 172.21.15.7:0/504043141 learned_addr learned my addr 172.21.15.7:0/504043141 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14fca72640 1 --2- 172.21.15.7:0/504043141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14f77fe640 1 -- 172.21.15.7:0/504043141 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645670151 0 0) 0x7f14f815c390 con 0x7f14f806e7c0 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.352+0000 7f14f77fe640 1 -- 172.21.15.7:0/504043141 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8003530 con 0x7f14f806e7c0 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14f77fe640 1 -- 172.21.15.7:0/504043141 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735382706 0 0) 0x7f14f8157f60 con 0x7f14f806e3a0 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14f77fe640 1 -- 172.21.15.7:0/504043141 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f815c390 con 0x7f14f806e3a0 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14fca72640 1 -- 172.21.15.7:0/504043141 >> v1:172.21.15.193:6789/0 conn(0x7f14f806e3a0 legacy=0x7f14f80c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14fca72640 1 -- 172.21.15.7:0/504043141 >> v1:172.21.15.7:6789/0 conn(0x7f14f806e7c0 legacy=0x7f14f8072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14fca72640 1 -- 172.21.15.7:0/504043141 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f8158520 con 0x7f14f8141030 2023-11-28T18:23:01.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14fca72640 1 --2- 172.21.15.7:0/504043141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f815be50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097f0 tx=0x7f14ec01ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4232c9fb203de48 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14f77fe640 1 -- 172.21.15.7:0/504043141 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0a89b0 con 0x7f14f8141030 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14f77fe640 1 -- 172.21.15.7:0/504043141 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec0a8b10 con 0x7f14f8141030 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14f77fe640 1 -- 172.21.15.7:0/504043141 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0a8ea0 con 0x7f14f8141030 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14fecd6640 1 -- 172.21.15.7:0/504043141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 msgr2=0x7f14f815be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.356+0000 7f14fecd6640 1 --2- 172.21.15.7:0/504043141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f815be50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097f0 tx=0x7f14ec01ecb0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 -- 172.21.15.7:0/504043141 shutdown_connections 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 --2- 172.21.15.7:0/504043141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f815be50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 -- 172.21.15.7:0/504043141 >> 172.21.15.7:0/504043141 conn(0x7f14f8073f00 msgr2=0x7f14f80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 -- 172.21.15.7:0/504043141 shutdown_connections 2023-11-28T18:23:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 -- 172.21.15.7:0/504043141 wait complete. 2023-11-28T18:23:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 Processor -- start 2023-11-28T18:23:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 -- start start 2023-11-28T18:23:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14f806e3a0 0x7f14f8154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14f806e7c0 0x7f14f8157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f81ef450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f8155930 con 0x7f14f806e7c0 2023-11-28T18:23:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f81eefc0 con 0x7f14f806e3a0 2023-11-28T18:23:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fecd6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f8155370 con 0x7f14f8141030 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14f806e7c0 0x7f14f8157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14f806e7c0 0x7f14f8157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46202/0 (socket says 172.21.15.7:46202) 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 -- 172.21.15.7:0/3053441792 learned_addr learned my addr 172.21.15.7:0/3053441792 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fd273640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f81ef450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14fca72640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14f806e3a0 0x7f14f8154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 -- 172.21.15.7:0/3053441792 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 msgr2=0x7f14f81ef450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f81ef450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 -- 172.21.15.7:0/3053441792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14f806e3a0 msgr2=0x7f14f8154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14f806e3a0 0x7f14f8154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 -- 172.21.15.7:0/3053441792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec009420 con 0x7f14f806e7c0 2023-11-28T18:23:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.360+0000 7f14f7fff640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14f806e7c0 0x7f14f8157510 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f14e400c490 tx=0x7f14e400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14f57fa640 1 -- 172.21.15.7:0/3053441792 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e40052d0 con 0x7f14f806e7c0 2023-11-28T18:23:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14f57fa640 1 -- 172.21.15.7:0/3053441792 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e4005430 con 0x7f14f806e7c0 2023-11-28T18:23:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14f57fa640 1 -- 172.21.15.7:0/3053441792 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4012870 con 0x7f14f806e7c0 2023-11-28T18:23:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f8155650 con 0x7f14f806e7c0 2023-11-28T18:23:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f81eec50 con 0x7f14f806e7c0 2023-11-28T18:23:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f81eee10 con 0x7f14f806e7c0 2023-11-28T18:23:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14f57fa640 1 -- 172.21.15.7:0/3053441792 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f14e40129d0 con 0x7f14f806e7c0 2023-11-28T18:23:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14f57fa640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14dc0668d0 0x7f14dc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14f57fa640 1 -- 172.21.15.7:0/3053441792 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f14e4083b90 con 0x7f14f806e7c0 2023-11-28T18:23:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.364+0000 7f14fca72640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14dc0668d0 0x7f14dc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.368+0000 7f14fca72640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14dc0668d0 0x7f14dc068d90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0b2000 tx=0x7f14ec0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.368+0000 7f14f57fa640 1 -- 172.21.15.7:0/3053441792 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f14e40511c0 con 0x7f14f806e7c0 2023-11-28T18:23:01.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.528+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14f81ef130 con 0x7f14f806e7c0 2023-11-28T18:23:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.528+0000 7f14f57fa640 1 -- 172.21.15.7:0/3053441792 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14e40547e0 con 0x7f14f806e7c0 2023-11-28T18:23:01.534 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14dc0668d0 msgr2=0x7f14dc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14dc0668d0 0x7f14dc068d90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0b2000 tx=0x7f14ec0aa070 comp rx=0 tx=0).stop 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14f806e7c0 msgr2=0x7f14f8157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14f806e7c0 0x7f14f8157510 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f14e400c490 tx=0x7f14e400c950 comp rx=0 tx=0).stop 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 shutdown_connections 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14dc0668d0 0x7f14dc068d90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14f8141030 0x7f14f81ef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14f806e7c0 0x7f14f8157510 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 --2- 172.21.15.7:0/3053441792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14f806e3a0 0x7f14f8154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 >> 172.21.15.7:0/3053441792 conn(0x7f14f8073f00 msgr2=0x7f14f80c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:01.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 shutdown_connections 2023-11-28T18:23:01.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.532+0000 7f14fecd6640 1 -- 172.21.15.7:0/3053441792 wait complete. 2023-11-28T18:23:01.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.692+0000 7fac95a5f640 1 Processor -- start 2023-11-28T18:23:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.692+0000 7fac95a5f640 1 -- start start 2023-11-28T18:23:01.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac95a5f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac900ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac95a5f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9015e380 con 0x7fac900cf6c0 2023-11-28T18:23:01.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac95a5f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9015d7e0 con 0x7fac9015a8a0 2023-11-28T18:23:01.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac95a5f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac900cab80 con 0x7fac901540e0 2023-11-28T18:23:01.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fac900cf6c0 0x7fac90164760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60710/0 (socket says 172.21.15.7:60710) 2023-11-28T18:23:01.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8f7fe640 1 -- 172.21.15.7:0/1396585200 learned_addr learned my addr 172.21.15.7:0/1396585200 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:01.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8effd640 1 --2- 172.21.15.7:0/1396585200 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac900ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8e7fc640 1 -- 172.21.15.7:0/1396585200 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354982773 0 0) 0x7fac880014a0 con 0x7fac900cf6c0 2023-11-28T18:23:01.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8e7fc640 1 -- 172.21.15.7:0/1396585200 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7c003530 con 0x7fac900cf6c0 2023-11-28T18:23:01.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8e7fc640 1 -- 172.21.15.7:0/1396585200 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746563191 0 0) 0x7fac9015d7e0 con 0x7fac9015a8a0 2023-11-28T18:23:01.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8e7fc640 1 -- 172.21.15.7:0/1396585200 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac880014a0 con 0x7fac9015a8a0 2023-11-28T18:23:01.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8effd640 1 -- 172.21.15.7:0/1396585200 >> v1:172.21.15.193:6789/0 conn(0x7fac9015a8a0 legacy=0x7fac9015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8effd640 1 -- 172.21.15.7:0/1396585200 >> v1:172.21.15.7:6789/0 conn(0x7fac900cf6c0 legacy=0x7fac90164760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.696+0000 7fac8effd640 1 -- 172.21.15.7:0/1396585200 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac900cb850 con 0x7fac901540e0 2023-11-28T18:23:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac8effd640 1 --2- 172.21.15.7:0/1396585200 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac900ca4d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fac800097f0 tx=0x7fac8001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac5eabae453d7c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac8e7fc640 1 -- 172.21.15.7:0/1396585200 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac800ac410 con 0x7fac901540e0 2023-11-28T18:23:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac8e7fc640 1 -- 172.21.15.7:0/1396585200 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac800ac570 con 0x7fac901540e0 2023-11-28T18:23:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac8e7fc640 1 -- 172.21.15.7:0/1396585200 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac800ac8e0 con 0x7fac901540e0 2023-11-28T18:23:01.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 -- 172.21.15.7:0/1396585200 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 msgr2=0x7fac900ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 --2- 172.21.15.7:0/1396585200 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac900ca4d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fac800097f0 tx=0x7fac8001c4c0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 -- 172.21.15.7:0/1396585200 shutdown_connections 2023-11-28T18:23:01.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 --2- 172.21.15.7:0/1396585200 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac900ca4d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 -- 172.21.15.7:0/1396585200 >> 172.21.15.7:0/1396585200 conn(0x7fac900662e0 msgr2=0x7fac900666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:01.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 -- 172.21.15.7:0/1396585200 shutdown_connections 2023-11-28T18:23:01.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 -- 172.21.15.7:0/1396585200 wait complete. 2023-11-28T18:23:01.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 Processor -- start 2023-11-28T18:23:01.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.700+0000 7fac95a5f640 1 -- start start 2023-11-28T18:23:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac900cf6c0 0x7fac901f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac9015ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac9015a8a0 0x7fac901460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac900b2410 con 0x7fac900cf6c0 2023-11-28T18:23:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9015c730 con 0x7fac9015a8a0 2023-11-28T18:23:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac90157a80 con 0x7fac901540e0 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac900cf6c0 0x7fac901f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac900cf6c0 0x7fac901f6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46206/0 (socket says 172.21.15.7:46206) 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 -- 172.21.15.7:0/185127539 learned_addr learned my addr 172.21.15.7:0/185127539 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac87fff640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac9015ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8f7fe640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac9015a8a0 0x7fac901460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 -- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 msgr2=0x7fac9015ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac9015ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 -- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac9015a8a0 msgr2=0x7fac901460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac9015a8a0 0x7fac901460e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 -- 172.21.15.7:0/185127539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac80009420 con 0x7fac900cf6c0 2023-11-28T18:23:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8f7fe640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac9015a8a0 0x7fac901460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8effd640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac900cf6c0 0x7fac901f6f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fac80000c10 tx=0x7fac80019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8cff9640 1 -- 172.21.15.7:0/185127539 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac800ac8f0 con 0x7fac900cf6c0 2023-11-28T18:23:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8cff9640 1 -- 172.21.15.7:0/185127539 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac800aca50 con 0x7fac900cf6c0 2023-11-28T18:23:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac8cff9640 1 -- 172.21.15.7:0/185127539 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac800acdc0 con 0x7fac900cf6c0 2023-11-28T18:23:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac90157d60 con 0x7fac900cf6c0 2023-11-28T18:23:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9015afa0 con 0x7fac900cf6c0 2023-11-28T18:23:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.704+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9004e890 con 0x7fac900cf6c0 2023-11-28T18:23:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.708+0000 7fac8cff9640 1 -- 172.21.15.7:0/185127539 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fac800b3040 con 0x7fac900cf6c0 2023-11-28T18:23:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.708+0000 7fac8cff9640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac700668d0 0x7fac70068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.708+0000 7fac8cff9640 1 -- 172.21.15.7:0/185127539 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fac8011e0b0 con 0x7fac900cf6c0 2023-11-28T18:23:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.708+0000 7fac87fff640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac700668d0 0x7fac70068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.708+0000 7fac87fff640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac700668d0 0x7fac70068d90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac90147c40 tx=0x7fac78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:01.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.712+0000 7fac8cff9640 1 -- 172.21.15.7:0/185127539 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fac800eb630 con 0x7fac900cf6c0 2023-11-28T18:23:01.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac900b57e0 con 0x7fac900cf6c0 2023-11-28T18:23:01.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac8cff9640 1 -- 172.21.15.7:0/185127539 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fac800eec50 con 0x7fac900cf6c0 2023-11-28T18:23:01.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:01.907 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-11-28T18:23:01.911 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 2 2023-11-28T18:23:01.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac700668d0 msgr2=0x7fac70068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac700668d0 0x7fac70068d90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac90147c40 tx=0x7fac78009040 comp rx=0 tx=0).stop 2023-11-28T18:23:01.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac900cf6c0 msgr2=0x7fac901f6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:01.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac900cf6c0 0x7fac901f6f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fac80000c10 tx=0x7fac80019260 comp rx=0 tx=0).stop 2023-11-28T18:23:01.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 shutdown_connections 2023-11-28T18:23:01.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac700668d0 0x7fac70068d90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac9015a8a0 0x7fac901460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac901540e0 0x7fac9015ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 --2- 172.21.15.7:0/185127539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac900cf6c0 0x7fac901f6f70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:01.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 >> 172.21.15.7:0/185127539 conn(0x7fac900662e0 msgr2=0x7fac900726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:01.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 shutdown_connections 2023-11-28T18:23:01.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:01.900+0000 7fac95a5f640 1 -- 172.21.15.7:0/185127539 wait complete. 2023-11-28T18:23:03.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.076+0000 7f8976b0f640 1 Processor -- start 2023-11-28T18:23:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.076+0000 7f8976b0f640 1 -- start start 2023-11-28T18:23:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f8976b0f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89701540a0 0x7f8970072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f8976b0f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89700c8750 con 0x7f89700c8ec0 2023-11-28T18:23:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f8976b0f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89700c8a30 con 0x7f897015d350 2023-11-28T18:23:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f8976b0f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897015d840 con 0x7f89701540a0 2023-11-28T18:23:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f89750ac640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f89700c8ec0 0x7f89701646e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33842/0 (socket says 172.21.15.7:33842) 2023-11-28T18:23:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f89750ac640 1 -- 172.21.15.7:0/3603980026 learned_addr learned my addr 172.21.15.7:0/3603980026 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f89748ab640 1 --2- 172.21.15.7:0/3603980026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89701540a0 0x7f8970072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f896f7fe640 1 -- 172.21.15.7:0/3603980026 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100362665 0 0) 0x7f89700c8750 con 0x7f89700c8ec0 2023-11-28T18:23:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f896f7fe640 1 -- 172.21.15.7:0/3603980026 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8960003530 con 0x7f89700c8ec0 2023-11-28T18:23:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f896f7fe640 1 -- 172.21.15.7:0/3603980026 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293226849 0 0) 0x7f89700c8a30 con 0x7f897015d350 2023-11-28T18:23:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f896f7fe640 1 -- 172.21.15.7:0/3603980026 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89700c8750 con 0x7f897015d350 2023-11-28T18:23:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f89748ab640 1 -- 172.21.15.7:0/3603980026 >> v1:172.21.15.193:6789/0 conn(0x7f897015d350 legacy=0x7f897006a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f89748ab640 1 -- 172.21.15.7:0/3603980026 >> v1:172.21.15.7:6789/0 conn(0x7f89700c8ec0 legacy=0x7f89701646e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f896f7fe640 1 -- 172.21.15.7:0/3603980026 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207869727 0 0) 0x7f8960003530 con 0x7f89700c8ec0 2023-11-28T18:23:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.080+0000 7f89748ab640 1 -- 172.21.15.7:0/3603980026 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897015db20 con 0x7f89701540a0 2023-11-28T18:23:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f89748ab640 1 --2- 172.21.15.7:0/3603980026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89701540a0 0x7f8970072ab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8964009820 tx=0x7f896401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c58edeeb37620c10 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f896f7fe640 1 -- 172.21.15.7:0/3603980026 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968001a00 con 0x7f89701540a0 2023-11-28T18:23:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f896f7fe640 1 -- 172.21.15.7:0/3603980026 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89640a7050 con 0x7f89701540a0 2023-11-28T18:23:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f896f7fe640 1 -- 172.21.15.7:0/3603980026 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89640ab600 con 0x7f89701540a0 2023-11-28T18:23:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f8976b0f640 1 -- 172.21.15.7:0/3603980026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89701540a0 msgr2=0x7f8970072ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f8976b0f640 1 --2- 172.21.15.7:0/3603980026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89701540a0 0x7f8970072ab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8964009820 tx=0x7f896401c3c0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f8976b0f640 1 -- 172.21.15.7:0/3603980026 shutdown_connections 2023-11-28T18:23:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f8976b0f640 1 --2- 172.21.15.7:0/3603980026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89701540a0 0x7f8970072ab0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f8976b0f640 1 -- 172.21.15.7:0/3603980026 >> 172.21.15.7:0/3603980026 conn(0x7f8970073cd0 msgr2=0x7f89700662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f8976b0f640 1 -- 172.21.15.7:0/3603980026 shutdown_connections 2023-11-28T18:23:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f8976b0f640 1 -- 172.21.15.7:0/3603980026 wait complete. 2023-11-28T18:23:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.084+0000 7f8976b0f640 1 Processor -- start 2023-11-28T18:23:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 -- start start 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89700c8ec0 0x7f89701f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89701540a0 0x7f8970153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f897015d350 0x7f89701f5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89700bdc10 con 0x7f89700c8ec0 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8970159500 con 0x7f89701540a0 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89701f6580 con 0x7f897015d350 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89700c8ec0 0x7f89701f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89700c8ec0 0x7f89701f2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48024/0 (socket says 172.21.15.7:48024) 2023-11-28T18:23:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 -- 172.21.15.7:0/1519930822 learned_addr learned my addr 172.21.15.7:0/1519930822 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89750ac640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f897015d350 0x7f89701f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f896ffff640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89701540a0 0x7f8970153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 -- 172.21.15.7:0/1519930822 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f897015d350 msgr2=0x7f89701f5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f897015d350 0x7f89701f5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 -- 172.21.15.7:0/1519930822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89701540a0 msgr2=0x7f8970153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89701540a0 0x7f8970153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 -- 172.21.15.7:0/1519930822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8964009420 con 0x7f89700c8ec0 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f89748ab640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89700c8ec0 0x7f89701f2ad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8964009820 tx=0x7f896401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f896d7fa640 1 -- 172.21.15.7:0/1519930822 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89640bc070 con 0x7f89700c8ec0 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f896d7fa640 1 -- 172.21.15.7:0/1519930822 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89640abcf0 con 0x7f89700c8ec0 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89701f6860 con 0x7f89700c8ec0 2023-11-28T18:23:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f896d7fa640 1 -- 172.21.15.7:0/1519930822 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89640b7670 con 0x7f89700c8ec0 2023-11-28T18:23:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89701f6e20 con 0x7f89700c8ec0 2023-11-28T18:23:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.088+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897004e890 con 0x7f89700c8ec0 2023-11-28T18:23:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.092+0000 7f896d7fa640 1 -- 172.21.15.7:0/1519930822 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f89640b77d0 con 0x7f89700c8ec0 2023-11-28T18:23:04.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.092+0000 7f896d7fa640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f89540668d0 0x7f8954068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.092+0000 7f896d7fa640 1 -- 172.21.15.7:0/1519930822 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f896411dff0 con 0x7f89700c8ec0 2023-11-28T18:23:04.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.092+0000 7f896ffff640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f89540668d0 0x7f8954068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.092+0000 7f896ffff640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f89540668d0 0x7f8954068d90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89701f0700 tx=0x7f895c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.096+0000 7f896d7fa640 1 -- 172.21.15.7:0/1519930822 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f89640eb4f0 con 0x7f89700c8ec0 2023-11-28T18:23:04.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.284+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8970073410 con 0x7f89700c8ec0 2023-11-28T18:23:04.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f896d7fa640 1 -- 172.21.15.7:0/1519930822 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5274 (secure 0 0 0) 0x7f89640abe60 con 0x7f89700c8ec0 2023-11-28T18:23:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:04.293 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4291,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-11-28T18:23:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 3 2023-11-28T18:23:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f89540668d0 msgr2=0x7f8954068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f89540668d0 0x7f8954068d90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89701f0700 tx=0x7f895c009040 comp rx=0 tx=0).stop 2023-11-28T18:23:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89700c8ec0 msgr2=0x7f89701f2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89700c8ec0 0x7f89701f2ad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8964009820 tx=0x7f896401ac00 comp rx=0 tx=0).stop 2023-11-28T18:23:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 shutdown_connections 2023-11-28T18:23:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f89540668d0 0x7f8954068d90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f897015d350 0x7f89701f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89701540a0 0x7f8970153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 --2- 172.21.15.7:0/1519930822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89700c8ec0 0x7f89701f2ad0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 >> 172.21.15.7:0/1519930822 conn(0x7f8970073cd0 msgr2=0x7f897015b130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 shutdown_connections 2023-11-28T18:23:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.288+0000 7f8976b0f640 1 -- 172.21.15.7:0/1519930822 wait complete. 2023-11-28T18:23:04.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc10c596640 1 Processor -- start 2023-11-28T18:23:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc10c596640 1 -- start start 2023-11-28T18:23:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc10c596640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1040cbb70 0x7fc1040cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc10c596640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1040ca920 con 0x7fc1040cc810 2023-11-28T18:23:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc10c596640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1040c4cd0 con 0x7fc1041542e0 2023-11-28T18:23:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc10c596640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc104071d50 con 0x7fc1040cbb70 2023-11-28T18:23:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc109b31640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1040cbb70 0x7fc1040cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc10ab33640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc1040cc810 0x7fc104072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33848/0 (socket says 172.21.15.7:33848) 2023-11-28T18:23:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc10ab33640 1 -- 172.21.15.7:0/1568931397 learned_addr learned my addr 172.21.15.7:0/1568931397 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc109b31640 1 -- 172.21.15.7:0/1568931397 >> v1:172.21.15.193:6789/0 conn(0x7fc1041542e0 legacy=0x7fc10415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc109b31640 1 -- 172.21.15.7:0/1568931397 >> v1:172.21.15.7:6789/0 conn(0x7fc1040cc810 legacy=0x7fc104072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.452+0000 7fc109b31640 1 -- 172.21.15.7:0/1568931397 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1040cab70 con 0x7fc1040cbb70 2023-11-28T18:23:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc109b31640 1 --2- 172.21.15.7:0/1568931397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1040cbb70 0x7fc1040cc2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0f40097c0 tx=0x7fc0f401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f52b4d4e08cd6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc109330640 1 -- 172.21.15.7:0/1568931397 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f40a8050 con 0x7fc1040cbb70 2023-11-28T18:23:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc109330640 1 -- 172.21.15.7:0/1568931397 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f40ac5f0 con 0x7fc1040cbb70 2023-11-28T18:23:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc109330640 1 -- 172.21.15.7:0/1568931397 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f40ac980 con 0x7fc1040cbb70 2023-11-28T18:23:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc10c596640 1 -- 172.21.15.7:0/1568931397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1040cbb70 msgr2=0x7fc1040cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc10c596640 1 --2- 172.21.15.7:0/1568931397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1040cbb70 0x7fc1040cc2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0f40097c0 tx=0x7fc0f401c480 comp rx=0 tx=0).stop 2023-11-28T18:23:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc10c596640 1 -- 172.21.15.7:0/1568931397 shutdown_connections 2023-11-28T18:23:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc10c596640 1 --2- 172.21.15.7:0/1568931397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1040cbb70 0x7fc1040cc2d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc10c596640 1 -- 172.21.15.7:0/1568931397 >> 172.21.15.7:0/1568931397 conn(0x7fc1040717e0 msgr2=0x7fc1040662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc10c596640 1 -- 172.21.15.7:0/1568931397 shutdown_connections 2023-11-28T18:23:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc10c596640 1 -- 172.21.15.7:0/1568931397 wait complete. 2023-11-28T18:23:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.456+0000 7fc10c596640 1 Processor -- start 2023-11-28T18:23:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 -- start start 2023-11-28T18:23:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1040cbb70 0x7fc10414f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1040cc810 0x7fc10414fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1041542e0 0x7fc1041503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc104072030 con 0x7fc1041542e0 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10414bd60 con 0x7fc1040cc810 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10414a5b0 con 0x7fc1040cbb70 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1041542e0 0x7fc1041503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1041542e0 0x7fc1041503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48034/0 (socket says 172.21.15.7:48034) 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 -- 172.21.15.7:0/2477367372 learned_addr learned my addr 172.21.15.7:0/2477367372 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc109b31640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1040cc810 0x7fc10414fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10a332640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1040cbb70 0x7fc10414f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 -- 172.21.15.7:0/2477367372 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1040cbb70 msgr2=0x7fc10414f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1040cbb70 0x7fc10414f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 -- 172.21.15.7:0/2477367372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1040cc810 msgr2=0x7fc10414fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1040cc810 0x7fc10414fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 -- 172.21.15.7:0/2477367372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f4009420 con 0x7fc1041542e0 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10ab33640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1041542e0 0x7fc1041503b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00cbc0 tx=0x7fc0fc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc0faffd640 1 -- 172.21.15.7:0/2477367372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc00ad40 con 0x7fc1041542e0 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc0faffd640 1 -- 172.21.15.7:0/2477367372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0fc012400 con 0x7fc1041542e0 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc0faffd640 1 -- 172.21.15.7:0/2477367372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc00aea0 con 0x7fc1041542e0 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10414ab30 con 0x7fc1041542e0 2023-11-28T18:23:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1040c9080 con 0x7fc1041542e0 2023-11-28T18:23:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.460+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10406c670 con 0x7fc1041542e0 2023-11-28T18:23:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.464+0000 7fc0faffd640 1 -- 172.21.15.7:0/2477367372 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc0fc007070 con 0x7fc1041542e0 2023-11-28T18:23:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.464+0000 7fc0faffd640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc0e80668d0 0x7fc0e8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.464+0000 7fc0faffd640 1 -- 172.21.15.7:0/2477367372 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc0fc083fa0 con 0x7fc1041542e0 2023-11-28T18:23:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.464+0000 7fc10a332640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc0e80668d0 0x7fc0e8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.464+0000 7fc10a332640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc0e80668d0 0x7fc0e8068d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc100007fd0 tx=0x7fc100009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.468+0000 7fc0faffd640 1 -- 172.21.15.7:0/2477367372 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc0fc0b5280 con 0x7fc1041542e0 2023-11-28T18:23:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc10404e890 con 0x7fc1041542e0 2023-11-28T18:23:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc0faffd640 1 -- 172.21.15.7:0/2477367372 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12790 (secure 0 0 0) 0x7fc0fc0515e0 con 0x7fc1041542e0 2023-11-28T18:23:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:04.633 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:22:57.605286+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:04.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc0e80668d0 msgr2=0x7fc0e8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc0e80668d0 0x7fc0e8068d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc100007fd0 tx=0x7fc100009040 comp rx=0 tx=0).stop 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1041542e0 msgr2=0x7fc1041503b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1041542e0 0x7fc1041503b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00cbc0 tx=0x7fc0fc00a430 comp rx=0 tx=0).stop 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 shutdown_connections 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc0e80668d0 0x7fc0e8068d90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1041542e0 0x7fc1041503b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1040cc810 0x7fc10414fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 --2- 172.21.15.7:0/2477367372 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1040cbb70 0x7fc10414f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 >> 172.21.15.7:0/2477367372 conn(0x7fc1040717e0 msgr2=0x7fc104072450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 shutdown_connections 2023-11-28T18:23:04.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.628+0000 7fc10c596640 1 -- 172.21.15.7:0/2477367372 wait complete. 2023-11-28T18:23:04.655 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:23:04.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:23:04.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.796+0000 7fadb1817640 1 Processor -- start 2023-11-28T18:23:04.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.796+0000 7fadb1817640 1 -- start start 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.796+0000 7fadb1817640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.796+0000 7fadb1817640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadac0ca280 0x7fadac15d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.796+0000 7fadb1817640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac0ca800 con 0x7fadac0cf5e0 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.796+0000 7fadb1817640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadac15e020 con 0x7fadac0ca280 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.796+0000 7fadb1817640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadac15a1a0 con 0x7fadac0b3da0 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadabfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadac0ca280 0x7fadac15d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadb1016640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fadac0cf5e0 0x7fadac0cc060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33862/0 (socket says 172.21.15.7:33862) 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadb1016640 1 -- 172.21.15.7:0/371003527 learned_addr learned my addr 172.21.15.7:0/371003527 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadb0815640 1 --2- 172.21.15.7:0/371003527 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadabfff640 1 -- 172.21.15.7:0/371003527 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 msgr2=0x7fadac0cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadabfff640 1 --2- 172.21.15.7:0/371003527 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadabfff640 1 -- 172.21.15.7:0/371003527 >> v1:172.21.15.7:6789/0 conn(0x7fadac0cf5e0 legacy=0x7fadac0cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadabfff640 1 -- 172.21.15.7:0/371003527 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac15a480 con 0x7fadac0ca280 2023-11-28T18:23:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadabfff640 1 --2- 172.21.15.7:0/371003527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac15d6a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fad980097c0 tx=0x7fad9801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257c23735c06c1fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadab7fe640 1 -- 172.21.15.7:0/371003527 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980ab650 con 0x7fadac0ca280 2023-11-28T18:23:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadab7fe640 1 -- 172.21.15.7:0/371003527 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad980ab7b0 con 0x7fadac0ca280 2023-11-28T18:23:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.800+0000 7fadab7fe640 1 -- 172.21.15.7:0/371003527 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980abb40 con 0x7fadac0ca280 2023-11-28T18:23:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- 172.21.15.7:0/371003527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 msgr2=0x7fadac15d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 --2- 172.21.15.7:0/371003527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac15d6a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fad980097c0 tx=0x7fad9801c450 comp rx=0 tx=0).stop 2023-11-28T18:23:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- 172.21.15.7:0/371003527 shutdown_connections 2023-11-28T18:23:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 --2- 172.21.15.7:0/371003527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac15d6a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 --2- 172.21.15.7:0/371003527 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- 172.21.15.7:0/371003527 >> 172.21.15.7:0/371003527 conn(0x7fadac071210 msgr2=0x7fadac0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- 172.21.15.7:0/371003527 shutdown_connections 2023-11-28T18:23:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- 172.21.15.7:0/371003527 wait complete. 2023-11-28T18:23:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 Processor -- start 2023-11-28T18:23:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- start start 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0c5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac0cf5e0 0x7fadac1547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac0c5660 con 0x7fadac0ca280 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac1f2b80 con 0x7fadac0cf5e0 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1817640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadac1f3b10 con 0x7fadac0b3da0 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadabfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadabfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48050/0 (socket says 172.21.15.7:48050) 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadabfff640 1 -- 172.21.15.7:0/3814579712 learned_addr learned my addr 172.21.15.7:0/3814579712 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb0815640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb1016640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac0cf5e0 0x7fadac1547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadabfff640 1 -- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 msgr2=0x7fadac0c5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadabfff640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0c5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadabfff640 1 -- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac0cf5e0 msgr2=0x7fadac1547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadabfff640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac0cf5e0 0x7fadac1547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadabfff640 1 -- 172.21.15.7:0/3814579712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad98009420 con 0x7fadac0ca280 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.804+0000 7fadb0815640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0c5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fadabfff640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac153900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fad980ad040 tx=0x7fad9801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fada97fa640 1 -- 172.21.15.7:0/3814579712 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980bf610 con 0x7fadac0ca280 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fada97fa640 1 -- 172.21.15.7:0/3814579712 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad980bf790 con 0x7fadac0ca280 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fada97fa640 1 -- 172.21.15.7:0/3814579712 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980bfb40 con 0x7fadac0ca280 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac1f2d90 con 0x7fadac0ca280 2023-11-28T18:23:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac1f3030 con 0x7fadac0ca280 2023-11-28T18:23:04.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac04e890 con 0x7fadac0ca280 2023-11-28T18:23:04.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fada97fa640 1 -- 172.21.15.7:0/3814579712 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fad980b5400 con 0x7fadac0ca280 2023-11-28T18:23:04.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fada97fa640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066950 0x7fad90068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:04.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fada97fa640 1 -- 172.21.15.7:0/3814579712 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad9811f220 con 0x7fadac0ca280 2023-11-28T18:23:04.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.808+0000 7fadb0815640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066950 0x7fad90068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:04.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.812+0000 7fadb0815640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066950 0x7fad90068e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fada000b9f0 tx=0x7fada0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.812+0000 7fada97fa640 1 -- 172.21.15.7:0/3814579712 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fad980ec8b0 con 0x7fadac0ca280 2023-11-28T18:23:04.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fadac0cade0 con 0x7fadac0ca280 2023-11-28T18:23:04.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fada97fa640 1 -- 172.21.15.7:0/3814579712 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad980efed0 con 0x7fadac0ca280 2023-11-28T18:23:04.977 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:23:04.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066950 msgr2=0x7fad90068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066950 0x7fad90068e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fada000b9f0 tx=0x7fada0008040 comp rx=0 tx=0).stop 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 msgr2=0x7fadac153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac153900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fad980ad040 tx=0x7fad9801ac90 comp rx=0 tx=0).stop 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 shutdown_connections 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066950 0x7fad90068e10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac0cf5e0 0x7fadac1547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac0ca280 0x7fadac153900 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 --2- 172.21.15.7:0/3814579712 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac0b3da0 0x7fadac0c5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 >> 172.21.15.7:0/3814579712 conn(0x7fadac071210 msgr2=0x7fadac1595e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 shutdown_connections 2023-11-28T18:23:04.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:04.972+0000 7fadb1817640 1 -- 172.21.15.7:0/3814579712 wait complete. 2023-11-28T18:23:04.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T18:23:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9152879640 1 Processor -- start 2023-11-28T18:23:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9152879640 1 -- start start 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9152879640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f914c150b30 0x7f914c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9152879640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f914c1575c0 0x7f914c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9152879640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9152879640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914c0c72b0 con 0x7f914c1575c0 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9152879640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914c070c00 con 0x7f914c150b30 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9152879640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0c6140 con 0x7f914c0c4020 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f914c1575c0 0x7f914c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f914c1575c0 0x7f914c0c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48062/0 (socket says 172.21.15.7:48062) 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914b7fe640 1 -- 172.21.15.7:0/2227254883 learned_addr learned my addr 172.21.15.7:0/2227254883 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:05.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914bfff640 1 --2- 172.21.15.7:0/2227254883 >> v2:172.21.15.193:3300/0 conn(0x7f914c150b30 0x7f914c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f9150e16640 1 --2- 172.21.15.7:0/2227254883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914b7fe640 1 -- 172.21.15.7:0/2227254883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 msgr2=0x7f914c0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:05.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914b7fe640 1 --2- 172.21.15.7:0/2227254883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c0c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914b7fe640 1 -- 172.21.15.7:0/2227254883 >> v2:172.21.15.193:3300/0 conn(0x7f914c150b30 msgr2=0x7f914c0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:05.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914b7fe640 1 --2- 172.21.15.7:0/2227254883 >> v2:172.21.15.193:3300/0 conn(0x7f914c150b30 0x7f914c0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.136+0000 7f914b7fe640 1 -- 172.21.15.7:0/2227254883 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c0c6370 con 0x7f914c1575c0 2023-11-28T18:23:05.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f914b7fe640 1 --2- 172.21.15.7:0/2227254883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c3ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f914c06ec80 tx=0x7f914001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be3373694f0e239 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f914affd640 1 -- 172.21.15.7:0/2227254883 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91400a89b0 con 0x7f914c1575c0 2023-11-28T18:23:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f914affd640 1 -- 172.21.15.7:0/2227254883 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91400a8b10 con 0x7f914c1575c0 2023-11-28T18:23:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f914affd640 1 -- 172.21.15.7:0/2227254883 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91400a8ea0 con 0x7f914c1575c0 2023-11-28T18:23:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 -- 172.21.15.7:0/2227254883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 msgr2=0x7f914c0c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 --2- 172.21.15.7:0/2227254883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c3ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f914c06ec80 tx=0x7f914001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 -- 172.21.15.7:0/2227254883 shutdown_connections 2023-11-28T18:23:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 --2- 172.21.15.7:0/2227254883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 --2- 172.21.15.7:0/2227254883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c3ae0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 --2- 172.21.15.7:0/2227254883 >> v2:172.21.15.193:3300/0 conn(0x7f914c150b30 0x7f914c0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 -- 172.21.15.7:0/2227254883 >> 172.21.15.7:0/2227254883 conn(0x7f914c0662e0 msgr2=0x7f914c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:05.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 -- 172.21.15.7:0/2227254883 shutdown_connections 2023-11-28T18:23:05.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.140+0000 7f9152879640 1 -- 172.21.15.7:0/2227254883 wait complete. 2023-11-28T18:23:05.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 Processor -- start 2023-11-28T18:23:05.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 -- start start 2023-11-28T18:23:05.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f914c150b30 0x7f914c147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c1f3930 con 0x7f914c1575c0 2023-11-28T18:23:05.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c1e1640 con 0x7f914c150b30 2023-11-28T18:23:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c071fd0 con 0x7f914c0c4020 2023-11-28T18:23:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48066/0 (socket says 172.21.15.7:48066) 2023-11-28T18:23:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 -- 172.21.15.7:0/2790233069 learned_addr learned my addr 172.21.15.7:0/2790233069 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f914b7fe640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f914c150b30 0x7f914c147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f914bfff640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 -- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 msgr2=0x7f914c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 -- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f914c150b30 msgr2=0x7f914c147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f914c150b30 0x7f914c147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 -- 172.21.15.7:0/2790233069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91400094d0 con 0x7f914c1575c0 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f914b7fe640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f914c150b30 0x7f914c147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f914bfff640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9150e16640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c4510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9144007e60 tx=0x7f914400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9148ff9640 1 -- 172.21.15.7:0/2790233069 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9144014620 con 0x7f914c1575c0 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9148ff9640 1 -- 172.21.15.7:0/2790233069 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914400ee90 con 0x7f914c1575c0 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9148ff9640 1 -- 172.21.15.7:0/2790233069 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91440149e0 con 0x7f914c1575c0 2023-11-28T18:23:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.144+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c1f2cb0 con 0x7f914c1575c0 2023-11-28T18:23:05.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.148+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c1f3270 con 0x7f914c1575c0 2023-11-28T18:23:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.148+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c1f3a70 con 0x7f914c1575c0 2023-11-28T18:23:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.148+0000 7f9148ff9640 1 -- 172.21.15.7:0/2790233069 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9144014b40 con 0x7f914c1575c0 2023-11-28T18:23:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.148+0000 7f9148ff9640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9138066900 0x7f9138068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.148+0000 7f9148ff9640 1 -- 172.21.15.7:0/2790233069 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9144084a90 con 0x7f914c1575c0 2023-11-28T18:23:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.148+0000 7f914bfff640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9138066900 0x7f9138068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.148+0000 7f914bfff640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9138066900 0x7f9138068dc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f913c007a70 tx=0x7f913c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:05.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.152+0000 7f9148ff9640 1 -- 172.21.15.7:0/2790233069 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f914c1f3a70 con 0x7f914c1575c0 2023-11-28T18:23:05.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.312+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f914c1f3a70 con 0x7f914c1575c0 2023-11-28T18:23:05.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.312+0000 7f9148ff9640 1 -- 172.21.15.7:0/2790233069 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f914c1f3a70 con 0x7f914c1575c0 2023-11-28T18:23:05.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:05.319 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2023-11-28T18:23:05.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-11-28T18:23:05.319 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 1, 2023-11-28T18:23:05.319 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": ".mgr" 2023-11-28T18:23:05.320 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-11-28T18:23:05.320 INFO:teuthology.orchestra.run.smithi007.stdout:] 2023-11-28T18:23:05.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9138066900 msgr2=0x7f9138068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9138066900 0x7f9138068dc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f913c007a70 tx=0x7f913c008040 comp rx=0 tx=0).stop 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 msgr2=0x7f914c0c4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c4510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9144007e60 tx=0x7f914400e4b0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 shutdown_connections 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9138066900 0x7f9138068dc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f914c1575c0 0x7f914c0c4510 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f914c150b30 0x7f914c147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 --2- 172.21.15.7:0/2790233069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f914c0c4020 0x7f914c071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 >> 172.21.15.7:0/2790233069 conn(0x7f914c0662e0 msgr2=0x7f914c14f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:05.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 shutdown_connections 2023-11-28T18:23:05.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.316+0000 7f9152879640 1 -- 172.21.15.7:0/2790233069 wait complete. 2023-11-28T18:23:05.338 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T18:23:05.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T18:23:05.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.480+0000 7f6eb2bdf640 1 Processor -- start 2023-11-28T18:23:05.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.480+0000 7f6eb2bdf640 1 -- start start 2023-11-28T18:23:05.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.480+0000 7f6eb2bdf640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6eac0c5ff0 0x7f6eac072a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.480+0000 7f6eb2bdf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eac070010 con 0x7f6eac152100 2023-11-28T18:23:05.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.480+0000 7f6eb2bdf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eac06e680 con 0x7f6eac151d20 2023-11-28T18:23:05.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.480+0000 7f6eb2bdf640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eac072610 con 0x7f6eac0c5ff0 2023-11-28T18:23:05.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eb117c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6eac0c5ff0 0x7f6eac072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eb117c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6eac0c5ff0 0x7f6eac072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48080/0 (socket says 172.21.15.7:48080) 2023-11-28T18:23:05.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eb117c640 1 -- 172.21.15.7:0/1893549169 learned_addr learned my addr 172.21.15.7:0/1893549169 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:05.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eabfff640 1 --1- 172.21.15.7:0/1893549169 >> v1:172.21.15.7:6789/0 conn(0x7f6eac152100 0x7f6eac0c4c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33872/0 (socket says 172.21.15.7:33872) 2023-11-28T18:23:05.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eb117c640 1 -- 172.21.15.7:0/1893549169 >> v1:172.21.15.193:6789/0 conn(0x7f6eac151d20 legacy=0x7f6eac158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eb117c640 1 -- 172.21.15.7:0/1893549169 >> v1:172.21.15.7:6789/0 conn(0x7f6eac152100 legacy=0x7f6eac0c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eb117c640 1 -- 172.21.15.7:0/1893549169 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eac070240 con 0x7f6eac0c5ff0 2023-11-28T18:23:05.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eb117c640 1 --2- 172.21.15.7:0/1893549169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac0c5ff0 0x7f6eac072a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009e50 tx=0x7f6ea401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b1b9fbe38d75c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eab7fe640 1 -- 172.21.15.7:0/1893549169 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40af400 con 0x7f6eac0c5ff0 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eab7fe640 1 -- 172.21.15.7:0/1893549169 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea40ab050 con 0x7f6eac0c5ff0 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.484+0000 7f6eab7fe640 1 -- 172.21.15.7:0/1893549169 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40af780 con 0x7f6eac0c5ff0 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/1893549169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac0c5ff0 msgr2=0x7f6eac072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 --2- 172.21.15.7:0/1893549169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac0c5ff0 0x7f6eac072a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009e50 tx=0x7f6ea401f1f0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/1893549169 shutdown_connections 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 --2- 172.21.15.7:0/1893549169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac0c5ff0 0x7f6eac072a30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/1893549169 >> 172.21.15.7:0/1893549169 conn(0x7f6eac06c7a0 msgr2=0x7f6eac0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/1893549169 shutdown_connections 2023-11-28T18:23:05.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/1893549169 wait complete. 2023-11-28T18:23:05.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 Processor -- start 2023-11-28T18:23:05.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- start start 2023-11-28T18:23:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6eac0c5ff0 0x7f6eac14f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac151d20 0x7f6eac0c3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eac152100 0x7f6eac15a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac14f7e0 con 0x7f6eac151d20 2023-11-28T18:23:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac158dd0 con 0x7f6eac152100 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb2bdf640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac15c070 con 0x7f6eac0c5ff0 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac151d20 0x7f6eac0c3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac151d20 0x7f6eac0c3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48088/0 (socket says 172.21.15.7:48088) 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eabfff640 1 -- 172.21.15.7:0/812455774 learned_addr learned my addr 172.21.15.7:0/812455774 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb117c640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eac152100 0x7f6eac15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eb097b640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6eac0c5ff0 0x7f6eac14f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eabfff640 1 -- 172.21.15.7:0/812455774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6eac0c5ff0 msgr2=0x7f6eac14f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eabfff640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6eac0c5ff0 0x7f6eac14f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eabfff640 1 -- 172.21.15.7:0/812455774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eac152100 msgr2=0x7f6eac15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eabfff640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eac152100 0x7f6eac15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.488+0000 7f6eabfff640 1 -- 172.21.15.7:0/812455774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4009b30 con 0x7f6eac151d20 2023-11-28T18:23:05.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6eabfff640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac151d20 0x7f6eac0c3d10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c430 tx=0x7f6e9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:05.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6ea97fa640 1 -- 172.21.15.7:0/812455774 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e980074e0 con 0x7f6eac151d20 2023-11-28T18:23:05.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6ea97fa640 1 -- 172.21.15.7:0/812455774 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e98007640 con 0x7f6eac151d20 2023-11-28T18:23:05.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6ea97fa640 1 -- 172.21.15.7:0/812455774 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98004a10 con 0x7f6eac151d20 2023-11-28T18:23:05.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac1f0a00 con 0x7f6eac151d20 2023-11-28T18:23:05.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac159300 con 0x7f6eac151d20 2023-11-28T18:23:05.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac04e890 con 0x7f6eac151d20 2023-11-28T18:23:05.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6ea97fa640 1 -- 172.21.15.7:0/812455774 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6e98004b70 con 0x7f6eac151d20 2023-11-28T18:23:05.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6ea97fa640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e900668d0 0x7f6e90068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:05.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6ea97fa640 1 -- 172.21.15.7:0/812455774 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6e98083ae0 con 0x7f6eac151d20 2023-11-28T18:23:05.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6eb097b640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e900668d0 0x7f6e90068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:05.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.492+0000 7f6eb097b640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e900668d0 0x7f6e90068d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007fd0 tx=0x7f6ea0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:05.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.496+0000 7f6ea97fa640 1 -- 172.21.15.7:0/812455774 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6e98050ff0 con 0x7f6eac151d20 2023-11-28T18:23:05.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:05.656+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6eac155860 con 0x7f6eac151d20 2023-11-28T18:23:06.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.636+0000 7f6ea97fa640 1 -- 172.21.15.7:0/812455774 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6e98054610 con 0x7f6eac151d20 2023-11-28T18:23:06.641 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' created 2023-11-28T18:23:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e900668d0 msgr2=0x7f6e90068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e900668d0 0x7f6e90068d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007fd0 tx=0x7f6ea0009040 comp rx=0 tx=0).stop 2023-11-28T18:23:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac151d20 msgr2=0x7f6eac0c3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac151d20 0x7f6eac0c3d10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c430 tx=0x7f6e9800c8f0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 shutdown_connections 2023-11-28T18:23:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e900668d0 0x7f6e90068d90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eac152100 0x7f6eac15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6eac151d20 0x7f6eac0c3d10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 --2- 172.21.15.7:0/812455774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6eac0c5ff0 0x7f6eac14f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 >> 172.21.15.7:0/812455774 conn(0x7f6eac06c7a0 msgr2=0x7f6eac150e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 shutdown_connections 2023-11-28T18:23:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.640+0000 7f6eb2bdf640 1 -- 172.21.15.7:0/812455774 wait complete. 2023-11-28T18:23:06.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T18:23:06.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3bc39d640 1 Processor -- start 2023-11-28T18:23:06.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3bc39d640 1 -- start start 2023-11-28T18:23:06.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3bc39d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3b406e7c0 0x7fd3b4072880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:06.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3bc39d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b4157f60 con 0x7fd3b406e3a0 2023-11-28T18:23:06.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3bc39d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b4158240 con 0x7fd3b4141030 2023-11-28T18:23:06.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3bc39d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4158520 con 0x7fd3b406e7c0 2023-11-28T18:23:06.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9938640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd3b406e3a0 0x7fd3b40c4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33882/0 (socket says 172.21.15.7:33882) 2023-11-28T18:23:06.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9938640 1 -- 172.21.15.7:0/1834183027 learned_addr learned my addr 172.21.15.7:0/1834183027 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:06.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3ba93a640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3b406e7c0 0x7fd3b4072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:06.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970194700 0 0) 0x7fd3b4157f60 con 0x7fd3b406e3a0 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a0003530 con 0x7fd3b406e3a0 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386618437 0 0) 0x7fd3a0003530 con 0x7fd3b406e3a0 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3b406e7c0 msgr2=0x7fd3b4072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3b406e7c0 0x7fd3b4072880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 >> v1:172.21.15.193:6789/0 conn(0x7fd3b4141030 legacy=0x7fd3b415be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b4158800 con 0x7fd3b406e3a0 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fd3a4001680 con 0x7fd3b406e3a0 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 >> v1:172.21.15.7:6789/0 conn(0x7fd3b406e3a0 legacy=0x7fd3b40c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3b406e7c0 0x7fd3a00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0003f30 0x7fd3a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a0004d10 0x7fd3a0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4158520 con 0x7fd3a0003f30 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a000bb00 con 0x7fd3a0004d10 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a000bcc0 con 0x7fd3b406e7c0 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9938640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0003f30 0x7fd3a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3ba139640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3b406e7c0 0x7fd3a00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3ba93a640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a0004d10 0x7fd3a0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9938640 1 -- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3b406e7c0 msgr2=0x7fd3a00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9938640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3b406e7c0 0x7fd3a00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9938640 1 -- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a0004d10 msgr2=0x7fd3a0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9938640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a0004d10 0x7fd3a0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.824+0000 7fd3b9938640 1 -- 172.21.15.7:0/1834183027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a0003530 con 0x7fd3a0003f30 2023-11-28T18:23:06.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3b9938640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0003f30 0x7fd3a00047d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c1c0 tx=0x7fd3a401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0e74728bc50286 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:06.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40a5b20 con 0x7fd3a0003f30 2023-11-28T18:23:06.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a40a5c80 con 0x7fd3a0003f30 2023-11-28T18:23:06.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3b9137640 1 -- 172.21.15.7:0/1834183027 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4005b20 con 0x7fd3a0003f30 2023-11-28T18:23:06.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1834183027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0003f30 msgr2=0x7fd3a00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:06.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3bc39d640 1 --2- 172.21.15.7:0/1834183027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0003f30 0x7fd3a00047d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c1c0 tx=0x7fd3a401cc20 comp rx=0 tx=0).stop 2023-11-28T18:23:06.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3ba93a640 1 -- 172.21.15.7:0/1834183027 reap_dead start 2023-11-28T18:23:06.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1834183027 shutdown_connections 2023-11-28T18:23:06.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1834183027 >> 172.21.15.7:0/1834183027 conn(0x7fd3b4073f00 msgr2=0x7fd3b40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:06.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1834183027 shutdown_connections 2023-11-28T18:23:06.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.828+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1834183027 wait complete. 2023-11-28T18:23:06.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 Processor -- start 2023-11-28T18:23:06.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 -- start start 2023-11-28T18:23:06.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3a0003f30 0x7fd3b413a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b41e6840 0x7fd3b41f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3b41e6ca0 0x7fd3b4134b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b413ac60 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4155700 con 0x7fd3b41e6840 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b41528a0 con 0x7fd3a0003f30 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3b41e6ca0 0x7fd3b4134b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3b41e6ca0 0x7fd3b4134b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48102/0 (socket says 172.21.15.7:48102) 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 -- 172.21.15.7:0/1223605454 learned_addr learned my addr 172.21.15.7:0/1223605454 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:06.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba139640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3a0003f30 0x7fd3b413a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:06.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3b9938640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b41e6840 0x7fd3b41f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:06.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 -- 172.21.15.7:0/1223605454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3a0003f30 msgr2=0x7fd3b413a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:06.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3a0003f30 0x7fd3b413a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 -- 172.21.15.7:0/1223605454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b41e6840 msgr2=0x7fd3b41f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:06.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b41e6840 0x7fd3b41f3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:06.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 -- 172.21.15.7:0/1223605454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b413a720 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3ba93a640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3b41e6ca0 0x7fd3b4134b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4155a40 tx=0x7fd3ac00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:06.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3aaffd640 1 -- 172.21.15.7:0/1223605454 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00e040 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3aaffd640 1 -- 172.21.15.7:0/1223605454 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac00f040 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b4155140 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3aaffd640 1 -- 172.21.15.7:0/1223605454 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac015430 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.832+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b4071710 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.836+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b413ada0 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.836+0000 7fd3aaffd640 1 -- 172.21.15.7:0/1223605454 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd3ac015590 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.836+0000 7fd3aaffd640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd3980668d0 0x7fd398068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.836+0000 7fd3aaffd640 1 -- 172.21.15.7:0/1223605454 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd3ac085300 con 0x7fd3b41e6ca0 2023-11-28T18:23:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.836+0000 7fd3ba139640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd3980668d0 0x7fd398068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:06.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.840+0000 7fd3ba139640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd3980668d0 0x7fd398068d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0007fd0 tx=0x7fd3b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:06.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:06.840+0000 7fd3aaffd640 1 -- 172.21.15.7:0/1223605454 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd3ac052700 con 0x7fd3b41e6ca0 2023-11-28T18:23:07.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.000+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd3b413ada0 con 0x7fd3b41e6ca0 2023-11-28T18:23:07.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.640+0000 7fd3aaffd640 1 -- 172.21.15.7:0/1223605454 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd3b413ada0 con 0x7fd3b41e6ca0 2023-11-28T18:23:07.644 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' created 2023-11-28T18:23:07.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd3980668d0 msgr2=0x7fd398068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:07.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd3980668d0 0x7fd398068d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0007fd0 tx=0x7fd3b0009040 comp rx=0 tx=0).stop 2023-11-28T18:23:07.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3b41e6ca0 msgr2=0x7fd3b4134b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:07.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3b41e6ca0 0x7fd3b4134b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4155a40 tx=0x7fd3ac00f580 comp rx=0 tx=0).stop 2023-11-28T18:23:07.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 shutdown_connections 2023-11-28T18:23:07.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd3980668d0 0x7fd398068d90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3b41e6ca0 0x7fd3b4134b50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b41e6840 0x7fd3b41f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 --2- 172.21.15.7:0/1223605454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3a0003f30 0x7fd3b413a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 >> 172.21.15.7:0/1223605454 conn(0x7fd3b4073f00 msgr2=0x7fd3b40c4660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:07.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 shutdown_connections 2023-11-28T18:23:07.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.644+0000 7fd3bc39d640 1 -- 172.21.15.7:0/1223605454 wait complete. 2023-11-28T18:23:07.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T18:23:07.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.828+0000 7f704cff5640 1 Processor -- start 2023-11-28T18:23:07.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.828+0000 7f704cff5640 1 -- start start 2023-11-28T18:23:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.828+0000 7f704cff5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70481512c0 0x7f70480c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.828+0000 7f704cff5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70480c7f10 0x7f70480c8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.828+0000 7f704cff5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70480c5e80 con 0x7f7048150ee0 2023-11-28T18:23:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.828+0000 7f704cff5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7048070c10 con 0x7f70480c7f10 2023-11-28T18:23:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.828+0000 7f704cff5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70480c6f10 con 0x7f70481512c0 2023-11-28T18:23:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7046d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70480c7f10 0x7f70480c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7046d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70480c7f10 0x7f70480c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48104/0 (socket says 172.21.15.7:48104) 2023-11-28T18:23:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7046d9d640 1 -- 172.21.15.7:0/1587443112 learned_addr learned my addr 172.21.15.7:0/1587443112 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7046d9d640 1 -- 172.21.15.7:0/1587443112 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70481512c0 msgr2=0x7f70480c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f703ffff640 1 --2- 172.21.15.7:0/1587443112 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70481512c0 0x7f70480c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7046d9d640 1 --2- 172.21.15.7:0/1587443112 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70481512c0 0x7f70480c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7046d9d640 1 -- 172.21.15.7:0/1587443112 >> v1:172.21.15.193:6789/0 conn(0x7f7048150ee0 legacy=0x7f7048157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7046d9d640 1 -- 172.21.15.7:0/1587443112 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70480c2c90 con 0x7f70480c7f10 2023-11-28T18:23:07.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f703ffff640 1 --2- 172.21.15.7:0/1587443112 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70481512c0 0x7f70480c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7046d9d640 1 --2- 172.21.15.7:0/1587443112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f70480c8380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7040009e50 tx=0x7f704001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fad0f0f56bbfd2f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7045d9b640 1 -- 172.21.15.7:0/1587443112 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70400a5820 con 0x7f70480c7f10 2023-11-28T18:23:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7045d9b640 1 -- 172.21.15.7:0/1587443112 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70400a5980 con 0x7f70480c7f10 2023-11-28T18:23:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.832+0000 7f7045d9b640 1 -- 172.21.15.7:0/1587443112 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70400a5d10 con 0x7f70480c7f10 2023-11-28T18:23:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- 172.21.15.7:0/1587443112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 msgr2=0x7f70480c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 --2- 172.21.15.7:0/1587443112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f70480c8380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7040009e50 tx=0x7f704001c9a0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- 172.21.15.7:0/1587443112 shutdown_connections 2023-11-28T18:23:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 --2- 172.21.15.7:0/1587443112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f70480c8380 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 --2- 172.21.15.7:0/1587443112 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70481512c0 0x7f70480c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- 172.21.15.7:0/1587443112 >> 172.21.15.7:0/1587443112 conn(0x7f7048073f00 msgr2=0x7f70480662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:07.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- 172.21.15.7:0/1587443112 shutdown_connections 2023-11-28T18:23:07.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- 172.21.15.7:0/1587443112 wait complete. 2023-11-28T18:23:07.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 Processor -- start 2023-11-28T18:23:07.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- start start 2023-11-28T18:23:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f7048145e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7048150ee0 0x7f704813ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70481512c0 0x7f704814b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70481ee8a0 con 0x7f70480c7f10 2023-11-28T18:23:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70481463b0 con 0x7f70481512c0 2023-11-28T18:23:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704cff5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704813f010 con 0x7f7048150ee0 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f7048145e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f7048145e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48112/0 (socket says 172.21.15.7:48112) 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f704659c640 1 -- 172.21.15.7:0/4016815360 learned_addr learned my addr 172.21.15.7:0/4016815360 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f703ffff640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7048150ee0 0x7f704813ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.836+0000 7f7046d9d640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70481512c0 0x7f704814b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704659c640 1 -- 172.21.15.7:0/4016815360 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7048150ee0 msgr2=0x7f704813ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704659c640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7048150ee0 0x7f704813ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704659c640 1 -- 172.21.15.7:0/4016815360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70481512c0 msgr2=0x7f704814b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704659c640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70481512c0 0x7f704814b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704659c640 1 -- 172.21.15.7:0/4016815360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040009b30 con 0x7f70480c7f10 2023-11-28T18:23:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704659c640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f7048145e70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f703800c4b0 tx=0x7f703800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:07.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f703effd640 1 -- 172.21.15.7:0/4016815360 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038012830 con 0x7f70480c7f10 2023-11-28T18:23:07.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f703effd640 1 -- 172.21.15.7:0/4016815360 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70380129b0 con 0x7f70480c7f10 2023-11-28T18:23:07.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704813fe20 con 0x7f70480c7f10 2023-11-28T18:23:07.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f703effd640 1 -- 172.21.15.7:0/4016815360 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038012d60 con 0x7f70480c7f10 2023-11-28T18:23:07.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7048140510 con 0x7f70480c7f10 2023-11-28T18:23:07.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704804e890 con 0x7f70480c7f10 2023-11-28T18:23:07.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.840+0000 7f703effd640 1 -- 172.21.15.7:0/4016815360 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f703801c460 con 0x7f70480c7f10 2023-11-28T18:23:07.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.844+0000 7f703effd640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7028066900 0x7f7028068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:07.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.844+0000 7f703effd640 1 -- 172.21.15.7:0/4016815360 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f70380841e0 con 0x7f70480c7f10 2023-11-28T18:23:07.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.844+0000 7f703ffff640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7028066900 0x7f7028068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:07.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.844+0000 7f703ffff640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7028066900 0x7f7028068dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f70480c28a0 tx=0x7f7030008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:07.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:07.848+0000 7f703effd640 1 -- 172.21.15.7:0/4016815360 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7038051400 con 0x7f70480c7f10 2023-11-28T18:23:08.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.036+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7048070f90 con 0x7f70480c7f10 2023-11-28T18:23:08.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f703effd640 1 -- 172.21.15.7:0/4016815360 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7038054a20 con 0x7f70480c7f10 2023-11-28T18:23:08.655 INFO:teuthology.orchestra.run.smithi007.stderr:new fs with metadata pool 2 and data pool 3 2023-11-28T18:23:08.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7028066900 msgr2=0x7f7028068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:08.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7028066900 0x7f7028068dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f70480c28a0 tx=0x7f7030008040 comp rx=0 tx=0).stop 2023-11-28T18:23:08.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 msgr2=0x7f7048145e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:08.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f7048145e70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f703800c4b0 tx=0x7f703800c970 comp rx=0 tx=0).stop 2023-11-28T18:23:08.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 shutdown_connections 2023-11-28T18:23:08.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7028066900 0x7f7028068dc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70481512c0 0x7f704814b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7048150ee0 0x7f704813ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 --2- 172.21.15.7:0/4016815360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70480c7f10 0x7f7048145e70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 >> 172.21.15.7:0/4016815360 conn(0x7f7048073f00 msgr2=0x7f704806e480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:08.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 shutdown_connections 2023-11-28T18:23:08.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.652+0000 7f704cff5640 1 -- 172.21.15.7:0/4016815360 wait complete. 2023-11-28T18:23:08.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.824+0000 7f7656a5a640 1 Processor -- start 2023-11-28T18:23:08.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.824+0000 7f7656a5a640 1 -- start start 2023-11-28T18:23:08.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f7656a5a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76501503c0 0x7f76500c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:08.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f7656a5a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f76500c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:08.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f7656a5a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76500c3920 con 0x7f76500c3bc0 2023-11-28T18:23:08.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f7656a5a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7650157240 con 0x7f76501503c0 2023-11-28T18:23:08.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f7656a5a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76500c2ce0 con 0x7f76500c3490 2023-11-28T18:23:08.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f764ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76501503c0 0x7f76500c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:08.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f764ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76501503c0 0x7f76500c7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48126/0 (socket says 172.21.15.7:48126) 2023-11-28T18:23:08.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f764ffff640 1 -- 172.21.15.7:0/3260774213 learned_addr learned my addr 172.21.15.7:0/3260774213 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:08.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f7654ff7640 1 --2- 172.21.15.7:0/3260774213 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f76500c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:08.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f764ffff640 1 -- 172.21.15.7:0/3260774213 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 msgr2=0x7f76500c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:08.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f764ffff640 1 --2- 172.21.15.7:0/3260774213 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f76500c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f764ffff640 1 -- 172.21.15.7:0/3260774213 >> v1:172.21.15.193:6789/0 conn(0x7f76500c3bc0 legacy=0x7f765006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:08.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.828+0000 7f764ffff640 1 -- 172.21.15.7:0/3260774213 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76500c2f80 con 0x7f76501503c0 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f764ffff640 1 --2- 172.21.15.7:0/3260774213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76501503c0 0x7f76500c7c60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7640009870 tx=0x7f764001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e090d4b309a34c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f764effd640 1 -- 172.21.15.7:0/3260774213 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76400a8a30 con 0x7f76501503c0 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f764effd640 1 -- 172.21.15.7:0/3260774213 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76400a8b90 con 0x7f76501503c0 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f764effd640 1 -- 172.21.15.7:0/3260774213 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76400b0440 con 0x7f76501503c0 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 -- 172.21.15.7:0/3260774213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76501503c0 msgr2=0x7f76500c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 --2- 172.21.15.7:0/3260774213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76501503c0 0x7f76500c7c60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7640009870 tx=0x7f764001ed30 comp rx=0 tx=0).stop 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 -- 172.21.15.7:0/3260774213 shutdown_connections 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 --2- 172.21.15.7:0/3260774213 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f76500c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 --2- 172.21.15.7:0/3260774213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76501503c0 0x7f76500c7c60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 -- 172.21.15.7:0/3260774213 >> 172.21.15.7:0/3260774213 conn(0x7f76500662e0 msgr2=0x7f76500666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:08.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 -- 172.21.15.7:0/3260774213 shutdown_connections 2023-11-28T18:23:08.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 -- 172.21.15.7:0/3260774213 wait complete. 2023-11-28T18:23:08.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 Processor -- start 2023-11-28T18:23:08.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.832+0000 7f7656a5a640 1 -- start start 2023-11-28T18:23:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f7650141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76500c3bc0 0x7f76500c19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76501503c0 0x7f76501f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76501f8580 con 0x7f76500c3bc0 2023-11-28T18:23:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7650152a10 con 0x7f76501503c0 2023-11-28T18:23:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7650141850 con 0x7f76500c3490 2023-11-28T18:23:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76500c3bc0 0x7f76500c19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76500c3bc0 0x7f76500c19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48140/0 (socket says 172.21.15.7:48140) 2023-11-28T18:23:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 -- 172.21.15.7:0/564933555 learned_addr learned my addr 172.21.15.7:0/564933555 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764ffff640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f7650141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7654ff7640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76501503c0 0x7f76501f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 -- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 msgr2=0x7f7650141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f7650141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 -- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76501503c0 msgr2=0x7f76501f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76501503c0 0x7f76501f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 -- 172.21.15.7:0/564933555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76400094d0 con 0x7f76500c3bc0 2023-11-28T18:23:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764ffff640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f7650141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7654ff7640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76501503c0 0x7f76501f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764f7fe640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76500c3bc0 0x7f76500c19a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f764400c530 tx=0x7f764400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764cff9640 1 -- 172.21.15.7:0/564933555 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644005220 con 0x7f76500c3bc0 2023-11-28T18:23:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764cff9640 1 -- 172.21.15.7:0/564933555 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644005380 con 0x7f76500c3bc0 2023-11-28T18:23:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764cff9640 1 -- 172.21.15.7:0/564933555 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644007270 con 0x7f76500c3bc0 2023-11-28T18:23:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76501f7fe0 con 0x7f76500c3bc0 2023-11-28T18:23:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7650142330 con 0x7f76500c3bc0 2023-11-28T18:23:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76501f4060 con 0x7f76500c3bc0 2023-11-28T18:23:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.836+0000 7f764cff9640 1 -- 172.21.15.7:0/564933555 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f76440073d0 con 0x7f76500c3bc0 2023-11-28T18:23:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.840+0000 7f764cff9640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f763c0668d0 0x7f763c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.840+0000 7f764cff9640 1 -- 172.21.15.7:0/564933555 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7644083c70 con 0x7f76500c3bc0 2023-11-28T18:23:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.840+0000 7f764ffff640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f763c0668d0 0x7f763c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.840+0000 7f764ffff640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f763c0668d0 0x7f763c068d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76400b2000 tx=0x7f7640008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:08.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:08.844+0000 7f764cff9640 1 -- 172.21.15.7:0/564933555 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7644050ec0 con 0x7f76500c3bc0 2023-11-28T18:23:09.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.000+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76500c3920 con 0x7f76500c3bc0 2023-11-28T18:23:09.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f764cff9640 1 -- 172.21.15.7:0/564933555 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16458 (secure 0 0 0) 0x7f76500c3920 con 0x7f76500c3bc0 2023-11-28T18:23:09.008 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:09.008 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:08.041261+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:09.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f763c0668d0 msgr2=0x7f763c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f763c0668d0 0x7f763c068d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76400b2000 tx=0x7f7640008040 comp rx=0 tx=0).stop 2023-11-28T18:23:09.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76500c3bc0 msgr2=0x7f76500c19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76500c3bc0 0x7f76500c19a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f764400c530 tx=0x7f764400c9f0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 shutdown_connections 2023-11-28T18:23:09.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f763c0668d0 0x7f763c068d90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76501503c0 0x7f76501f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76500c3bc0 0x7f76500c19a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 --2- 172.21.15.7:0/564933555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f76500c3490 0x7f7650141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 >> 172.21.15.7:0/564933555 conn(0x7f76500662e0 msgr2=0x7f7650147010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:09.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 shutdown_connections 2023-11-28T18:23:09.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.004+0000 7f7656a5a640 1 -- 172.21.15.7:0/564933555 wait complete. 2023-11-28T18:23:09.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:09.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.168+0000 7f2ceb079640 1 Processor -- start 2023-11-28T18:23:09.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ceb079640 1 -- start start 2023-11-28T18:23:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ceb079640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 0x7f2ce41576a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ceb079640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ce40c3e00 0x7f2ce415a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ceb079640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce40c36b0 con 0x7f2ce4150da0 2023-11-28T18:23:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ceb079640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce40c3be0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ceb079640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce415a1f0 con 0x7f2ce40c2a60 2023-11-28T18:23:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce8e15640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2ce4150da0 0x7f2ce40c7070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33888/0 (socket says 172.21.15.7:33888) 2023-11-28T18:23:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce8e15640 1 -- 172.21.15.7:0/2063403546 learned_addr learned my addr 172.21.15.7:0/2063403546 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce9616640 1 --2- 172.21.15.7:0/2063403546 >> v2:172.21.15.7:3300/0 conn(0x7f2ce40c3e00 0x7f2ce415a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce3fff640 1 --2- 172.21.15.7:0/2063403546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 0x7f2ce41576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce37fe640 1 -- 172.21.15.7:0/2063403546 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131241660 0 0) 0x7f2ce40c36b0 con 0x7f2ce4150da0 2023-11-28T18:23:09.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce37fe640 1 -- 172.21.15.7:0/2063403546 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd4003530 con 0x7f2ce4150da0 2023-11-28T18:23:09.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce9616640 1 -- 172.21.15.7:0/2063403546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 msgr2=0x7f2ce41576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:09.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce9616640 1 --2- 172.21.15.7:0/2063403546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 0x7f2ce41576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce9616640 1 -- 172.21.15.7:0/2063403546 >> v1:172.21.15.7:6789/0 conn(0x7f2ce4150da0 legacy=0x7f2ce40c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.172+0000 7f2ce9616640 1 -- 172.21.15.7:0/2063403546 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce40c3900 con 0x7f2ce40c3e00 2023-11-28T18:23:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ce9616640 1 --2- 172.21.15.7:0/2063403546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce415a680 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009d90 tx=0x7f2cdc01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e165f5935426eed4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ce37fe640 1 -- 172.21.15.7:0/2063403546 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0a5950 con 0x7f2ce40c3e00 2023-11-28T18:23:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ce37fe640 1 -- 172.21.15.7:0/2063403546 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc0a5ab0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ce37fe640 1 -- 172.21.15.7:0/2063403546 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0a5e20 con 0x7f2ce40c3e00 2023-11-28T18:23:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 -- 172.21.15.7:0/2063403546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 msgr2=0x7f2ce415a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 --2- 172.21.15.7:0/2063403546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce415a680 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009d90 tx=0x7f2cdc01c890 comp rx=0 tx=0).stop 2023-11-28T18:23:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 -- 172.21.15.7:0/2063403546 shutdown_connections 2023-11-28T18:23:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 --2- 172.21.15.7:0/2063403546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce415a680 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 --2- 172.21.15.7:0/2063403546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 0x7f2ce41576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 -- 172.21.15.7:0/2063403546 >> 172.21.15.7:0/2063403546 conn(0x7f2ce4073ea0 msgr2=0x7f2ce40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 -- 172.21.15.7:0/2063403546 shutdown_connections 2023-11-28T18:23:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 -- 172.21.15.7:0/2063403546 wait complete. 2023-11-28T18:23:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 Processor -- start 2023-11-28T18:23:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.176+0000 7f2ceb079640 1 -- start start 2023-11-28T18:23:09.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 0x7f2ce414e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce4072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ce4150da0 0x7f2ce41f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce41f34f0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce4073330 con 0x7f2ce4150da0 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce41f2870 con 0x7f2ce40c2a60 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce4072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce4072a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48164/0 (socket says 172.21.15.7:48164) 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 -- 172.21.15.7:0/1846028990 learned_addr learned my addr 172.21.15.7:0/1846028990 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce9616640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ce4150da0 0x7f2ce41f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce8e15640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 0x7f2ce414e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 -- 172.21.15.7:0/1846028990 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 msgr2=0x7f2ce414e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 0x7f2ce414e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 -- 172.21.15.7:0/1846028990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ce4150da0 msgr2=0x7f2ce41f5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ce4150da0 0x7f2ce41f5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 -- 172.21.15.7:0/1846028990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc009a70 con 0x7f2ce40c3e00 2023-11-28T18:23:09.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce3fff640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce4072a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000d800 tx=0x7f2cd000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:09.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce17fa640 1 -- 172.21.15.7:0/1846028990 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd000b6c0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce17fa640 1 -- 172.21.15.7:0/1846028990 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd000b820 con 0x7f2ce40c3e00 2023-11-28T18:23:09.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ce17fa640 1 -- 172.21.15.7:0/1846028990 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd000bb90 con 0x7f2ce40c3e00 2023-11-28T18:23:09.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce41f2b50 con 0x7f2ce40c3e00 2023-11-28T18:23:09.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce41f3110 con 0x7f2ce40c3e00 2023-11-28T18:23:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.180+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce41f32d0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.184+0000 7f2ce17fa640 1 -- 172.21.15.7:0/1846028990 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2cd000bcf0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.184+0000 7f2ce17fa640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cc80668d0 0x7f2cc8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.184+0000 7f2ce17fa640 1 -- 172.21.15.7:0/1846028990 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cd0084aa0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.184+0000 7f2ce8e15640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cc80668d0 0x7f2cc8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.184+0000 7f2ce8e15640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cc80668d0 0x7f2cc8068d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8005890 tx=0x7f2cd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.188+0000 7f2ce17fa640 1 -- 172.21.15.7:0/1846028990 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2cd0051bb0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.348+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ce404e890 con 0x7f2ce40c3e00 2023-11-28T18:23:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.348+0000 7f2ce17fa640 1 -- 172.21.15.7:0/1846028990 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f2cd00551d0 con 0x7f2ce40c3e00 2023-11-28T18:23:09.354 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:09.354 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cc80668d0 msgr2=0x7f2cc8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cc80668d0 0x7f2cc8068d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8005890 tx=0x7f2cd8009040 comp rx=0 tx=0).stop 2023-11-28T18:23:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 msgr2=0x7f2ce4072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce4072a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000d800 tx=0x7f2cd000dcc0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 shutdown_connections 2023-11-28T18:23:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cc80668d0 0x7f2cc8068d90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ce4150da0 0x7f2ce41f5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce40c3e00 0x7f2ce4072a10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 --2- 172.21.15.7:0/1846028990 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2ce40c2a60 0x7f2ce414e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 >> 172.21.15.7:0/1846028990 conn(0x7f2ce4073ea0 msgr2=0x7f2ce4160dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 shutdown_connections 2023-11-28T18:23:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.352+0000 7f2ceb079640 1 -- 172.21.15.7:0/1846028990 wait complete. 2023-11-28T18:23:09.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T18:23:09.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.516+0000 7fb782d2a640 1 Processor -- start 2023-11-28T18:23:09.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb782d2a640 1 -- start start 2023-11-28T18:23:09.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb782d2a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb77c1540d0 0x7fb77c072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb782d2a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fb77c15db70 0x7fb77c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb782d2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77c15ae90 con 0x7fb77c0cd560 2023-11-28T18:23:09.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb782d2a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c15e4c0 con 0x7fb77c1540d0 2023-11-28T18:23:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb782d2a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c15a370 con 0x7fb77c15db70 2023-11-28T18:23:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb780ac6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb77c1540d0 0x7fb77c072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb780ac6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb77c1540d0 0x7fb77c072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48168/0 (socket says 172.21.15.7:48168) 2023-11-28T18:23:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb780ac6640 1 -- 172.21.15.7:0/3213905170 learned_addr learned my addr 172.21.15.7:0/3213905170 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb77bfff640 1 --2- 172.21.15.7:0/3213905170 >> v2:172.21.15.193:3300/0 conn(0x7fb77c15db70 0x7fb77c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb780ac6640 1 -- 172.21.15.7:0/3213905170 >> v2:172.21.15.193:3300/0 conn(0x7fb77c15db70 msgr2=0x7fb77c15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb780ac6640 1 --2- 172.21.15.7:0/3213905170 >> v2:172.21.15.193:3300/0 conn(0x7fb77c15db70 0x7fb77c15df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb780ac6640 1 -- 172.21.15.7:0/3213905170 >> v1:172.21.15.193:6789/0 conn(0x7fb77c0cd560 legacy=0x7fb77c0ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.520+0000 7fb780ac6640 1 -- 172.21.15.7:0/3213905170 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c15a610 con 0x7fb77c1540d0 2023-11-28T18:23:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb780ac6640 1 --2- 172.21.15.7:0/3213905170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c1540d0 0x7fb77c072940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb77c15a800 tx=0x7fb77001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87c26e07bcb7c88 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb77b7fe640 1 -- 172.21.15.7:0/3213905170 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700ab5f0 con 0x7fb77c1540d0 2023-11-28T18:23:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb77b7fe640 1 -- 172.21.15.7:0/3213905170 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7700ab750 con 0x7fb77c1540d0 2023-11-28T18:23:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb77b7fe640 1 -- 172.21.15.7:0/3213905170 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700abae0 con 0x7fb77c1540d0 2023-11-28T18:23:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 -- 172.21.15.7:0/3213905170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c1540d0 msgr2=0x7fb77c072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 --2- 172.21.15.7:0/3213905170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c1540d0 0x7fb77c072940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb77c15a800 tx=0x7fb77001c500 comp rx=0 tx=0).stop 2023-11-28T18:23:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 -- 172.21.15.7:0/3213905170 shutdown_connections 2023-11-28T18:23:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 --2- 172.21.15.7:0/3213905170 >> v2:172.21.15.193:3300/0 conn(0x7fb77c15db70 0x7fb77c15df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 --2- 172.21.15.7:0/3213905170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c1540d0 0x7fb77c072940 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 -- 172.21.15.7:0/3213905170 >> 172.21.15.7:0/3213905170 conn(0x7fb77c0662e0 msgr2=0x7fb77c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 -- 172.21.15.7:0/3213905170 shutdown_connections 2023-11-28T18:23:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 -- 172.21.15.7:0/3213905170 wait complete. 2023-11-28T18:23:09.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 Processor -- start 2023-11-28T18:23:09.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.524+0000 7fb782d2a640 1 -- start start 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c0cd560 0x7fb77c1f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb77c1540d0 0x7fb77c1497c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb77c15db70 0x7fb77c142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c149d00 con 0x7fb77c0cd560 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c148bf0 con 0x7fb77c1540d0 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c149330 con 0x7fb77c15db70 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c0cd560 0x7fb77c1f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c0cd560 0x7fb77c1f60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48176/0 (socket says 172.21.15.7:48176) 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 -- 172.21.15.7:0/4088075571 learned_addr learned my addr 172.21.15.7:0/4088075571 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb77bfff640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb77c1540d0 0x7fb77c1497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb7812c7640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb77c15db70 0x7fb77c142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 -- 172.21.15.7:0/4088075571 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb77c15db70 msgr2=0x7fb77c142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb77c15db70 0x7fb77c142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 -- 172.21.15.7:0/4088075571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb77c1540d0 msgr2=0x7fb77c1497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb77c1540d0 0x7fb77c1497c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 -- 172.21.15.7:0/4088075571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7700094d0 con 0x7fb77c0cd560 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb780ac6640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c0cd560 0x7fb77c1f60a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb77001c9e0 tx=0x7fb77001ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb7797fa640 1 -- 172.21.15.7:0/4088075571 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700be5f0 con 0x7fb77c0cd560 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb7797fa640 1 -- 172.21.15.7:0/4088075571 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7700be770 con 0x7fb77c0cd560 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77c14a0f0 con 0x7fb77c0cd560 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb7797fa640 1 -- 172.21.15.7:0/4088075571 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700beb20 con 0x7fb77c0cd560 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77c1403c0 con 0x7fb77c0cd560 2023-11-28T18:23:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.528+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77c04e890 con 0x7fb77c0cd560 2023-11-28T18:23:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.532+0000 7fb7797fa640 1 -- 172.21.15.7:0/4088075571 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb7700b43f0 con 0x7fb77c0cd560 2023-11-28T18:23:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.532+0000 7fb7797fa640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7600668d0 0x7fb760068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.532+0000 7fb7797fa640 1 -- 172.21.15.7:0/4088075571 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7700a8050 con 0x7fb77c0cd560 2023-11-28T18:23:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.532+0000 7fb77bfff640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7600668d0 0x7fb760068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.532+0000 7fb77bfff640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7600668d0 0x7fb760068d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb77c147260 tx=0x7fb768009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:09.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.536+0000 7fb7797fa640 1 -- 172.21.15.7:0/4088075571 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb7700ec170 con 0x7fb77c0cd560 2023-11-28T18:23:09.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:09.728+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb77c14e160 con 0x7fb77c0cd560 2023-11-28T18:23:10.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.668+0000 7fb7797fa640 1 -- 172.21.15.7:0/4088075571 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb7700ef790 con 0x7fb77c0cd560 2023-11-28T18:23:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7600668d0 msgr2=0x7fb760068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7600668d0 0x7fb760068d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb77c147260 tx=0x7fb768009190 comp rx=0 tx=0).stop 2023-11-28T18:23:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c0cd560 msgr2=0x7fb77c1f60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c0cd560 0x7fb77c1f60a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb77001c9e0 tx=0x7fb77001ad40 comp rx=0 tx=0).stop 2023-11-28T18:23:10.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 shutdown_connections 2023-11-28T18:23:10.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7600668d0 0x7fb760068d90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:10.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb77c15db70 0x7fb77c142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:10.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb77c1540d0 0x7fb77c1497c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:10.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 --2- 172.21.15.7:0/4088075571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb77c0cd560 0x7fb77c1f60a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:10.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 >> 172.21.15.7:0/4088075571 conn(0x7fb77c0662e0 msgr2=0x7fb77c0cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:10.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 shutdown_connections 2023-11-28T18:23:10.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.672+0000 7fb782d2a640 1 -- 172.21.15.7:0/4088075571 wait complete. 2023-11-28T18:23:10.701 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2023-11-28T18:23:10.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-11-28T18:23:10.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19fa0d6640 1 Processor -- start 2023-11-28T18:23:10.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19fa0d6640 1 -- start start 2023-11-28T18:23:10.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19fa0d6640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f19f4154130 0x7f19f4159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:10.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19fa0d6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f40caeb0 con 0x7f19f40ca530 2023-11-28T18:23:10.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19fa0d6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f415d9b0 con 0x7f19f40c4c80 2023-11-28T18:23:10.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19fa0d6640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f415a3f0 con 0x7f19f4154130 2023-11-28T18:23:10.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19f3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f19f40ca530 0x7f19f40cb2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33892/0 (socket says 172.21.15.7:33892) 2023-11-28T18:23:10.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19f3fff640 1 -- 172.21.15.7:0/2548460538 learned_addr learned my addr 172.21.15.7:0/2548460538 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:10.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.848+0000 7f19f37fe640 1 --2- 172.21.15.7:0/2548460538 >> v2:172.21.15.193:3300/0 conn(0x7f19f4154130 0x7f19f4159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:10.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f27fc640 1 -- 172.21.15.7:0/2548460538 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474975656 0 0) 0x7f19f40caeb0 con 0x7f19f40ca530 2023-11-28T18:23:10.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f27fc640 1 -- 172.21.15.7:0/2548460538 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19dc003530 con 0x7f19f40ca530 2023-11-28T18:23:10.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f37fe640 1 -- 172.21.15.7:0/2548460538 >> v1:172.21.15.193:6789/0 conn(0x7f19f40c4c80 legacy=0x7f19f415de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:10.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f37fe640 1 -- 172.21.15.7:0/2548460538 >> v1:172.21.15.7:6789/0 conn(0x7f19f40ca530 legacy=0x7f19f40cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:10.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f37fe640 1 -- 172.21.15.7:0/2548460538 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f415a5d0 con 0x7f19f4154130 2023-11-28T18:23:10.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f37fe640 1 --2- 172.21.15.7:0/2548460538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f4154130 0x7f19f4159eb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f19e40097c0 tx=0x7f19e401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6ba369b9bc92aaa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f27fc640 1 -- 172.21.15.7:0/2548460538 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e40a8970 con 0x7f19f4154130 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f27fc640 1 -- 172.21.15.7:0/2548460538 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e40a8ad0 con 0x7f19f4154130 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.852+0000 7f19f27fc640 1 -- 172.21.15.7:0/2548460538 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e40a8e60 con 0x7f19f4154130 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- 172.21.15.7:0/2548460538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f4154130 msgr2=0x7f19f4159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 --2- 172.21.15.7:0/2548460538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f4154130 0x7f19f4159eb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f19e40097c0 tx=0x7f19e401ec70 comp rx=0 tx=0).stop 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- 172.21.15.7:0/2548460538 shutdown_connections 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 --2- 172.21.15.7:0/2548460538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f4154130 0x7f19f4159eb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- 172.21.15.7:0/2548460538 >> 172.21.15.7:0/2548460538 conn(0x7f19f4073ca0 msgr2=0x7f19f4071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- 172.21.15.7:0/2548460538 shutdown_connections 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- 172.21.15.7:0/2548460538 wait complete. 2023-11-28T18:23:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 Processor -- start 2023-11-28T18:23:10.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- start start 2023-11-28T18:23:10.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f19f40c4c80 0x7f19f41f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:10.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f40ca530 0x7f19f41f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19f4154130 0x7f19f415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f414d580 con 0x7f19f4154130 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f41f2910 con 0x7f19f40ca530 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19fa0d6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f41f2bf0 con 0x7f19f40c4c80 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19f4154130 0x7f19f415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19f4154130 0x7f19f415c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48192/0 (socket says 172.21.15.7:48192) 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f3fff640 1 -- 172.21.15.7:0/3398903744 learned_addr learned my addr 172.21.15.7:0/3398903744 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f37fe640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f19f40c4c80 0x7f19f41f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f2ffd640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f40ca530 0x7f19f41f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f3fff640 1 -- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f19f40c4c80 msgr2=0x7f19f41f20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f3fff640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f19f40c4c80 0x7f19f41f20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f3fff640 1 -- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f40ca530 msgr2=0x7f19f41f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f3fff640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f40ca530 0x7f19f41f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f3fff640 1 -- 172.21.15.7:0/3398903744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4009420 con 0x7f19f4154130 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.856+0000 7f19f2ffd640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f40ca530 0x7f19f41f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19f3fff640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19f4154130 0x7f19f415c7e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00cb70 tx=0x7f19ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19dbfff640 1 -- 172.21.15.7:0/3398903744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec00ad40 con 0x7f19f4154130 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19dbfff640 1 -- 172.21.15.7:0/3398903744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec012400 con 0x7f19f4154130 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f41f0810 con 0x7f19f4154130 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19dbfff640 1 -- 172.21.15.7:0/3398903744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec00aea0 con 0x7f19f4154130 2023-11-28T18:23:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f41efb40 con 0x7f19f4154130 2023-11-28T18:23:10.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f40678b0 con 0x7f19f4154130 2023-11-28T18:23:10.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19dbfff640 1 -- 172.21.15.7:0/3398903744 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f19ec007070 con 0x7f19f4154130 2023-11-28T18:23:10.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19dbfff640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19e0066900 0x7f19e0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:10.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19dbfff640 1 -- 172.21.15.7:0/3398903744 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19ec0844f0 con 0x7f19f4154130 2023-11-28T18:23:10.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.860+0000 7f19f37fe640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19e0066900 0x7f19e0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.864+0000 7f19f37fe640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19e0066900 0x7f19e0068dc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f19e40b2000 tx=0x7f19e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:10.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:10.864+0000 7f19dbfff640 1 -- 172.21.15.7:0/3398903744 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f19ec051690 con 0x7f19f4154130 2023-11-28T18:23:11.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.060+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f19f404e890 con 0x7f19f4154130 2023-11-28T18:23:11.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19dbfff640 1 -- 172.21.15.7:0/3398903744 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f19ec054cb0 con 0x7f19f4154130 2023-11-28T18:23:11.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19e0066900 msgr2=0x7f19e0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19e0066900 0x7f19e0068dc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f19e40b2000 tx=0x7f19e4008040 comp rx=0 tx=0).stop 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19f4154130 msgr2=0x7f19f415c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19f4154130 0x7f19f415c7e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00cb70 tx=0x7f19ec00a430 comp rx=0 tx=0).stop 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 shutdown_connections 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19e0066900 0x7f19e0068dc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19f4154130 0x7f19f415c7e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f40ca530 0x7f19f41f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 --2- 172.21.15.7:0/3398903744 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f19f40c4c80 0x7f19f41f20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 >> 172.21.15.7:0/3398903744 conn(0x7f19f4073ca0 msgr2=0x7f19f40c5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 shutdown_connections 2023-11-28T18:23:11.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.680+0000 7f19fa0d6640 1 -- 172.21.15.7:0/3398903744 wait complete. 2023-11-28T18:23:11.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T18:23:11.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f692557d640 1 Processor -- start 2023-11-28T18:23:11.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f692557d640 1 -- start start 2023-11-28T18:23:11.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f692557d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6920154150 0x7f69200cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:11.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f692557d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69200b27d0 0x7f69200cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:11.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f692557d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692015ab70 con 0x7f69200cb300 2023-11-28T18:23:11.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f692557d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69200d00e0 con 0x7f69200b27d0 2023-11-28T18:23:11.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f692557d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6920159f60 con 0x7f6920154150 2023-11-28T18:23:11.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f6924d7c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69200b27d0 0x7f69200cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:11.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f6924d7c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69200b27d0 0x7f69200cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48200/0 (socket says 172.21.15.7:48200) 2023-11-28T18:23:11.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f6924d7c640 1 -- 172.21.15.7:0/3797495568 learned_addr learned my addr 172.21.15.7:0/3797495568 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:11.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f691ffff640 1 --2- 172.21.15.7:0/3797495568 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6920154150 0x7f69200cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:11.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f6924d7c640 1 -- 172.21.15.7:0/3797495568 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6920154150 msgr2=0x7f69200cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:11.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f6924d7c640 1 --2- 172.21.15.7:0/3797495568 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6920154150 0x7f69200cf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f6924d7c640 1 -- 172.21.15.7:0/3797495568 >> v1:172.21.15.193:6789/0 conn(0x7f69200cb300 legacy=0x7f69200b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:11.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.856+0000 7f6924d7c640 1 -- 172.21.15.7:0/3797495568 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692015a240 con 0x7f69200b27d0 2023-11-28T18:23:11.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f6924d7c640 1 --2- 172.21.15.7:0/3797495568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200b27d0 0x7f69200cfb00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6918009e50 tx=0x7f691801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851124753c9ebcdb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:11.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f691effd640 1 -- 172.21.15.7:0/3797495568 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180a57f0 con 0x7f69200b27d0 2023-11-28T18:23:11.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f691effd640 1 -- 172.21.15.7:0/3797495568 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69180a5950 con 0x7f69200b27d0 2023-11-28T18:23:11.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f691effd640 1 -- 172.21.15.7:0/3797495568 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180a5ce0 con 0x7f69200b27d0 2023-11-28T18:23:11.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f692557d640 1 -- 172.21.15.7:0/3797495568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200b27d0 msgr2=0x7f69200cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:11.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f692557d640 1 --2- 172.21.15.7:0/3797495568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200b27d0 0x7f69200cfb00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6918009e50 tx=0x7f691801ca10 comp rx=0 tx=0).stop 2023-11-28T18:23:11.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f692557d640 1 -- 172.21.15.7:0/3797495568 shutdown_connections 2023-11-28T18:23:11.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f692557d640 1 --2- 172.21.15.7:0/3797495568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200b27d0 0x7f69200cfb00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f692557d640 1 --2- 172.21.15.7:0/3797495568 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6920154150 0x7f69200cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f692557d640 1 -- 172.21.15.7:0/3797495568 >> 172.21.15.7:0/3797495568 conn(0x7f69200662e0 msgr2=0x7f69200666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:11.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f692557d640 1 -- 172.21.15.7:0/3797495568 shutdown_connections 2023-11-28T18:23:11.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.860+0000 7f692557d640 1 -- 172.21.15.7:0/3797495568 wait complete. 2023-11-28T18:23:11.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 Processor -- start 2023-11-28T18:23:11.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 -- start start 2023-11-28T18:23:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200cb300 0x7f69200ab770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6920154150 0x7f69200cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69200aca70 0x7f69201f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69201f6180 con 0x7f69200cb300 2023-11-28T18:23:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69200c7e30 con 0x7f6920154150 2023-11-28T18:23:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69200c8110 con 0x7f69200aca70 2023-11-28T18:23:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200cb300 0x7f69200ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:11.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200cb300 0x7f69200ab770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48208/0 (socket says 172.21.15.7:48208) 2023-11-28T18:23:11.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 -- 172.21.15.7:0/1892345784 learned_addr learned my addr 172.21.15.7:0/1892345784 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:11.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691f7fe640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6920154150 0x7f69200cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:11.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f6924d7c640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69200aca70 0x7f69201f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:11.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 -- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69200aca70 msgr2=0x7f69201f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:11.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69200aca70 0x7f69201f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 -- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6920154150 msgr2=0x7f69200cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:11.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6920154150 0x7f69200cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 -- 172.21.15.7:0/1892345784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6918009b30 con 0x7f69200cb300 2023-11-28T18:23:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691f7fe640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6920154150 0x7f69200cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691ffff640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200cb300 0x7f69200ab770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f691000e870 tx=0x7f691000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691cff9640 1 -- 172.21.15.7:0/1892345784 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910009510 con 0x7f69200cb300 2023-11-28T18:23:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691cff9640 1 -- 172.21.15.7:0/1892345784 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691000eec0 con 0x7f69200cb300 2023-11-28T18:23:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f691cff9640 1 -- 172.21.15.7:0/1892345784 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691000cb40 con 0x7f69200cb300 2023-11-28T18:23:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69201f63d0 con 0x7f69200cb300 2023-11-28T18:23:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.864+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69201f6990 con 0x7f69200cb300 2023-11-28T18:23:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.868+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692004e890 con 0x7f69200cb300 2023-11-28T18:23:11.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.868+0000 7f691cff9640 1 -- 172.21.15.7:0/1892345784 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f691000cca0 con 0x7f69200cb300 2023-11-28T18:23:11.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.868+0000 7f691cff9640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f690c066900 0x7f690c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:11.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.868+0000 7f691cff9640 1 -- 172.21.15.7:0/1892345784 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6910084420 con 0x7f69200cb300 2023-11-28T18:23:11.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.868+0000 7f691f7fe640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f690c066900 0x7f690c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:11.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.868+0000 7f691f7fe640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f690c066900 0x7f690c068dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f692014f4e0 tx=0x7f6914009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:11.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:11.872+0000 7f691cff9640 1 -- 172.21.15.7:0/1892345784 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6910051700 con 0x7f69200cb300 2023-11-28T18:23:12.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.068+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f69200caea0 con 0x7f69200cb300 2023-11-28T18:23:12.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.680+0000 7f691cff9640 1 -- 172.21.15.7:0/1892345784 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6910054d20 con 0x7f69200cb300 2023-11-28T18:23:12.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f690c066900 msgr2=0x7f690c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f690c066900 0x7f690c068dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f692014f4e0 tx=0x7f6914009040 comp rx=0 tx=0).stop 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200cb300 msgr2=0x7f69200ab770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200cb300 0x7f69200ab770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f691000e870 tx=0x7f691000ec70 comp rx=0 tx=0).stop 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 shutdown_connections 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f690c066900 0x7f690c068dc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69200aca70 0x7f69201f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6920154150 0x7f69200cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 --2- 172.21.15.7:0/1892345784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69200cb300 0x7f69200ab770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 >> 172.21.15.7:0/1892345784 conn(0x7f69200662e0 msgr2=0x7f69200cf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 shutdown_connections 2023-11-28T18:23:12.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.684+0000 7f692557d640 1 -- 172.21.15.7:0/1892345784 wait complete. 2023-11-28T18:23:12.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-11-28T18:23:12.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6ad80640 1 Processor -- start 2023-11-28T18:23:12.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6ad80640 1 -- start start 2023-11-28T18:23:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6ad80640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2d6415a5c0 0x7f2d640c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6ad80640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2d640c42b0 0x7f2d6415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6ad80640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d640c8530 con 0x7f2d6415a9a0 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6ad80640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d640c7680 con 0x7f2d640c42b0 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6ad80640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d640c7920 con 0x7f2d6415a5c0 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6931d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2d640c42b0 0x7f2d6415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6931d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2d640c42b0 0x7f2d6415adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48212/0 (socket says 172.21.15.7:48212) 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6931d640 1 -- 172.21.15.7:0/1065723549 learned_addr learned my addr 172.21.15.7:0/1065723549 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d68b1c640 1 --2- 172.21.15.7:0/1065723549 >> v2:172.21.15.193:3300/0 conn(0x7f2d6415a5c0 0x7f2d640c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6931d640 1 -- 172.21.15.7:0/1065723549 >> v2:172.21.15.193:3300/0 conn(0x7f2d6415a5c0 msgr2=0x7f2d640c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6931d640 1 --2- 172.21.15.7:0/1065723549 >> v2:172.21.15.193:3300/0 conn(0x7f2d6415a5c0 0x7f2d640c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6931d640 1 -- 172.21.15.7:0/1065723549 >> v1:172.21.15.7:6789/0 conn(0x7f2d6415a9a0 legacy=0x7f2d640c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d6931d640 1 -- 172.21.15.7:0/1065723549 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d640c7c00 con 0x7f2d640c42b0 2023-11-28T18:23:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.860+0000 7f2d68b1c640 1 --2- 172.21.15.7:0/1065723549 >> v2:172.21.15.193:3300/0 conn(0x7f2d6415a5c0 0x7f2d640c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:12.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6931d640 1 --2- 172.21.15.7:0/1065723549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d640c42b0 0x7f2d6415adc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2d640c8fd0 tx=0x7f2d5c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9779c0331a23db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:12.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d637fe640 1 -- 172.21.15.7:0/1065723549 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0af400 con 0x7f2d640c42b0 2023-11-28T18:23:12.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d637fe640 1 -- 172.21.15.7:0/1065723549 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5c0ab050 con 0x7f2d640c42b0 2023-11-28T18:23:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d637fe640 1 -- 172.21.15.7:0/1065723549 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0af780 con 0x7f2d640c42b0 2023-11-28T18:23:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1065723549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d640c42b0 msgr2=0x7f2d6415adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1065723549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d640c42b0 0x7f2d6415adc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2d640c8fd0 tx=0x7f2d5c01f1c0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1065723549 shutdown_connections 2023-11-28T18:23:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1065723549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d640c42b0 0x7f2d6415adc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1065723549 >> v2:172.21.15.193:3300/0 conn(0x7f2d6415a5c0 0x7f2d640c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1065723549 >> 172.21.15.7:0/1065723549 conn(0x7f2d64073f00 msgr2=0x7f2d640662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1065723549 shutdown_connections 2023-11-28T18:23:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.864+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1065723549 wait complete. 2023-11-28T18:23:12.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 Processor -- start 2023-11-28T18:23:12.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 -- start start 2023-11-28T18:23:12.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d640c42b0 0x7f2d641ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:12.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d6415a5c0 0x7f2d6413ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6415a9a0 0x7f2d641404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6414eb20 con 0x7f2d6415a9a0 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d64141e10 con 0x7f2d640c42b0 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d641f3a80 con 0x7f2d6415a5c0 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6415a9a0 0x7f2d641404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6415a9a0 0x7f2d641404b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48226/0 (socket says 172.21.15.7:48226) 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 -- 172.21.15.7:0/1823684204 learned_addr learned my addr 172.21.15.7:0/1823684204 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d63fff640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d6415a5c0 0x7f2d6413ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d68b1c640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d640c42b0 0x7f2d641ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:12.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 -- 172.21.15.7:0/1823684204 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d6415a5c0 msgr2=0x7f2d6413ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:12.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d6415a5c0 0x7f2d6413ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 -- 172.21.15.7:0/1823684204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d640c42b0 msgr2=0x7f2d641ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:12.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d640c42b0 0x7f2d641ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:12.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 -- 172.21.15.7:0/1823684204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c009b30 con 0x7f2d6415a9a0 2023-11-28T18:23:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6931d640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6415a9a0 0x7f2d641404b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0b2040 tx=0x7f2d5c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d617fa640 1 -- 172.21.15.7:0/1823684204 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0c0070 con 0x7f2d6415a9a0 2023-11-28T18:23:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d617fa640 1 -- 172.21.15.7:0/1823684204 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5c0afc00 con 0x7f2d6415a9a0 2023-11-28T18:23:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d617fa640 1 -- 172.21.15.7:0/1823684204 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0b8400 con 0x7f2d6415a9a0 2023-11-28T18:23:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d64141b30 con 0x7f2d6415a9a0 2023-11-28T18:23:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.868+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d641f2fa0 con 0x7f2d6415a9a0 2023-11-28T18:23:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.872+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6404e890 con 0x7f2d6415a9a0 2023-11-28T18:23:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.872+0000 7f2d617fa640 1 -- 172.21.15.7:0/1823684204 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2d5c0b6040 con 0x7f2d6415a9a0 2023-11-28T18:23:12.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.872+0000 7f2d617fa640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2d48066900 0x7f2d48068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:12.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.872+0000 7f2d617fa640 1 -- 172.21.15.7:0/1823684204 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d5c121bc0 con 0x7f2d6415a9a0 2023-11-28T18:23:12.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.872+0000 7f2d68b1c640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2d48066900 0x7f2d48068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:12.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.872+0000 7f2d68b1c640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2d48066900 0x7f2d48068dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800ab60 tx=0x7f2d58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:12.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:12.876+0000 7f2d617fa640 1 -- 172.21.15.7:0/1823684204 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2d5c0ef090 con 0x7f2d6415a9a0 2023-11-28T18:23:13.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.072+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f2d640c3e00 con 0x7f2d6415a9a0 2023-11-28T18:23:13.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.692+0000 7f2d617fa640 1 -- 172.21.15.7:0/1823684204 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2d5c0f26b0 con 0x7f2d6415a9a0 2023-11-28T18:23:13.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2d48066900 msgr2=0x7f2d48068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:13.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2d48066900 0x7f2d48068dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800ab60 tx=0x7f2d58008040 comp rx=0 tx=0).stop 2023-11-28T18:23:13.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6415a9a0 msgr2=0x7f2d641404b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6415a9a0 0x7f2d641404b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0b2040 tx=0x7f2d5c01bf60 comp rx=0 tx=0).stop 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 shutdown_connections 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2d48066900 0x7f2d48068dc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6415a9a0 0x7f2d641404b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d6415a5c0 0x7f2d6413ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 --2- 172.21.15.7:0/1823684204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d640c42b0 0x7f2d641ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 >> 172.21.15.7:0/1823684204 conn(0x7f2d64073f00 msgr2=0x7f2d640c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 shutdown_connections 2023-11-28T18:23:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.696+0000 7f2d6ad80640 1 -- 172.21.15.7:0/1823684204 wait complete. 2023-11-28T18:23:13.722 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T18:23:13.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:13.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a8b93640 1 Processor -- start 2023-11-28T18:23:13.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a8b93640 1 -- start start 2023-11-28T18:23:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a8b93640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f41a40cafe0 0x7f41a40cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a8b93640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a415a3a0 con 0x7f41a415ca30 2023-11-28T18:23:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a8b93640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a415a620 con 0x7f41a415ce10 2023-11-28T18:23:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a8b93640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a415a900 con 0x7f41a40cafe0 2023-11-28T18:23:13.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f41a40cafe0 0x7f41a40cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:13.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f41a40cafe0 0x7f41a40cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59272/0 (socket says 172.21.15.7:59272) 2023-11-28T18:23:13.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a3fff640 1 -- 172.21.15.7:0/822042590 learned_addr learned my addr 172.21.15.7:0/822042590 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:13.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a3fff640 1 -- 172.21.15.7:0/822042590 >> v1:172.21.15.193:6789/0 conn(0x7f41a415ce10 legacy=0x7f41a40ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:13.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a3fff640 1 -- 172.21.15.7:0/822042590 >> v1:172.21.15.7:6789/0 conn(0x7f41a415ca30 legacy=0x7f41a4159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:13.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.864+0000 7f41a3fff640 1 -- 172.21.15.7:0/822042590 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a415abe0 con 0x7f41a40cafe0 2023-11-28T18:23:13.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a3fff640 1 --2- 172.21.15.7:0/822042590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40cc5f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f419c009e50 tx=0x7f419c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee873dc1a7b5a53d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:13.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a27fc640 1 -- 172.21.15.7:0/822042590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c0a5770 con 0x7f41a40cafe0 2023-11-28T18:23:13.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a27fc640 1 -- 172.21.15.7:0/822042590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c0a58d0 con 0x7f41a40cafe0 2023-11-28T18:23:13.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a27fc640 1 -- 172.21.15.7:0/822042590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c0a5c60 con 0x7f41a40cafe0 2023-11-28T18:23:13.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 -- 172.21.15.7:0/822042590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 msgr2=0x7f41a40cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:13.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 --2- 172.21.15.7:0/822042590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40cc5f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f419c009e50 tx=0x7f419c01c9a0 comp rx=0 tx=0).stop 2023-11-28T18:23:13.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 -- 172.21.15.7:0/822042590 shutdown_connections 2023-11-28T18:23:13.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 --2- 172.21.15.7:0/822042590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40cc5f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:13.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 -- 172.21.15.7:0/822042590 >> 172.21.15.7:0/822042590 conn(0x7f41a406c060 msgr2=0x7f41a40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:13.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 -- 172.21.15.7:0/822042590 shutdown_connections 2023-11-28T18:23:13.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 -- 172.21.15.7:0/822042590 wait complete. 2023-11-28T18:23:13.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 Processor -- start 2023-11-28T18:23:13.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.868+0000 7f41a8b93640 1 -- start start 2023-11-28T18:23:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f41a415ca30 0x7f41a40c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a415ce10 0x7f41a414b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a414b370 con 0x7f41a40cafe0 2023-11-28T18:23:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a414ce90 con 0x7f41a415ce10 2023-11-28T18:23:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a4148c90 con 0x7f41a415ca30 2023-11-28T18:23:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59280/0 (socket says 172.21.15.7:59280) 2023-11-28T18:23:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 -- 172.21.15.7:0/538380946 learned_addr learned my addr 172.21.15.7:0/538380946 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:13.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 -- 172.21.15.7:0/538380946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f41a415ca30 msgr2=0x7f41a40c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:13.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a2ffd640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f41a415ca30 0x7f41a40c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:13.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f41a415ca30 0x7f41a40c5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:13.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 -- 172.21.15.7:0/538380946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a415ce10 msgr2=0x7f41a414b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:13.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a415ce10 0x7f41a414b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:13.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 -- 172.21.15.7:0/538380946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c009b30 con 0x7f41a40cafe0 2023-11-28T18:23:13.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a2ffd640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f41a415ca30 0x7f41a40c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:13.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a37fe640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40c90b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f419400c500 tx=0x7f419400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:13.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f418bfff640 1 -- 172.21.15.7:0/538380946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194005220 con 0x7f41a40cafe0 2023-11-28T18:23:13.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f418bfff640 1 -- 172.21.15.7:0/538380946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4194005380 con 0x7f41a40cafe0 2023-11-28T18:23:13.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a4148f30 con 0x7f41a40cafe0 2023-11-28T18:23:13.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f418bfff640 1 -- 172.21.15.7:0/538380946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194007270 con 0x7f41a40cafe0 2023-11-28T18:23:13.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a414d0e0 con 0x7f41a40cafe0 2023-11-28T18:23:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.872+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a404e890 con 0x7f41a40cafe0 2023-11-28T18:23:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.876+0000 7f418bfff640 1 -- 172.21.15.7:0/538380946 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f41940073d0 con 0x7f41a40cafe0 2023-11-28T18:23:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.876+0000 7f418bfff640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f41900668d0 0x7f4190068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.876+0000 7f418bfff640 1 -- 172.21.15.7:0/538380946 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4194083d50 con 0x7f41a40cafe0 2023-11-28T18:23:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.876+0000 7f41a2ffd640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f41900668d0 0x7f4190068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.876+0000 7f41a2ffd640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f41900668d0 0x7f4190068d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41a4066630 tx=0x7f4198009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:13.880+0000 7f418bfff640 1 -- 172.21.15.7:0/538380946 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4194087030 con 0x7f41a40cafe0 2023-11-28T18:23:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41a4149bc0 con 0x7f41a40cafe0 2023-11-28T18:23:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f418bfff640 1 -- 172.21.15.7:0/538380946 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6504 (secure 0 0 0) 0x7f4194050fa0 con 0x7f41a40cafe0 2023-11-28T18:23:14.073 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:14.073 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:23:13.700126+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4291,"mds_1":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:23:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 9 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f41900668d0 msgr2=0x7f4190068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f41900668d0 0x7f4190068d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41a4066630 tx=0x7f4198009040 comp rx=0 tx=0).stop 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 msgr2=0x7f41a40c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40c90b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f419400c500 tx=0x7f419400c9c0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 shutdown_connections 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f41900668d0 0x7f4190068d90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a415ce10 0x7f41a414b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f41a415ca30 0x7f41a40c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 --2- 172.21.15.7:0/538380946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41a40cafe0 0x7f41a40c90b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.068+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 >> 172.21.15.7:0/538380946 conn(0x7f41a406c060 msgr2=0x7f41a415b210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.072+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 shutdown_connections 2023-11-28T18:23:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.072+0000 7f41a8b93640 1 -- 172.21.15.7:0/538380946 wait complete. 2023-11-28T18:23:14.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.232+0000 7f996791d640 1 Processor -- start 2023-11-28T18:23:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.232+0000 7f996791d640 1 -- start start 2023-11-28T18:23:14.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.232+0000 7f996791d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f9960072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.232+0000 7f996791d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f996015d350 0x7f996006a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f996791d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600c8750 con 0x7f996006ab30 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f996791d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99600c8a30 con 0x7f996015d350 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f996791d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996015d840 con 0x7f99601540a0 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9964eb8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f996015d350 0x7f996006a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9964eb8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f996015d350 0x7f996006a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59294/0 (socket says 172.21.15.7:59294) 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9964eb8640 1 -- 172.21.15.7:0/3630744493 learned_addr learned my addr 172.21.15.7:0/3630744493 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f99656b9640 1 --2- 172.21.15.7:0/3630744493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f9960072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9964eb8640 1 -- 172.21.15.7:0/3630744493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 msgr2=0x7f9960072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9964eb8640 1 --2- 172.21.15.7:0/3630744493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f9960072ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9964eb8640 1 -- 172.21.15.7:0/3630744493 >> v1:172.21.15.193:6789/0 conn(0x7f996006ab30 legacy=0x7f99600c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9964eb8640 1 -- 172.21.15.7:0/3630744493 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996015db20 con 0x7f996015d350 2023-11-28T18:23:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f99656b9640 1 --2- 172.21.15.7:0/3630744493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f9960072ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9964eb8640 1 --2- 172.21.15.7:0/3630744493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996015d350 0x7f996006a5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f99500098a0 tx=0x7f995001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6658f35d1a40e2d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9957fff640 1 -- 172.21.15.7:0/3630744493 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99500ab600 con 0x7f996015d350 2023-11-28T18:23:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9957fff640 1 -- 172.21.15.7:0/3630744493 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99500a7050 con 0x7f996015d350 2023-11-28T18:23:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.236+0000 7f9957fff640 1 -- 172.21.15.7:0/3630744493 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99500ab980 con 0x7f996015d350 2023-11-28T18:23:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- 172.21.15.7:0/3630744493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996015d350 msgr2=0x7f996006a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 --2- 172.21.15.7:0/3630744493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996015d350 0x7f996006a5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f99500098a0 tx=0x7f995001c520 comp rx=0 tx=0).stop 2023-11-28T18:23:14.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- 172.21.15.7:0/3630744493 shutdown_connections 2023-11-28T18:23:14.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 --2- 172.21.15.7:0/3630744493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996015d350 0x7f996006a5f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 --2- 172.21.15.7:0/3630744493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f9960072ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- 172.21.15.7:0/3630744493 >> 172.21.15.7:0/3630744493 conn(0x7f9960073cd0 msgr2=0x7f99600662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:14.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- 172.21.15.7:0/3630744493 shutdown_connections 2023-11-28T18:23:14.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- 172.21.15.7:0/3630744493 wait complete. 2023-11-28T18:23:14.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 Processor -- start 2023-11-28T18:23:14.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- start start 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996006ab30 0x7f996015fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f99600c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996015d350 0x7f9960150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600c9a90 con 0x7f996006ab30 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600c8ec0 con 0x7f996015d350 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f996791d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99600cafd0 con 0x7f99601540a0 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f99656b9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996006ab30 0x7f996015fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f99656b9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996006ab30 0x7f996015fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59308/0 (socket says 172.21.15.7:59308) 2023-11-28T18:23:14.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f99656b9640 1 -- 172.21.15.7:0/277989650 learned_addr learned my addr 172.21.15.7:0/277989650 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f9964eb8640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f99600c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f9965eba640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996015d350 0x7f9960150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f99656b9640 1 -- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 msgr2=0x7f99600c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.240+0000 7f99656b9640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f99600c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f99656b9640 1 -- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996015d350 msgr2=0x7f9960150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f99656b9640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996015d350 0x7f9960150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f99656b9640 1 -- 172.21.15.7:0/277989650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500094d0 con 0x7f996006ab30 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9964eb8640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f99600c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9965eba640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996015d350 0x7f9960150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f99656b9640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996006ab30 0x7f996015fa60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f995c00e810 tx=0x7f995c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9955ffb640 1 -- 172.21.15.7:0/277989650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c00c8d0 con 0x7f996006ab30 2023-11-28T18:23:14.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9955ffb640 1 -- 172.21.15.7:0/277989650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995c00ca30 con 0x7f996006ab30 2023-11-28T18:23:14.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9955ffb640 1 -- 172.21.15.7:0/277989650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c00cdc0 con 0x7f996006ab30 2023-11-28T18:23:14.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99600c9160 con 0x7f996006ab30 2023-11-28T18:23:14.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99600c9720 con 0x7f996006ab30 2023-11-28T18:23:14.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996004e890 con 0x7f996006ab30 2023-11-28T18:23:14.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9955ffb640 1 -- 172.21.15.7:0/277989650 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f995c013410 con 0x7f996006ab30 2023-11-28T18:23:14.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9955ffb640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9944066900 0x7f9944068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9955ffb640 1 -- 172.21.15.7:0/277989650 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f995c084210 con 0x7f996006ab30 2023-11-28T18:23:14.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.244+0000 7f9964eb8640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9944066900 0x7f9944068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.248+0000 7f9964eb8640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9944066900 0x7f9944068dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f99500ad000 tx=0x7f99500a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.248+0000 7f9955ffb640 1 -- 172.21.15.7:0/277989650 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f995c051450 con 0x7f996006ab30 2023-11-28T18:23:14.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99600c4900 con 0x7f996006ab30 2023-11-28T18:23:14.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f9955ffb640 1 -- 172.21.15.7:0/277989650 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f995c009480 con 0x7f996006ab30 2023-11-28T18:23:14.417 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:14.417 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:14.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9944066900 msgr2=0x7f9944068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9944066900 0x7f9944068dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f99500ad000 tx=0x7f99500a6000 comp rx=0 tx=0).stop 2023-11-28T18:23:14.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996006ab30 msgr2=0x7f996015fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996006ab30 0x7f996015fa60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f995c00e810 tx=0x7f995c00ecd0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 shutdown_connections 2023-11-28T18:23:14.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9944066900 0x7f9944068dc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996015d350 0x7f9960150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99601540a0 0x7f99600c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 --2- 172.21.15.7:0/277989650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f996006ab30 0x7f996015fa60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 >> 172.21.15.7:0/277989650 conn(0x7f9960073cd0 msgr2=0x7f99600c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:14.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 shutdown_connections 2023-11-28T18:23:14.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.412+0000 7f996791d640 1 -- 172.21.15.7:0/277989650 wait complete. 2023-11-28T18:23:14.439 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T18:23:14.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T18:23:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.576+0000 7f624bf55640 1 Processor -- start 2023-11-28T18:23:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.576+0000 7f624bf55640 1 -- start start 2023-11-28T18:23:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624bf55640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624415a5b0 0x7f62440c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624bf55640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f624406a8a0 0x7f62440c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624bf55640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62440c8430 con 0x7f6244154040 2023-11-28T18:23:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624bf55640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f624406ae50 con 0x7f624406a8a0 2023-11-28T18:23:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624bf55640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624406b110 con 0x7f624415a5b0 2023-11-28T18:23:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f6249cf1640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6244154040 0x7f6244159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55138/0 (socket says 172.21.15.7:55138) 2023-11-28T18:23:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f6249cf1640 1 -- 172.21.15.7:0/2359129546 learned_addr learned my addr 172.21.15.7:0/2359129546 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f62494f0640 1 --2- 172.21.15.7:0/2359129546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624415a5b0 0x7f62440c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624a4f2640 1 --2- 172.21.15.7:0/2359129546 >> v2:172.21.15.193:3300/0 conn(0x7f624406a8a0 0x7f62440c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f6248cef640 1 -- 172.21.15.7:0/2359129546 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820345330 0 0) 0x7f62440c8430 con 0x7f6244154040 2023-11-28T18:23:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f6248cef640 1 -- 172.21.15.7:0/2359129546 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6230003530 con 0x7f6244154040 2023-11-28T18:23:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624a4f2640 1 -- 172.21.15.7:0/2359129546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624415a5b0 msgr2=0x7f62440c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624a4f2640 1 --2- 172.21.15.7:0/2359129546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624415a5b0 0x7f62440c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f6248cef640 1 -- 172.21.15.7:0/2359129546 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 202832439 0 0) 0x7f6230003530 con 0x7f6244154040 2023-11-28T18:23:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624a4f2640 1 -- 172.21.15.7:0/2359129546 >> v1:172.21.15.7:6789/0 conn(0x7f6244154040 legacy=0x7f6244159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.580+0000 7f624a4f2640 1 -- 172.21.15.7:0/2359129546 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62440c7820 con 0x7f624406a8a0 2023-11-28T18:23:14.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624a4f2640 1 --2- 172.21.15.7:0/2359129546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f624406a8a0 0x7f62440c8ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f623c009d90 tx=0x7f623c01f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3e6f7573159121d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f6248cef640 1 -- 172.21.15.7:0/2359129546 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6240001680 con 0x7f624406a8a0 2023-11-28T18:23:14.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f6248cef640 1 -- 172.21.15.7:0/2359129546 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c0a8bb0 con 0x7f624406a8a0 2023-11-28T18:23:14.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f6248cef640 1 -- 172.21.15.7:0/2359129546 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c0b0440 con 0x7f624406a8a0 2023-11-28T18:23:14.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 -- 172.21.15.7:0/2359129546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f624406a8a0 msgr2=0x7f62440c8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 --2- 172.21.15.7:0/2359129546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f624406a8a0 0x7f62440c8ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f623c009d90 tx=0x7f623c01f060 comp rx=0 tx=0).stop 2023-11-28T18:23:14.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 -- 172.21.15.7:0/2359129546 shutdown_connections 2023-11-28T18:23:14.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 --2- 172.21.15.7:0/2359129546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624415a5b0 0x7f62440c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 --2- 172.21.15.7:0/2359129546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f624406a8a0 0x7f62440c8ef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 -- 172.21.15.7:0/2359129546 >> 172.21.15.7:0/2359129546 conn(0x7f624406d4d0 msgr2=0x7f624406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:14.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 -- 172.21.15.7:0/2359129546 shutdown_connections 2023-11-28T18:23:14.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 -- 172.21.15.7:0/2359129546 wait complete. 2023-11-28T18:23:14.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 Processor -- start 2023-11-28T18:23:14.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.584+0000 7f624bf55640 1 -- start start 2023-11-28T18:23:14.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624406a8a0 0x7f624414bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6244154040 0x7f624406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624415a5b0 0x7f624414d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624414c5e0 con 0x7f624415a5b0 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62441ef990 con 0x7f6244154040 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624414ee90 con 0x7f624406a8a0 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624415a5b0 0x7f624414d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624415a5b0 0x7f624414d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59312/0 (socket says 172.21.15.7:59312) 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 -- 172.21.15.7:0/2467671834 learned_addr learned my addr 172.21.15.7:0/2467671834 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f6249cf1640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624406a8a0 0x7f624414bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f62494f0640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6244154040 0x7f624406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 -- 172.21.15.7:0/2467671834 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624406a8a0 msgr2=0x7f624414bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624406a8a0 0x7f624414bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 -- 172.21.15.7:0/2467671834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6244154040 msgr2=0x7f624406f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6244154040 0x7f624406f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 -- 172.21.15.7:0/2467671834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c009a70 con 0x7f624415a5b0 2023-11-28T18:23:14.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624a4f2640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624415a5b0 0x7f624414d090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f623c01d710 tx=0x7f623c01be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f623a7fc640 1 -- 172.21.15.7:0/2467671834 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c009450 con 0x7f624415a5b0 2023-11-28T18:23:14.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f623a7fc640 1 -- 172.21.15.7:0/2467671834 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c0ba410 con 0x7f624415a5b0 2023-11-28T18:23:14.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62441efc30 con 0x7f624415a5b0 2023-11-28T18:23:14.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f623a7fc640 1 -- 172.21.15.7:0/2467671834 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c0ba780 con 0x7f624415a5b0 2023-11-28T18:23:14.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62441f01f0 con 0x7f624415a5b0 2023-11-28T18:23:14.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f624404e890 con 0x7f624415a5b0 2023-11-28T18:23:14.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.588+0000 7f623a7fc640 1 -- 172.21.15.7:0/2467671834 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f623c0b7050 con 0x7f624415a5b0 2023-11-28T18:23:14.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.592+0000 7f623a7fc640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6228066900 0x7f6228068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.592+0000 7f623a7fc640 1 -- 172.21.15.7:0/2467671834 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f623c122350 con 0x7f624415a5b0 2023-11-28T18:23:14.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.592+0000 7f6249cf1640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6228066900 0x7f6228068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.592+0000 7f6249cf1640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6228066900 0x7f6228068dc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6240000fc0 tx=0x7f6240009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.596+0000 7f623a7fc640 1 -- 172.21.15.7:0/2467671834 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f623c0ef460 con 0x7f624415a5b0 2023-11-28T18:23:14.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.748+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 --> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62440c80c0 con 0x7f6228066900 2023-11-28T18:23:14.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.760+0000 7f623a7fc640 1 -- 172.21.15.7:0/2467671834 <== mgr.4102 v2:172.21.15.7:6832/264069339 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264530 (secure 0 0 0) 0x7f62400058e0 con 0x7f6228066900 2023-11-28T18:23:14.766 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-28T18:23:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6228066900 msgr2=0x7f6228068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6228066900 0x7f6228068dc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6240000fc0 tx=0x7f6240009040 comp rx=0 tx=0).stop 2023-11-28T18:23:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624415a5b0 msgr2=0x7f624414d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624415a5b0 0x7f624414d090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f623c01d710 tx=0x7f623c01be00 comp rx=0 tx=0).stop 2023-11-28T18:23:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 shutdown_connections 2023-11-28T18:23:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6228066900 0x7f6228068dc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624415a5b0 0x7f624414d090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6244154040 0x7f624406f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 --2- 172.21.15.7:0/2467671834 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f624406a8a0 0x7f624414bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 >> 172.21.15.7:0/2467671834 conn(0x7f624406d4d0 msgr2=0x7f62441587f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:14.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 shutdown_connections 2023-11-28T18:23:14.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.764+0000 7f624bf55640 1 -- 172.21.15.7:0/2467671834 wait complete. 2023-11-28T18:23:14.786 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-11-28T18:23:13.462317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235480,"kb_used_data":2664,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739240,"statfs":{"total":773094113280,"available":772852981760,"internally_reserved":0,"allocated":2727936,"data_stored":1697616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001423"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142633+0000","last_change":"2023-11-28T18:23:08.134633+0000","last_active":"2023-11-28T18:23:08.142633+0000","last_peered":"2023-11-28T18:23:08.142633+0000","last_clean":"2023-11-28T18:23:08.142633+0000","last_became_active":"2023-11-28T18:23:08.134320+0000","last_became_peered":"2023-11-28T18:23:08.134320+0000","last_unstale":"2023-11-28T18:23:08.142633+0000","last_undegraded":"2023-11-28T18:23:08.142633+0000","last_fullsized":"2023-11-28T18:23:08.142633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:07:32.049802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105857+0000","last_change":"2023-11-28T18:23:08.122460+0000","last_active":"2023-11-28T18:23:09.105857+0000","last_peered":"2023-11-28T18:23:09.105857+0000","last_clean":"2023-11-28T18:23:09.105857+0000","last_became_active":"2023-11-28T18:23:08.122153+0000","last_became_peered":"2023-11-28T18:23:08.122153+0000","last_unstale":"2023-11-28T18:23:09.105857+0000","last_undegraded":"2023-11-28T18:23:09.105857+0000","last_fullsized":"2023-11-28T18:23:09.105857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:17:31.074978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.113129+0000","last_change":"2023-11-28T18:23:08.110187+0000","last_active":"2023-11-28T18:23:08.113129+0000","last_peered":"2023-11-28T18:23:08.113129+0000","last_clean":"2023-11-28T18:23:08.113129+0000","last_became_active":"2023-11-28T18:23:08.109915+0000","last_became_peered":"2023-11-28T18:23:08.109915+0000","last_unstale":"2023-11-28T18:23:08.113129+0000","last_undegraded":"2023-11-28T18:23:08.113129+0000","last_fullsized":"2023-11-28T18:23:08.113129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:20.219583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113335+0000","last_change":"2023-11-28T18:23:08.122000+0000","last_active":"2023-11-28T18:23:09.113335+0000","last_peered":"2023-11-28T18:23:09.113335+0000","last_clean":"2023-11-28T18:23:09.113335+0000","last_became_active":"2023-11-28T18:23:08.121652+0000","last_became_peered":"2023-11-28T18:23:08.121652+0000","last_unstale":"2023-11-28T18:23:09.113335+0000","last_undegraded":"2023-11-28T18:23:09.113335+0000","last_fullsized":"2023-11-28T18:23:09.113335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:15.540606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.134612+0000","last_change":"2023-11-28T18:23:08.126587+0000","last_active":"2023-11-28T18:23:08.134612+0000","last_peered":"2023-11-28T18:23:08.134612+0000","last_clean":"2023-11-28T18:23:08.134612+0000","last_became_active":"2023-11-28T18:23:08.126313+0000","last_became_peered":"2023-11-28T18:23:08.126313+0000","last_unstale":"2023-11-28T18:23:08.134612+0000","last_undegraded":"2023-11-28T18:23:08.134612+0000","last_fullsized":"2023-11-28T18:23:08.134612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:38:58.943122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113040+0000","last_change":"2023-11-28T18:23:08.122878+0000","last_active":"2023-11-28T18:23:09.113040+0000","last_peered":"2023-11-28T18:23:09.113040+0000","last_clean":"2023-11-28T18:23:09.113040+0000","last_became_active":"2023-11-28T18:23:08.122609+0000","last_became_peered":"2023-11-28T18:23:08.122609+0000","last_unstale":"2023-11-28T18:23:09.113040+0000","last_undegraded":"2023-11-28T18:23:09.113040+0000","last_fullsized":"2023-11-28T18:23:09.113040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:00.610074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.139912+0000","last_change":"2023-11-28T18:23:08.133485+0000","last_active":"2023-11-28T18:23:08.139912+0000","last_peered":"2023-11-28T18:23:08.139912+0000","last_clean":"2023-11-28T18:23:08.139912+0000","last_became_active":"2023-11-28T18:23:08.133211+0000","last_became_peered":"2023-11-28T18:23:08.133211+0000","last_unstale":"2023-11-28T18:23:08.139912+0000","last_undegraded":"2023-11-28T18:23:08.139912+0000","last_fullsized":"2023-11-28T18:23:08.139912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:57:22.781432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.126558+0000","last_change":"2023-11-28T18:23:08.117903+0000","last_active":"2023-11-28T18:23:08.126558+0000","last_peered":"2023-11-28T18:23:08.126558+0000","last_clean":"2023-11-28T18:23:08.126558+0000","last_became_active":"2023-11-28T18:23:08.117482+0000","last_became_peered":"2023-11-28T18:23:08.117482+0000","last_unstale":"2023-11-28T18:23:08.126558+0000","last_undegraded":"2023-11-28T18:23:08.126558+0000","last_fullsized":"2023-11-28T18:23:08.126558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:39:20.437743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142696+0000","last_change":"2023-11-28T18:23:08.135606+0000","last_active":"2023-11-28T18:23:08.142696+0000","last_peered":"2023-11-28T18:23:08.142696+0000","last_clean":"2023-11-28T18:23:08.142696+0000","last_became_active":"2023-11-28T18:23:08.135328+0000","last_became_peered":"2023-11-28T18:23:08.135328+0000","last_unstale":"2023-11-28T18:23:08.142696+0000","last_undegraded":"2023-11-28T18:23:08.142696+0000","last_fullsized":"2023-11-28T18:23:08.142696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:51:22.378042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110046+0000","last_change":"2023-11-28T18:23:08.123993+0000","last_active":"2023-11-28T18:23:09.110046+0000","last_peered":"2023-11-28T18:23:09.110046+0000","last_clean":"2023-11-28T18:23:09.110046+0000","last_became_active":"2023-11-28T18:23:08.123583+0000","last_became_peered":"2023-11-28T18:23:08.123583+0000","last_unstale":"2023-11-28T18:23:09.110046+0000","last_undegraded":"2023-11-28T18:23:09.110046+0000","last_fullsized":"2023-11-28T18:23:09.110046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:16:33.089307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:08.120983+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:08.120486+0000","last_became_peered":"2023-11-28T18:23:08.120486+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:46:12.881629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.139158+0000","last_change":"2023-11-28T18:23:08.130169+0000","last_active":"2023-11-28T18:23:08.139158+0000","last_peered":"2023-11-28T18:23:08.139158+0000","last_clean":"2023-11-28T18:23:08.139158+0000","last_became_active":"2023-11-28T18:23:08.129771+0000","last_became_peered":"2023-11-28T18:23:08.129771+0000","last_unstale":"2023-11-28T18:23:08.139158+0000","last_undegraded":"2023-11-28T18:23:08.139158+0000","last_fullsized":"2023-11-28T18:23:08.139158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:02:13.375041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.130031+0000","last_change":"2023-11-28T18:23:08.123268+0000","last_active":"2023-11-28T18:23:08.130031+0000","last_peered":"2023-11-28T18:23:08.130031+0000","last_clean":"2023-11-28T18:23:08.130031+0000","last_became_active":"2023-11-28T18:23:08.122978+0000","last_became_peered":"2023-11-28T18:23:08.122978+0000","last_unstale":"2023-11-28T18:23:08.130031+0000","last_undegraded":"2023-11-28T18:23:08.130031+0000","last_fullsized":"2023-11-28T18:23:08.130031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:09:25.753235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.124886+0000","last_change":"2023-11-28T18:23:08.118361+0000","last_active":"2023-11-28T18:23:08.124886+0000","last_peered":"2023-11-28T18:23:08.124886+0000","last_clean":"2023-11-28T18:23:08.124886+0000","last_became_active":"2023-11-28T18:23:08.117973+0000","last_became_peered":"2023-11-28T18:23:08.117973+0000","last_unstale":"2023-11-28T18:23:08.124886+0000","last_undegraded":"2023-11-28T18:23:08.124886+0000","last_fullsized":"2023-11-28T18:23:08.124886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:07.540205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105530+0000","last_change":"2023-11-28T18:23:08.139744+0000","last_active":"2023-11-28T18:23:09.105530+0000","last_peered":"2023-11-28T18:23:09.105530+0000","last_clean":"2023-11-28T18:23:09.105530+0000","last_became_active":"2023-11-28T18:23:08.139464+0000","last_became_peered":"2023-11-28T18:23:08.139464+0000","last_unstale":"2023-11-28T18:23:09.105530+0000","last_undegraded":"2023-11-28T18:23:09.105530+0000","last_fullsized":"2023-11-28T18:23:09.105530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:19:34.780979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113036+0000","last_change":"2023-11-28T18:23:08.119103+0000","last_active":"2023-11-28T18:23:09.113036+0000","last_peered":"2023-11-28T18:23:09.113036+0000","last_clean":"2023-11-28T18:23:09.113036+0000","last_became_active":"2023-11-28T18:23:08.116080+0000","last_became_peered":"2023-11-28T18:23:08.116080+0000","last_unstale":"2023-11-28T18:23:09.113036+0000","last_undegraded":"2023-11-28T18:23:09.113036+0000","last_fullsized":"2023-11-28T18:23:09.113036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:44.279413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113688+0000","last_change":"2023-11-28T18:23:08.126817+0000","last_active":"2023-11-28T18:23:09.113688+0000","last_peered":"2023-11-28T18:23:09.113688+0000","last_clean":"2023-11-28T18:23:09.113688+0000","last_became_active":"2023-11-28T18:23:08.126540+0000","last_became_peered":"2023-11-28T18:23:08.126540+0000","last_unstale":"2023-11-28T18:23:09.113688+0000","last_undegraded":"2023-11-28T18:23:09.113688+0000","last_fullsized":"2023-11-28T18:23:09.113688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:08.807847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.124678+0000","last_change":"2023-11-28T18:23:08.121775+0000","last_active":"2023-11-28T18:23:08.124678+0000","last_peered":"2023-11-28T18:23:08.124678+0000","last_clean":"2023-11-28T18:23:08.124678+0000","last_became_active":"2023-11-28T18:23:08.121466+0000","last_became_peered":"2023-11-28T18:23:08.121466+0000","last_unstale":"2023-11-28T18:23:08.124678+0000","last_undegraded":"2023-11-28T18:23:08.124678+0000","last_fullsized":"2023-11-28T18:23:08.124678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:58.099267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.143459+0000","last_change":"2023-11-28T18:23:08.138282+0000","last_active":"2023-11-28T18:23:08.143459+0000","last_peered":"2023-11-28T18:23:08.143459+0000","last_clean":"2023-11-28T18:23:08.143459+0000","last_became_active":"2023-11-28T18:23:08.137171+0000","last_became_peered":"2023-11-28T18:23:08.137171+0000","last_unstale":"2023-11-28T18:23:08.143459+0000","last_undegraded":"2023-11-28T18:23:08.143459+0000","last_fullsized":"2023-11-28T18:23:08.143459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:30:18.669500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110124+0000","last_change":"2023-11-28T18:23:08.133521+0000","last_active":"2023-11-28T18:23:09.110124+0000","last_peered":"2023-11-28T18:23:09.110124+0000","last_clean":"2023-11-28T18:23:09.110124+0000","last_became_active":"2023-11-28T18:23:08.133248+0000","last_became_peered":"2023-11-28T18:23:08.133248+0000","last_unstale":"2023-11-28T18:23:09.110124+0000","last_undegraded":"2023-11-28T18:23:09.110124+0000","last_fullsized":"2023-11-28T18:23:09.110124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:17:34.585808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131127+0000","last_change":"2023-11-28T18:23:08.129150+0000","last_active":"2023-11-28T18:23:08.131127+0000","last_peered":"2023-11-28T18:23:08.131127+0000","last_clean":"2023-11-28T18:23:08.131127+0000","last_became_active":"2023-11-28T18:23:08.128622+0000","last_became_peered":"2023-11-28T18:23:08.128622+0000","last_unstale":"2023-11-28T18:23:08.131127+0000","last_undegraded":"2023-11-28T18:23:08.131127+0000","last_fullsized":"2023-11-28T18:23:08.131127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:56.050282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113274+0000","last_change":"2023-11-28T18:23:08.131333+0000","last_active":"2023-11-28T18:23:09.113274+0000","last_peered":"2023-11-28T18:23:09.113274+0000","last_clean":"2023-11-28T18:23:09.113274+0000","last_became_active":"2023-11-28T18:23:08.130917+0000","last_became_peered":"2023-11-28T18:23:08.130917+0000","last_unstale":"2023-11-28T18:23:09.113274+0000","last_undegraded":"2023-11-28T18:23:09.113274+0000","last_fullsized":"2023-11-28T18:23:09.113274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:04:48.114417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.133526+0000","last_change":"2023-11-28T18:23:08.132355+0000","last_active":"2023-11-28T18:23:08.133526+0000","last_peered":"2023-11-28T18:23:08.133526+0000","last_clean":"2023-11-28T18:23:08.133526+0000","last_became_active":"2023-11-28T18:23:08.132016+0000","last_became_peered":"2023-11-28T18:23:08.132016+0000","last_unstale":"2023-11-28T18:23:08.133526+0000","last_undegraded":"2023-11-28T18:23:08.133526+0000","last_fullsized":"2023-11-28T18:23:08.133526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:31:32.314253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113066+0000","last_change":"2023-11-28T18:23:08.119935+0000","last_active":"2023-11-28T18:23:09.113066+0000","last_peered":"2023-11-28T18:23:09.113066+0000","last_clean":"2023-11-28T18:23:09.113066+0000","last_became_active":"2023-11-28T18:23:08.118372+0000","last_became_peered":"2023-11-28T18:23:08.118372+0000","last_unstale":"2023-11-28T18:23:09.113066+0000","last_undegraded":"2023-11-28T18:23:09.113066+0000","last_fullsized":"2023-11-28T18:23:09.113066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:52.339024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.133518+0000","last_change":"2023-11-28T18:23:08.132272+0000","last_active":"2023-11-28T18:23:08.133518+0000","last_peered":"2023-11-28T18:23:08.133518+0000","last_clean":"2023-11-28T18:23:08.133518+0000","last_became_active":"2023-11-28T18:23:08.131985+0000","last_became_peered":"2023-11-28T18:23:08.131985+0000","last_unstale":"2023-11-28T18:23:08.133518+0000","last_undegraded":"2023-11-28T18:23:08.133518+0000","last_fullsized":"2023-11-28T18:23:08.133518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:00:03.297320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110084+0000","last_change":"2023-11-28T18:23:08.127816+0000","last_active":"2023-11-28T18:23:09.110084+0000","last_peered":"2023-11-28T18:23:09.110084+0000","last_clean":"2023-11-28T18:23:09.110084+0000","last_became_active":"2023-11-28T18:23:08.127380+0000","last_became_peered":"2023-11-28T18:23:08.127380+0000","last_unstale":"2023-11-28T18:23:09.110084+0000","last_undegraded":"2023-11-28T18:23:09.110084+0000","last_fullsized":"2023-11-28T18:23:09.110084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:31.242361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105706+0000","last_change":"2023-11-28T18:23:08.128188+0000","last_active":"2023-11-28T18:23:09.105706+0000","last_peered":"2023-11-28T18:23:09.105706+0000","last_clean":"2023-11-28T18:23:09.105706+0000","last_became_active":"2023-11-28T18:23:08.127860+0000","last_became_peered":"2023-11-28T18:23:08.127860+0000","last_unstale":"2023-11-28T18:23:09.105706+0000","last_undegraded":"2023-11-28T18:23:09.105706+0000","last_fullsized":"2023-11-28T18:23:09.105706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:12.095768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.133547+0000","last_change":"2023-11-28T18:23:08.132381+0000","last_active":"2023-11-28T18:23:08.133547+0000","last_peered":"2023-11-28T18:23:08.133547+0000","last_clean":"2023-11-28T18:23:08.133547+0000","last_became_active":"2023-11-28T18:23:08.132072+0000","last_became_peered":"2023-11-28T18:23:08.132072+0000","last_unstale":"2023-11-28T18:23:08.133547+0000","last_undegraded":"2023-11-28T18:23:08.133547+0000","last_fullsized":"2023-11-28T18:23:08.133547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:27:13.360596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117010+0000","last_change":"2023-11-28T18:23:08.115379+0000","last_active":"2023-11-28T18:23:08.117010+0000","last_peered":"2023-11-28T18:23:08.117010+0000","last_clean":"2023-11-28T18:23:08.117010+0000","last_became_active":"2023-11-28T18:23:08.115236+0000","last_became_peered":"2023-11-28T18:23:08.115236+0000","last_unstale":"2023-11-28T18:23:08.117010+0000","last_undegraded":"2023-11-28T18:23:08.117010+0000","last_fullsized":"2023-11-28T18:23:08.117010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:27.176600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113293+0000","last_change":"2023-11-28T18:23:08.121402+0000","last_active":"2023-11-28T18:23:09.113293+0000","last_peered":"2023-11-28T18:23:09.113293+0000","last_clean":"2023-11-28T18:23:09.113293+0000","last_became_active":"2023-11-28T18:23:08.121129+0000","last_became_peered":"2023-11-28T18:23:08.121129+0000","last_unstale":"2023-11-28T18:23:09.113293+0000","last_undegraded":"2023-11-28T18:23:09.113293+0000","last_fullsized":"2023-11-28T18:23:09.113293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:02.927891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.124746+0000","last_change":"2023-11-28T18:23:08.121594+0000","last_active":"2023-11-28T18:23:08.124746+0000","last_peered":"2023-11-28T18:23:08.124746+0000","last_clean":"2023-11-28T18:23:08.124746+0000","last_became_active":"2023-11-28T18:23:08.119009+0000","last_became_peered":"2023-11-28T18:23:08.119009+0000","last_unstale":"2023-11-28T18:23:08.124746+0000","last_undegraded":"2023-11-28T18:23:08.124746+0000","last_fullsized":"2023-11-28T18:23:08.124746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:03.581437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109676+0000","last_change":"2023-11-28T18:23:08.115403+0000","last_active":"2023-11-28T18:23:09.109676+0000","last_peered":"2023-11-28T18:23:09.109676+0000","last_clean":"2023-11-28T18:23:09.109676+0000","last_became_active":"2023-11-28T18:23:08.114955+0000","last_became_peered":"2023-11-28T18:23:08.114955+0000","last_unstale":"2023-11-28T18:23:09.109676+0000","last_undegraded":"2023-11-28T18:23:09.109676+0000","last_fullsized":"2023-11-28T18:23:09.109676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:41.242830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.138939+0000","last_change":"2023-11-28T18:23:08.129745+0000","last_active":"2023-11-28T18:23:08.138939+0000","last_peered":"2023-11-28T18:23:08.138939+0000","last_clean":"2023-11-28T18:23:08.138939+0000","last_became_active":"2023-11-28T18:23:08.129388+0000","last_became_peered":"2023-11-28T18:23:08.129388+0000","last_unstale":"2023-11-28T18:23:08.138939+0000","last_undegraded":"2023-11-28T18:23:08.138939+0000","last_fullsized":"2023-11-28T18:23:08.138939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:44:10.418671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.109812+0000","last_change":"2023-11-28T18:23:07.664773+0000","last_active":"2023-11-28T18:23:08.109812+0000","last_peered":"2023-11-28T18:23:08.109812+0000","last_clean":"2023-11-28T18:23:08.109812+0000","last_became_active":"2023-11-28T18:23:07.664478+0000","last_became_peered":"2023-11-28T18:23:07.664478+0000","last_unstale":"2023-11-28T18:23:08.109812+0000","last_undegraded":"2023-11-28T18:23:08.109812+0000","last_fullsized":"2023-11-28T18:23:08.109812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:28.322454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131144+0000","last_change":"2023-11-28T18:23:08.129441+0000","last_active":"2023-11-28T18:23:08.131144+0000","last_peered":"2023-11-28T18:23:08.131144+0000","last_clean":"2023-11-28T18:23:08.131144+0000","last_became_active":"2023-11-28T18:23:08.129143+0000","last_became_peered":"2023-11-28T18:23:08.129143+0000","last_unstale":"2023-11-28T18:23:08.131144+0000","last_undegraded":"2023-11-28T18:23:08.131144+0000","last_fullsized":"2023-11-28T18:23:08.131144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:01:48.849011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109569+0000","last_change":"2023-11-28T18:23:07.668192+0000","last_active":"2023-11-28T18:23:09.109569+0000","last_peered":"2023-11-28T18:23:09.109569+0000","last_clean":"2023-11-28T18:23:09.109569+0000","last_became_active":"2023-11-28T18:23:07.667780+0000","last_became_peered":"2023-11-28T18:23:07.667780+0000","last_unstale":"2023-11-28T18:23:09.109569+0000","last_undegraded":"2023-11-28T18:23:09.109569+0000","last_fullsized":"2023-11-28T18:23:09.109569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.109926+0000","last_change":"2023-11-28T18:23:07.665608+0000","last_active":"2023-11-28T18:23:08.109926+0000","last_peered":"2023-11-28T18:23:08.109926+0000","last_clean":"2023-11-28T18:23:08.109926+0000","last_became_active":"2023-11-28T18:23:07.665361+0000","last_became_peered":"2023-11-28T18:23:07.665361+0000","last_unstale":"2023-11-28T18:23:08.109926+0000","last_undegraded":"2023-11-28T18:23:08.109926+0000","last_fullsized":"2023-11-28T18:23:08.109926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:17.279597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.126580+0000","last_change":"2023-11-28T18:23:08.118091+0000","last_active":"2023-11-28T18:23:08.126580+0000","last_peered":"2023-11-28T18:23:08.126580+0000","last_clean":"2023-11-28T18:23:08.126580+0000","last_became_active":"2023-11-28T18:23:08.117524+0000","last_became_peered":"2023-11-28T18:23:08.117524+0000","last_unstale":"2023-11-28T18:23:08.126580+0000","last_undegraded":"2023-11-28T18:23:08.126580+0000","last_fullsized":"2023-11-28T18:23:08.126580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:19:55.437131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113169+0000","last_change":"2023-11-28T18:23:07.667847+0000","last_active":"2023-11-28T18:23:09.113169+0000","last_peered":"2023-11-28T18:23:09.113169+0000","last_clean":"2023-11-28T18:23:09.113169+0000","last_became_active":"2023-11-28T18:23:07.667466+0000","last_became_peered":"2023-11-28T18:23:07.667466+0000","last_unstale":"2023-11-28T18:23:09.113169+0000","last_undegraded":"2023-11-28T18:23:09.113169+0000","last_fullsized":"2023-11-28T18:23:09.113169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:06.807146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.127497+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.127216+0000","last_became_peered":"2023-11-28T18:23:08.127216+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:03:14.999484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109883+0000","last_change":"2023-11-28T18:23:07.681698+0000","last_active":"2023-11-28T18:23:09.109883+0000","last_peered":"2023-11-28T18:23:09.109883+0000","last_clean":"2023-11-28T18:23:09.109883+0000","last_became_active":"2023-11-28T18:23:07.681365+0000","last_became_peered":"2023-11-28T18:23:07.681365+0000","last_unstale":"2023-11-28T18:23:09.109883+0000","last_undegraded":"2023-11-28T18:23:09.109883+0000","last_fullsized":"2023-11-28T18:23:09.109883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:21.985029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113361+0000","last_change":"2023-11-28T18:23:08.132797+0000","last_active":"2023-11-28T18:23:09.113361+0000","last_peered":"2023-11-28T18:23:09.113361+0000","last_clean":"2023-11-28T18:23:09.113361+0000","last_became_active":"2023-11-28T18:23:08.132525+0000","last_became_peered":"2023-11-28T18:23:08.132525+0000","last_unstale":"2023-11-28T18:23:09.113361+0000","last_undegraded":"2023-11-28T18:23:09.113361+0000","last_fullsized":"2023-11-28T18:23:09.113361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:12:10.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113468+0000","last_change":"2023-11-28T18:23:07.672337+0000","last_active":"2023-11-28T18:23:09.113468+0000","last_peered":"2023-11-28T18:23:09.113468+0000","last_clean":"2023-11-28T18:23:09.113468+0000","last_became_active":"2023-11-28T18:23:07.671922+0000","last_became_peered":"2023-11-28T18:23:07.671922+0000","last_unstale":"2023-11-28T18:23:09.113468+0000","last_undegraded":"2023-11-28T18:23:09.113468+0000","last_fullsized":"2023-11-28T18:23:09.113468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:22.936770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.130231+0000","last_change":"2023-11-28T18:23:08.127919+0000","last_active":"2023-11-28T18:23:08.130231+0000","last_peered":"2023-11-28T18:23:08.130231+0000","last_clean":"2023-11-28T18:23:08.130231+0000","last_became_active":"2023-11-28T18:23:08.127602+0000","last_became_peered":"2023-11-28T18:23:08.127602+0000","last_unstale":"2023-11-28T18:23:08.130231+0000","last_undegraded":"2023-11-28T18:23:08.130231+0000","last_fullsized":"2023-11-28T18:23:08.130231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:09:42.431643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105765+0000","last_change":"2023-11-28T18:23:07.679219+0000","last_active":"2023-11-28T18:23:09.105765+0000","last_peered":"2023-11-28T18:23:09.105765+0000","last_clean":"2023-11-28T18:23:09.105765+0000","last_became_active":"2023-11-28T18:23:07.678920+0000","last_became_peered":"2023-11-28T18:23:07.678920+0000","last_unstale":"2023-11-28T18:23:09.105765+0000","last_undegraded":"2023-11-28T18:23:09.105765+0000","last_fullsized":"2023-11-28T18:23:09.105765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:02.332690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.140486+0000","last_change":"2023-11-28T18:23:08.132429+0000","last_active":"2023-11-28T18:23:08.140486+0000","last_peered":"2023-11-28T18:23:08.140486+0000","last_clean":"2023-11-28T18:23:08.140486+0000","last_became_active":"2023-11-28T18:23:08.132106+0000","last_became_peered":"2023-11-28T18:23:08.132106+0000","last_unstale":"2023-11-28T18:23:08.140486+0000","last_undegraded":"2023-11-28T18:23:08.140486+0000","last_fullsized":"2023-11-28T18:23:08.140486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:43:59.775353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:07.667479+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:07.665310+0000","last_became_peered":"2023-11-28T18:23:07.665310+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:03.174425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142790+0000","last_change":"2023-11-28T18:23:08.136812+0000","last_active":"2023-11-28T18:23:08.142790+0000","last_peered":"2023-11-28T18:23:08.142790+0000","last_clean":"2023-11-28T18:23:08.142790+0000","last_became_active":"2023-11-28T18:23:08.136525+0000","last_became_peered":"2023-11-28T18:23:08.136525+0000","last_unstale":"2023-11-28T18:23:08.142790+0000","last_undegraded":"2023-11-28T18:23:08.142790+0000","last_fullsized":"2023-11-28T18:23:08.142790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:09:17.327330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.104827+0000","last_change":"2023-11-28T18:23:07.660519+0000","last_active":"2023-11-28T18:23:08.104827+0000","last_peered":"2023-11-28T18:23:08.104827+0000","last_clean":"2023-11-28T18:23:08.104827+0000","last_became_active":"2023-11-28T18:23:07.660205+0000","last_became_peered":"2023-11-28T18:23:07.660205+0000","last_unstale":"2023-11-28T18:23:08.104827+0000","last_undegraded":"2023-11-28T18:23:08.104827+0000","last_fullsized":"2023-11-28T18:23:08.104827+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:04:54.125737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.130234+0000","last_change":"2023-11-28T18:23:08.127921+0000","last_active":"2023-11-28T18:23:08.130234+0000","last_peered":"2023-11-28T18:23:08.130234+0000","last_clean":"2023-11-28T18:23:08.130234+0000","last_became_active":"2023-11-28T18:23:08.127678+0000","last_became_peered":"2023-11-28T18:23:08.127678+0000","last_unstale":"2023-11-28T18:23:08.130234+0000","last_undegraded":"2023-11-28T18:23:08.130234+0000","last_fullsized":"2023-11-28T18:23:08.130234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:56:02.281565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113417+0000","last_change":"2023-11-28T18:23:07.671589+0000","last_active":"2023-11-28T18:23:09.113417+0000","last_peered":"2023-11-28T18:23:09.113417+0000","last_clean":"2023-11-28T18:23:09.113417+0000","last_became_active":"2023-11-28T18:23:07.671291+0000","last_became_peered":"2023-11-28T18:23:07.671291+0000","last_unstale":"2023-11-28T18:23:09.113417+0000","last_undegraded":"2023-11-28T18:23:09.113417+0000","last_fullsized":"2023-11-28T18:23:09.113417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:23.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.143552+0000","last_change":"2023-11-28T18:23:08.141316+0000","last_active":"2023-11-28T18:23:08.143552+0000","last_peered":"2023-11-28T18:23:08.143552+0000","last_clean":"2023-11-28T18:23:08.143552+0000","last_became_active":"2023-11-28T18:23:08.141011+0000","last_became_peered":"2023-11-28T18:23:08.141011+0000","last_unstale":"2023-11-28T18:23:08.143552+0000","last_undegraded":"2023-11-28T18:23:08.143552+0000","last_fullsized":"2023-11-28T18:23:08.143552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:05:09.149956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109447+0000","last_change":"2023-11-28T18:23:07.671192+0000","last_active":"2023-11-28T18:23:09.109447+0000","last_peered":"2023-11-28T18:23:09.109447+0000","last_clean":"2023-11-28T18:23:09.109447+0000","last_became_active":"2023-11-28T18:23:07.670812+0000","last_became_peered":"2023-11-28T18:23:07.670812+0000","last_unstale":"2023-11-28T18:23:09.109447+0000","last_undegraded":"2023-11-28T18:23:09.109447+0000","last_fullsized":"2023-11-28T18:23:09.109447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:28.806061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109506+0000","last_change":"2023-11-28T18:23:08.115229+0000","last_active":"2023-11-28T18:23:09.109506+0000","last_peered":"2023-11-28T18:23:09.109506+0000","last_clean":"2023-11-28T18:23:09.109506+0000","last_became_active":"2023-11-28T18:23:08.114541+0000","last_became_peered":"2023-11-28T18:23:08.114541+0000","last_unstale":"2023-11-28T18:23:09.109506+0000","last_undegraded":"2023-11-28T18:23:09.109506+0000","last_fullsized":"2023-11-28T18:23:09.109506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:38.816731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.106715+0000","last_change":"2023-11-28T18:23:07.673554+0000","last_active":"2023-11-28T18:23:08.106715+0000","last_peered":"2023-11-28T18:23:08.106715+0000","last_clean":"2023-11-28T18:23:08.106715+0000","last_became_active":"2023-11-28T18:23:07.672923+0000","last_became_peered":"2023-11-28T18:23:07.672923+0000","last_unstale":"2023-11-28T18:23:08.106715+0000","last_undegraded":"2023-11-28T18:23:08.106715+0000","last_fullsized":"2023-11-28T18:23:08.106715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:49.805572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.143546+0000","last_change":"2023-11-28T18:23:08.141169+0000","last_active":"2023-11-28T18:23:08.143546+0000","last_peered":"2023-11-28T18:23:08.143546+0000","last_clean":"2023-11-28T18:23:08.143546+0000","last_became_active":"2023-11-28T18:23:08.140801+0000","last_became_peered":"2023-11-28T18:23:08.140801+0000","last_unstale":"2023-11-28T18:23:08.143546+0000","last_undegraded":"2023-11-28T18:23:08.143546+0000","last_fullsized":"2023-11-28T18:23:08.143546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:53:42.320999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.110940+0000","last_change":"2023-11-28T18:23:07.680094+0000","last_active":"2023-11-28T18:23:08.110940+0000","last_peered":"2023-11-28T18:23:08.110940+0000","last_clean":"2023-11-28T18:23:08.110940+0000","last_became_active":"2023-11-28T18:23:07.679801+0000","last_became_peered":"2023-11-28T18:23:07.679801+0000","last_unstale":"2023-11-28T18:23:08.110940+0000","last_undegraded":"2023-11-28T18:23:08.110940+0000","last_fullsized":"2023-11-28T18:23:08.110940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:02.916155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142832+0000","last_change":"2023-11-28T18:23:08.137825+0000","last_active":"2023-11-28T18:23:08.142832+0000","last_peered":"2023-11-28T18:23:08.142832+0000","last_clean":"2023-11-28T18:23:08.142832+0000","last_became_active":"2023-11-28T18:23:08.137499+0000","last_became_peered":"2023-11-28T18:23:08.137499+0000","last_unstale":"2023-11-28T18:23:08.142832+0000","last_undegraded":"2023-11-28T18:23:08.142832+0000","last_fullsized":"2023-11-28T18:23:08.142832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:54.482524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.106617+0000","last_change":"2023-11-28T18:23:07.670359+0000","last_active":"2023-11-28T18:23:08.106617+0000","last_peered":"2023-11-28T18:23:08.106617+0000","last_clean":"2023-11-28T18:23:08.106617+0000","last_became_active":"2023-11-28T18:23:07.669687+0000","last_became_peered":"2023-11-28T18:23:07.669687+0000","last_unstale":"2023-11-28T18:23:08.106617+0000","last_undegraded":"2023-11-28T18:23:08.106617+0000","last_fullsized":"2023-11-28T18:23:08.106617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:45.450623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.143538+0000","last_change":"2023-11-28T18:23:08.141300+0000","last_active":"2023-11-28T18:23:08.143538+0000","last_peered":"2023-11-28T18:23:08.143538+0000","last_clean":"2023-11-28T18:23:08.143538+0000","last_became_active":"2023-11-28T18:23:08.141037+0000","last_became_peered":"2023-11-28T18:23:08.141037+0000","last_unstale":"2023-11-28T18:23:08.143538+0000","last_undegraded":"2023-11-28T18:23:08.143538+0000","last_fullsized":"2023-11-28T18:23:08.143538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:19:41.709261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105793+0000","last_change":"2023-11-28T18:23:07.673086+0000","last_active":"2023-11-28T18:23:09.105793+0000","last_peered":"2023-11-28T18:23:09.105793+0000","last_clean":"2023-11-28T18:23:09.105793+0000","last_became_active":"2023-11-28T18:23:07.672773+0000","last_became_peered":"2023-11-28T18:23:07.672773+0000","last_unstale":"2023-11-28T18:23:09.105793+0000","last_undegraded":"2023-11-28T18:23:09.105793+0000","last_fullsized":"2023-11-28T18:23:09.105793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:34.962428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113586+0000","last_change":"2023-11-28T18:23:08.121987+0000","last_active":"2023-11-28T18:23:09.113586+0000","last_peered":"2023-11-28T18:23:09.113586+0000","last_clean":"2023-11-28T18:23:09.113586+0000","last_became_active":"2023-11-28T18:23:08.121279+0000","last_became_peered":"2023-11-28T18:23:08.121279+0000","last_unstale":"2023-11-28T18:23:09.113586+0000","last_undegraded":"2023-11-28T18:23:09.113586+0000","last_fullsized":"2023-11-28T18:23:09.113586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:47.973973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.106199+0000","last_change":"2023-11-28T18:23:07.681312+0000","last_active":"2023-11-28T18:23:08.106199+0000","last_peered":"2023-11-28T18:23:08.106199+0000","last_clean":"2023-11-28T18:23:08.106199+0000","last_became_active":"2023-11-28T18:23:07.680806+0000","last_became_peered":"2023-11-28T18:23:07.680806+0000","last_unstale":"2023-11-28T18:23:08.106199+0000","last_undegraded":"2023-11-28T18:23:08.106199+0000","last_fullsized":"2023-11-28T18:23:08.106199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:58:41.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142828+0000","last_change":"2023-11-28T18:23:08.137874+0000","last_active":"2023-11-28T18:23:08.142828+0000","last_peered":"2023-11-28T18:23:08.142828+0000","last_clean":"2023-11-28T18:23:08.142828+0000","last_became_active":"2023-11-28T18:23:08.137104+0000","last_became_peered":"2023-11-28T18:23:08.137104+0000","last_unstale":"2023-11-28T18:23:08.142828+0000","last_undegraded":"2023-11-28T18:23:08.142828+0000","last_fullsized":"2023-11-28T18:23:08.142828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:28.565657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-11-28T18:23:07.667413+0000","last_change":"2023-11-28T18:22:56.774057+0000","last_active":"2023-11-28T18:23:07.667413+0000","last_peered":"2023-11-28T18:23:07.667413+0000","last_clean":"2023-11-28T18:23:07.667413+0000","last_became_active":"2023-11-28T18:22:56.772763+0000","last_became_peered":"2023-11-28T18:22:56.772763+0000","last_unstale":"2023-11-28T18:23:07.667413+0000","last_undegraded":"2023-11-28T18:23:07.667413+0000","last_fullsized":"2023-11-28T18:23:07.667413+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_clean_scrub_stamp":"2023-11-28T18:22:55.591288+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:05.470408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113059+0000","last_change":"2023-11-28T18:23:07.674072+0000","last_active":"2023-11-28T18:23:09.113059+0000","last_peered":"2023-11-28T18:23:09.113059+0000","last_clean":"2023-11-28T18:23:09.113059+0000","last_became_active":"2023-11-28T18:23:07.673220+0000","last_became_peered":"2023-11-28T18:23:07.673220+0000","last_unstale":"2023-11-28T18:23:09.113059+0000","last_undegraded":"2023-11-28T18:23:09.113059+0000","last_fullsized":"2023-11-28T18:23:09.113059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:58.524112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.132519+0000","last_change":"2023-11-28T18:23:08.130989+0000","last_active":"2023-11-28T18:23:08.132519+0000","last_peered":"2023-11-28T18:23:08.132519+0000","last_clean":"2023-11-28T18:23:08.132519+0000","last_became_active":"2023-11-28T18:23:08.130646+0000","last_became_peered":"2023-11-28T18:23:08.130646+0000","last_unstale":"2023-11-28T18:23:08.132519+0000","last_undegraded":"2023-11-28T18:23:08.132519+0000","last_fullsized":"2023-11-28T18:23:08.132519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:06.796275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.116946+0000","last_change":"2023-11-28T18:23:07.686046+0000","last_active":"2023-11-28T18:23:08.116946+0000","last_peered":"2023-11-28T18:23:08.116946+0000","last_clean":"2023-11-28T18:23:08.116946+0000","last_became_active":"2023-11-28T18:23:07.685739+0000","last_became_peered":"2023-11-28T18:23:07.685739+0000","last_unstale":"2023-11-28T18:23:08.116946+0000","last_undegraded":"2023-11-28T18:23:08.116946+0000","last_fullsized":"2023-11-28T18:23:08.116946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:06.485631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131851+0000","last_change":"2023-11-28T18:23:08.130355+0000","last_active":"2023-11-28T18:23:08.131851+0000","last_peered":"2023-11-28T18:23:08.131851+0000","last_clean":"2023-11-28T18:23:08.131851+0000","last_became_active":"2023-11-28T18:23:08.129946+0000","last_became_peered":"2023-11-28T18:23:08.129946+0000","last_unstale":"2023-11-28T18:23:08.131851+0000","last_undegraded":"2023-11-28T18:23:08.131851+0000","last_fullsized":"2023-11-28T18:23:08.131851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:05:01.564614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.106082+0000","last_change":"2023-11-28T18:23:07.664827+0000","last_active":"2023-11-28T18:23:08.106082+0000","last_peered":"2023-11-28T18:23:08.106082+0000","last_clean":"2023-11-28T18:23:08.106082+0000","last_became_active":"2023-11-28T18:23:07.664503+0000","last_became_peered":"2023-11-28T18:23:07.664503+0000","last_unstale":"2023-11-28T18:23:08.106082+0000","last_undegraded":"2023-11-28T18:23:08.106082+0000","last_fullsized":"2023-11-28T18:23:08.106082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.238694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.123498+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.123197+0000","last_became_peered":"2023-11-28T18:23:08.123197+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:08:42.817848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113555+0000","last_change":"2023-11-28T18:23:07.673345+0000","last_active":"2023-11-28T18:23:09.113555+0000","last_peered":"2023-11-28T18:23:09.113555+0000","last_clean":"2023-11-28T18:23:09.113555+0000","last_became_active":"2023-11-28T18:23:07.672982+0000","last_became_peered":"2023-11-28T18:23:07.672982+0000","last_unstale":"2023-11-28T18:23:09.113555+0000","last_undegraded":"2023-11-28T18:23:09.113555+0000","last_fullsized":"2023-11-28T18:23:09.113555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:26.594232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.130392+0000","last_change":"2023-11-28T18:23:08.129091+0000","last_active":"2023-11-28T18:23:08.130392+0000","last_peered":"2023-11-28T18:23:08.130392+0000","last_clean":"2023-11-28T18:23:08.130392+0000","last_became_active":"2023-11-28T18:23:08.128589+0000","last_became_peered":"2023-11-28T18:23:08.128589+0000","last_unstale":"2023-11-28T18:23:08.130392+0000","last_undegraded":"2023-11-28T18:23:08.130392+0000","last_fullsized":"2023-11-28T18:23:08.130392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:29:48.729543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105665+0000","last_change":"2023-11-28T18:23:07.671874+0000","last_active":"2023-11-28T18:23:09.105665+0000","last_peered":"2023-11-28T18:23:09.105665+0000","last_clean":"2023-11-28T18:23:09.105665+0000","last_became_active":"2023-11-28T18:23:07.671500+0000","last_became_peered":"2023-11-28T18:23:07.671500+0000","last_unstale":"2023-11-28T18:23:09.105665+0000","last_undegraded":"2023-11-28T18:23:09.105665+0000","last_fullsized":"2023-11-28T18:23:09.105665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:42.922770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113612+0000","last_change":"2023-11-28T18:23:08.127419+0000","last_active":"2023-11-28T18:23:09.113612+0000","last_peered":"2023-11-28T18:23:09.113612+0000","last_clean":"2023-11-28T18:23:09.113612+0000","last_became_active":"2023-11-28T18:23:08.127157+0000","last_became_peered":"2023-11-28T18:23:08.127157+0000","last_unstale":"2023-11-28T18:23:09.113612+0000","last_undegraded":"2023-11-28T18:23:09.113612+0000","last_fullsized":"2023-11-28T18:23:09.113612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:08.473882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.120536+0000","last_change":"2023-11-28T18:23:07.684725+0000","last_active":"2023-11-28T18:23:08.120536+0000","last_peered":"2023-11-28T18:23:08.120536+0000","last_clean":"2023-11-28T18:23:08.120536+0000","last_became_active":"2023-11-28T18:23:07.684363+0000","last_became_peered":"2023-11-28T18:23:07.684363+0000","last_unstale":"2023-11-28T18:23:08.120536+0000","last_undegraded":"2023-11-28T18:23:08.120536+0000","last_fullsized":"2023-11-28T18:23:08.120536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:16.380789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105690+0000","last_change":"2023-11-28T18:23:08.120741+0000","last_active":"2023-11-28T18:23:09.105690+0000","last_peered":"2023-11-28T18:23:09.105690+0000","last_clean":"2023-11-28T18:23:09.105690+0000","last_became_active":"2023-11-28T18:23:08.120421+0000","last_became_peered":"2023-11-28T18:23:08.120421+0000","last_unstale":"2023-11-28T18:23:09.105690+0000","last_undegraded":"2023-11-28T18:23:09.105690+0000","last_fullsized":"2023-11-28T18:23:09.105690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:53:27.131631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105889+0000","last_change":"2023-11-28T18:23:07.672505+0000","last_active":"2023-11-28T18:23:09.105889+0000","last_peered":"2023-11-28T18:23:09.105889+0000","last_clean":"2023-11-28T18:23:09.105889+0000","last_became_active":"2023-11-28T18:23:07.672204+0000","last_became_peered":"2023-11-28T18:23:07.672204+0000","last_unstale":"2023-11-28T18:23:09.105889+0000","last_undegraded":"2023-11-28T18:23:09.105889+0000","last_fullsized":"2023-11-28T18:23:09.105889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:01.341643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131034+0000","last_change":"2023-11-28T18:23:08.126198+0000","last_active":"2023-11-28T18:23:08.131034+0000","last_peered":"2023-11-28T18:23:08.131034+0000","last_clean":"2023-11-28T18:23:08.131034+0000","last_became_active":"2023-11-28T18:23:08.125935+0000","last_became_peered":"2023-11-28T18:23:08.125935+0000","last_unstale":"2023-11-28T18:23:08.131034+0000","last_undegraded":"2023-11-28T18:23:08.131034+0000","last_fullsized":"2023-11-28T18:23:08.131034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:13:34.328584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109909+0000","last_change":"2023-11-28T18:23:07.672170+0000","last_active":"2023-11-28T18:23:09.109909+0000","last_peered":"2023-11-28T18:23:09.109909+0000","last_clean":"2023-11-28T18:23:09.109909+0000","last_became_active":"2023-11-28T18:23:07.671683+0000","last_became_peered":"2023-11-28T18:23:07.671683+0000","last_unstale":"2023-11-28T18:23:09.109909+0000","last_undegraded":"2023-11-28T18:23:09.109909+0000","last_fullsized":"2023-11-28T18:23:09.109909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:41.860341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.125078+0000","last_change":"2023-11-28T18:23:08.119844+0000","last_active":"2023-11-28T18:23:08.125078+0000","last_peered":"2023-11-28T18:23:08.125078+0000","last_clean":"2023-11-28T18:23:08.125078+0000","last_became_active":"2023-11-28T18:23:08.119525+0000","last_became_peered":"2023-11-28T18:23:08.119525+0000","last_unstale":"2023-11-28T18:23:08.125078+0000","last_undegraded":"2023-11-28T18:23:08.125078+0000","last_fullsized":"2023-11-28T18:23:08.125078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:15.677195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.116787+0000","last_change":"2023-11-28T18:23:07.685651+0000","last_active":"2023-11-28T18:23:08.116787+0000","last_peered":"2023-11-28T18:23:08.116787+0000","last_clean":"2023-11-28T18:23:08.116787+0000","last_became_active":"2023-11-28T18:23:07.683869+0000","last_became_peered":"2023-11-28T18:23:07.683869+0000","last_unstale":"2023-11-28T18:23:08.116787+0000","last_undegraded":"2023-11-28T18:23:08.116787+0000","last_fullsized":"2023-11-28T18:23:08.116787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:35.383852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.139176+0000","last_change":"2023-11-28T18:23:08.130016+0000","last_active":"2023-11-28T18:23:08.139176+0000","last_peered":"2023-11-28T18:23:08.139176+0000","last_clean":"2023-11-28T18:23:08.139176+0000","last_became_active":"2023-11-28T18:23:08.129586+0000","last_became_peered":"2023-11-28T18:23:08.129586+0000","last_unstale":"2023-11-28T18:23:08.139176+0000","last_undegraded":"2023-11-28T18:23:08.139176+0000","last_fullsized":"2023-11-28T18:23:08.139176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:26:27.495717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105620+0000","last_change":"2023-11-28T18:23:07.671773+0000","last_active":"2023-11-28T18:23:09.105620+0000","last_peered":"2023-11-28T18:23:09.105620+0000","last_clean":"2023-11-28T18:23:09.105620+0000","last_became_active":"2023-11-28T18:23:07.671465+0000","last_became_peered":"2023-11-28T18:23:07.671465+0000","last_unstale":"2023-11-28T18:23:09.105620+0000","last_undegraded":"2023-11-28T18:23:09.105620+0000","last_fullsized":"2023-11-28T18:23:09.105620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:44.862780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.126598+0000","last_change":"2023-11-28T18:23:08.118146+0000","last_active":"2023-11-28T18:23:08.126598+0000","last_peered":"2023-11-28T18:23:08.126598+0000","last_clean":"2023-11-28T18:23:08.126598+0000","last_became_active":"2023-11-28T18:23:08.117848+0000","last_became_peered":"2023-11-28T18:23:08.117848+0000","last_unstale":"2023-11-28T18:23:08.126598+0000","last_undegraded":"2023-11-28T18:23:08.126598+0000","last_fullsized":"2023-11-28T18:23:08.126598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:25.367755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105627+0000","last_change":"2023-11-28T18:23:07.676709+0000","last_active":"2023-11-28T18:23:09.105627+0000","last_peered":"2023-11-28T18:23:09.105627+0000","last_clean":"2023-11-28T18:23:09.105627+0000","last_became_active":"2023-11-28T18:23:07.676414+0000","last_became_peered":"2023-11-28T18:23:07.676414+0000","last_unstale":"2023-11-28T18:23:09.105627+0000","last_undegraded":"2023-11-28T18:23:09.105627+0000","last_fullsized":"2023-11-28T18:23:09.105627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.282005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.114651+0000","last_change":"2023-11-28T18:23:08.112410+0000","last_active":"2023-11-28T18:23:08.114651+0000","last_peered":"2023-11-28T18:23:08.114651+0000","last_clean":"2023-11-28T18:23:08.114651+0000","last_became_active":"2023-11-28T18:23:08.112097+0000","last_became_peered":"2023-11-28T18:23:08.112097+0000","last_unstale":"2023-11-28T18:23:08.114651+0000","last_undegraded":"2023-11-28T18:23:08.114651+0000","last_fullsized":"2023-11-28T18:23:08.114651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:03:43.223257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.124291+0000","last_change":"2023-11-28T18:23:07.684818+0000","last_active":"2023-11-28T18:23:08.124291+0000","last_peered":"2023-11-28T18:23:08.124291+0000","last_clean":"2023-11-28T18:23:08.124291+0000","last_became_active":"2023-11-28T18:23:07.684502+0000","last_became_peered":"2023-11-28T18:23:07.684502+0000","last_unstale":"2023-11-28T18:23:08.124291+0000","last_undegraded":"2023-11-28T18:23:08.124291+0000","last_fullsized":"2023-11-28T18:23:08.124291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:16:21.223266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109527+0000","last_change":"2023-11-28T18:23:08.133972+0000","last_active":"2023-11-28T18:23:09.109527+0000","last_peered":"2023-11-28T18:23:09.109527+0000","last_clean":"2023-11-28T18:23:09.109527+0000","last_became_active":"2023-11-28T18:23:08.133747+0000","last_became_peered":"2023-11-28T18:23:08.133747+0000","last_unstale":"2023-11-28T18:23:09.109527+0000","last_undegraded":"2023-11-28T18:23:09.109527+0000","last_fullsized":"2023-11-28T18:23:09.109527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:24.968241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113538+0000","last_change":"2023-11-28T18:23:08.134117+0000","last_active":"2023-11-28T18:23:09.113538+0000","last_peered":"2023-11-28T18:23:09.113538+0000","last_clean":"2023-11-28T18:23:09.113538+0000","last_became_active":"2023-11-28T18:23:08.133801+0000","last_became_peered":"2023-11-28T18:23:08.133801+0000","last_unstale":"2023-11-28T18:23:09.113538+0000","last_undegraded":"2023-11-28T18:23:09.113538+0000","last_fullsized":"2023-11-28T18:23:09.113538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:02.370862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117018+0000","last_change":"2023-11-28T18:23:07.686399+0000","last_active":"2023-11-28T18:23:08.117018+0000","last_peered":"2023-11-28T18:23:08.117018+0000","last_clean":"2023-11-28T18:23:08.117018+0000","last_became_active":"2023-11-28T18:23:07.686051+0000","last_became_peered":"2023-11-28T18:23:07.686051+0000","last_unstale":"2023-11-28T18:23:08.117018+0000","last_undegraded":"2023-11-28T18:23:08.117018+0000","last_fullsized":"2023-11-28T18:23:08.117018+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:11.797516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113317+0000","last_change":"2023-11-28T18:23:08.134870+0000","last_active":"2023-11-28T18:23:09.113317+0000","last_peered":"2023-11-28T18:23:09.113317+0000","last_clean":"2023-11-28T18:23:09.113317+0000","last_became_active":"2023-11-28T18:23:08.133732+0000","last_became_peered":"2023-11-28T18:23:08.133732+0000","last_unstale":"2023-11-28T18:23:09.113317+0000","last_undegraded":"2023-11-28T18:23:09.113317+0000","last_fullsized":"2023-11-28T18:23:09.113317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:01.364242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.120013+0000","last_change":"2023-11-28T18:23:07.682230+0000","last_active":"2023-11-28T18:23:08.120013+0000","last_peered":"2023-11-28T18:23:08.120013+0000","last_clean":"2023-11-28T18:23:08.120013+0000","last_became_active":"2023-11-28T18:23:07.681715+0000","last_became_peered":"2023-11-28T18:23:07.681715+0000","last_unstale":"2023-11-28T18:23:08.120013+0000","last_undegraded":"2023-11-28T18:23:08.120013+0000","last_fullsized":"2023-11-28T18:23:08.120013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:02.343996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131084+0000","last_change":"2023-11-28T18:23:08.128927+0000","last_active":"2023-11-28T18:23:08.131084+0000","last_peered":"2023-11-28T18:23:08.131084+0000","last_clean":"2023-11-28T18:23:08.131084+0000","last_became_active":"2023-11-28T18:23:08.127775+0000","last_became_peered":"2023-11-28T18:23:08.127775+0000","last_unstale":"2023-11-28T18:23:08.131084+0000","last_undegraded":"2023-11-28T18:23:08.131084+0000","last_fullsized":"2023-11-28T18:23:08.131084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:03.993918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.109224+0000","last_change":"2023-11-28T18:23:07.667929+0000","last_active":"2023-11-28T18:23:08.109224+0000","last_peered":"2023-11-28T18:23:08.109224+0000","last_clean":"2023-11-28T18:23:08.109224+0000","last_became_active":"2023-11-28T18:23:07.667666+0000","last_became_peered":"2023-11-28T18:23:07.667666+0000","last_unstale":"2023-11-28T18:23:08.109224+0000","last_undegraded":"2023-11-28T18:23:08.109224+0000","last_fullsized":"2023-11-28T18:23:08.109224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:32.132092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131083+0000","last_change":"2023-11-28T18:23:08.126404+0000","last_active":"2023-11-28T18:23:08.131083+0000","last_peered":"2023-11-28T18:23:08.131083+0000","last_clean":"2023-11-28T18:23:08.131083+0000","last_became_active":"2023-11-28T18:23:08.125315+0000","last_became_peered":"2023-11-28T18:23:08.125315+0000","last_unstale":"2023-11-28T18:23:08.131083+0000","last_undegraded":"2023-11-28T18:23:08.131083+0000","last_fullsized":"2023-11-28T18:23:08.131083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:44.799540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.104198+0000","last_change":"2023-11-28T18:23:07.677758+0000","last_active":"2023-11-28T18:23:08.104198+0000","last_peered":"2023-11-28T18:23:08.104198+0000","last_clean":"2023-11-28T18:23:08.104198+0000","last_became_active":"2023-11-28T18:23:07.676902+0000","last_became_peered":"2023-11-28T18:23:07.676902+0000","last_unstale":"2023-11-28T18:23:08.104198+0000","last_undegraded":"2023-11-28T18:23:08.104198+0000","last_fullsized":"2023-11-28T18:23:08.104198+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:50.953647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":654978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":654978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T18:23:14.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T18:23:14.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4820fd1640 1 Processor -- start 2023-11-28T18:23:14.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4820fd1640 1 -- start start 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4820fd1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c151cd0 0x7f481c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4820fd1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f481c06e560 0x7f481c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4820fd1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c071e60 con 0x7f481c0701f0 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4820fd1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481c070930 con 0x7f481c06e560 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4820fd1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c070660 con 0x7f481c151cd0 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4819d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f481c06e560 0x7f481c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4819d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f481c06e560 0x7f481c157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59322/0 (socket says 172.21.15.7:59322) 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4819d9b640 1 -- 172.21.15.7:0/366703846 learned_addr learned my addr 172.21.15.7:0/366703846 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f481a59c640 1 --2- 172.21.15.7:0/366703846 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c151cd0 0x7f481c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4819d9b640 1 -- 172.21.15.7:0/366703846 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c151cd0 msgr2=0x7f481c15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4819d9b640 1 --2- 172.21.15.7:0/366703846 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c151cd0 0x7f481c15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4819d9b640 1 -- 172.21.15.7:0/366703846 >> v1:172.21.15.193:6789/0 conn(0x7f481c0701f0 legacy=0x7f481c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.964+0000 7f4819d9b640 1 -- 172.21.15.7:0/366703846 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c070bf0 con 0x7f481c06e560 2023-11-28T18:23:14.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4819d9b640 1 --2- 172.21.15.7:0/366703846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c06e560 0x7f481c157b90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4810009870 tx=0x7f481001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92db40b0257d250e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f481959a640 1 -- 172.21.15.7:0/366703846 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100ac600 con 0x7f481c06e560 2023-11-28T18:23:14.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f481959a640 1 -- 172.21.15.7:0/366703846 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48100a8050 con 0x7f481c06e560 2023-11-28T18:23:14.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f481959a640 1 -- 172.21.15.7:0/366703846 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100ac980 con 0x7f481c06e560 2023-11-28T18:23:14.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4820fd1640 1 -- 172.21.15.7:0/366703846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c06e560 msgr2=0x7f481c157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4820fd1640 1 --2- 172.21.15.7:0/366703846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c06e560 0x7f481c157b90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4810009870 tx=0x7f481001c4b0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4820fd1640 1 -- 172.21.15.7:0/366703846 shutdown_connections 2023-11-28T18:23:14.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4820fd1640 1 --2- 172.21.15.7:0/366703846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c06e560 0x7f481c157b90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4820fd1640 1 --2- 172.21.15.7:0/366703846 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c151cd0 0x7f481c15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4820fd1640 1 -- 172.21.15.7:0/366703846 >> 172.21.15.7:0/366703846 conn(0x7f481c0662e0 msgr2=0x7f481c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4820fd1640 1 -- 172.21.15.7:0/366703846 shutdown_connections 2023-11-28T18:23:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.968+0000 7f4820fd1640 1 -- 172.21.15.7:0/366703846 wait complete. 2023-11-28T18:23:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 Processor -- start 2023-11-28T18:23:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 -- start start 2023-11-28T18:23:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f481c06e560 0x7f481c1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c0701f0 0x7f481c157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c151cd0 0x7f481c152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c157030 con 0x7f481c151cd0 2023-11-28T18:23:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c157310 con 0x7f481c06e560 2023-11-28T18:23:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c1f4050 con 0x7f481c0701f0 2023-11-28T18:23:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c151cd0 0x7f481c152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c151cd0 0x7f481c152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59338/0 (socket says 172.21.15.7:59338) 2023-11-28T18:23:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 -- 172.21.15.7:0/420428342 learned_addr learned my addr 172.21.15.7:0/420428342 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4819d9b640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c0701f0 0x7f481c157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481a59c640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f481c06e560 0x7f481c1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 -- 172.21.15.7:0/420428342 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c0701f0 msgr2=0x7f481c157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c0701f0 0x7f481c157930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 -- 172.21.15.7:0/420428342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f481c06e560 msgr2=0x7f481c1566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f481c06e560 0x7f481c1566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 -- 172.21.15.7:0/420428342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48100094d0 con 0x7f481c151cd0 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f481ad9d640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c151cd0 0x7f481c152440 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f481400c770 tx=0x7f481400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f480affd640 1 -- 172.21.15.7:0/420428342 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814003b80 con 0x7f481c151cd0 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f480affd640 1 -- 172.21.15.7:0/420428342 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4814003ce0 con 0x7f481c151cd0 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c1f4330 con 0x7f481c151cd0 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f480affd640 1 -- 172.21.15.7:0/420428342 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481400aa90 con 0x7f481c151cd0 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.972+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c1f48f0 con 0x7f481c151cd0 2023-11-28T18:23:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.976+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c151820 con 0x7f481c151cd0 2023-11-28T18:23:14.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.976+0000 7f480affd640 1 -- 172.21.15.7:0/420428342 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f481400abf0 con 0x7f481c151cd0 2023-11-28T18:23:14.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.976+0000 7f480affd640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47fc0668d0 0x7f47fc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:14.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.976+0000 7f480affd640 1 -- 172.21.15.7:0/420428342 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4814084250 con 0x7f481c151cd0 2023-11-28T18:23:14.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.976+0000 7f481a59c640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47fc0668d0 0x7f47fc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:14.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.976+0000 7f481a59c640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47fc0668d0 0x7f47fc068d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f480c009ac0 tx=0x7f480c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:14.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:14.980+0000 7f480affd640 1 -- 172.21.15.7:0/420428342 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4814051420 con 0x7f481c151cd0 2023-11-28T18:23:15.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.136+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 --> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f481c151490 con 0x7f47fc0668d0 2023-11-28T18:23:15.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.140+0000 7f480affd640 1 -- 172.21.15.7:0/420428342 <== mgr.4102 v2:172.21.15.7:6832/264069339 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264530 (secure 0 0 0) 0x7f481c151490 con 0x7f47fc0668d0 2023-11-28T18:23:15.149 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:15.151 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47fc0668d0 msgr2=0x7f47fc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47fc0668d0 0x7f47fc068d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f480c009ac0 tx=0x7f480c009190 comp rx=0 tx=0).stop 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c151cd0 msgr2=0x7f481c152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c151cd0 0x7f481c152440 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f481400c770 tx=0x7f481400cc30 comp rx=0 tx=0).stop 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 shutdown_connections 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47fc0668d0 0x7f47fc068d90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f481c151cd0 0x7f481c152440 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f481c0701f0 0x7f481c157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 --2- 172.21.15.7:0/420428342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f481c06e560 0x7f481c1566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 >> 172.21.15.7:0/420428342 conn(0x7f481c0662e0 msgr2=0x7f481c14b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:15.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 shutdown_connections 2023-11-28T18:23:15.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.144+0000 7f4820fd1640 1 -- 172.21.15.7:0/420428342 wait complete. 2023-11-28T18:23:15.170 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-11-28T18:23:13.462317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235480,"kb_used_data":2664,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739240,"statfs":{"total":773094113280,"available":772852981760,"internally_reserved":0,"allocated":2727936,"data_stored":1697616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001423"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142633+0000","last_change":"2023-11-28T18:23:08.134633+0000","last_active":"2023-11-28T18:23:08.142633+0000","last_peered":"2023-11-28T18:23:08.142633+0000","last_clean":"2023-11-28T18:23:08.142633+0000","last_became_active":"2023-11-28T18:23:08.134320+0000","last_became_peered":"2023-11-28T18:23:08.134320+0000","last_unstale":"2023-11-28T18:23:08.142633+0000","last_undegraded":"2023-11-28T18:23:08.142633+0000","last_fullsized":"2023-11-28T18:23:08.142633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:07:32.049802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105857+0000","last_change":"2023-11-28T18:23:08.122460+0000","last_active":"2023-11-28T18:23:09.105857+0000","last_peered":"2023-11-28T18:23:09.105857+0000","last_clean":"2023-11-28T18:23:09.105857+0000","last_became_active":"2023-11-28T18:23:08.122153+0000","last_became_peered":"2023-11-28T18:23:08.122153+0000","last_unstale":"2023-11-28T18:23:09.105857+0000","last_undegraded":"2023-11-28T18:23:09.105857+0000","last_fullsized":"2023-11-28T18:23:09.105857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:17:31.074978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.113129+0000","last_change":"2023-11-28T18:23:08.110187+0000","last_active":"2023-11-28T18:23:08.113129+0000","last_peered":"2023-11-28T18:23:08.113129+0000","last_clean":"2023-11-28T18:23:08.113129+0000","last_became_active":"2023-11-28T18:23:08.109915+0000","last_became_peered":"2023-11-28T18:23:08.109915+0000","last_unstale":"2023-11-28T18:23:08.113129+0000","last_undegraded":"2023-11-28T18:23:08.113129+0000","last_fullsized":"2023-11-28T18:23:08.113129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:20.219583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113335+0000","last_change":"2023-11-28T18:23:08.122000+0000","last_active":"2023-11-28T18:23:09.113335+0000","last_peered":"2023-11-28T18:23:09.113335+0000","last_clean":"2023-11-28T18:23:09.113335+0000","last_became_active":"2023-11-28T18:23:08.121652+0000","last_became_peered":"2023-11-28T18:23:08.121652+0000","last_unstale":"2023-11-28T18:23:09.113335+0000","last_undegraded":"2023-11-28T18:23:09.113335+0000","last_fullsized":"2023-11-28T18:23:09.113335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:15.540606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.134612+0000","last_change":"2023-11-28T18:23:08.126587+0000","last_active":"2023-11-28T18:23:08.134612+0000","last_peered":"2023-11-28T18:23:08.134612+0000","last_clean":"2023-11-28T18:23:08.134612+0000","last_became_active":"2023-11-28T18:23:08.126313+0000","last_became_peered":"2023-11-28T18:23:08.126313+0000","last_unstale":"2023-11-28T18:23:08.134612+0000","last_undegraded":"2023-11-28T18:23:08.134612+0000","last_fullsized":"2023-11-28T18:23:08.134612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:38:58.943122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113040+0000","last_change":"2023-11-28T18:23:08.122878+0000","last_active":"2023-11-28T18:23:09.113040+0000","last_peered":"2023-11-28T18:23:09.113040+0000","last_clean":"2023-11-28T18:23:09.113040+0000","last_became_active":"2023-11-28T18:23:08.122609+0000","last_became_peered":"2023-11-28T18:23:08.122609+0000","last_unstale":"2023-11-28T18:23:09.113040+0000","last_undegraded":"2023-11-28T18:23:09.113040+0000","last_fullsized":"2023-11-28T18:23:09.113040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:00.610074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.139912+0000","last_change":"2023-11-28T18:23:08.133485+0000","last_active":"2023-11-28T18:23:08.139912+0000","last_peered":"2023-11-28T18:23:08.139912+0000","last_clean":"2023-11-28T18:23:08.139912+0000","last_became_active":"2023-11-28T18:23:08.133211+0000","last_became_peered":"2023-11-28T18:23:08.133211+0000","last_unstale":"2023-11-28T18:23:08.139912+0000","last_undegraded":"2023-11-28T18:23:08.139912+0000","last_fullsized":"2023-11-28T18:23:08.139912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:57:22.781432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.126558+0000","last_change":"2023-11-28T18:23:08.117903+0000","last_active":"2023-11-28T18:23:08.126558+0000","last_peered":"2023-11-28T18:23:08.126558+0000","last_clean":"2023-11-28T18:23:08.126558+0000","last_became_active":"2023-11-28T18:23:08.117482+0000","last_became_peered":"2023-11-28T18:23:08.117482+0000","last_unstale":"2023-11-28T18:23:08.126558+0000","last_undegraded":"2023-11-28T18:23:08.126558+0000","last_fullsized":"2023-11-28T18:23:08.126558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:39:20.437743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142696+0000","last_change":"2023-11-28T18:23:08.135606+0000","last_active":"2023-11-28T18:23:08.142696+0000","last_peered":"2023-11-28T18:23:08.142696+0000","last_clean":"2023-11-28T18:23:08.142696+0000","last_became_active":"2023-11-28T18:23:08.135328+0000","last_became_peered":"2023-11-28T18:23:08.135328+0000","last_unstale":"2023-11-28T18:23:08.142696+0000","last_undegraded":"2023-11-28T18:23:08.142696+0000","last_fullsized":"2023-11-28T18:23:08.142696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:51:22.378042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110046+0000","last_change":"2023-11-28T18:23:08.123993+0000","last_active":"2023-11-28T18:23:09.110046+0000","last_peered":"2023-11-28T18:23:09.110046+0000","last_clean":"2023-11-28T18:23:09.110046+0000","last_became_active":"2023-11-28T18:23:08.123583+0000","last_became_peered":"2023-11-28T18:23:08.123583+0000","last_unstale":"2023-11-28T18:23:09.110046+0000","last_undegraded":"2023-11-28T18:23:09.110046+0000","last_fullsized":"2023-11-28T18:23:09.110046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:16:33.089307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:08.120983+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:08.120486+0000","last_became_peered":"2023-11-28T18:23:08.120486+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:46:12.881629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.139158+0000","last_change":"2023-11-28T18:23:08.130169+0000","last_active":"2023-11-28T18:23:08.139158+0000","last_peered":"2023-11-28T18:23:08.139158+0000","last_clean":"2023-11-28T18:23:08.139158+0000","last_became_active":"2023-11-28T18:23:08.129771+0000","last_became_peered":"2023-11-28T18:23:08.129771+0000","last_unstale":"2023-11-28T18:23:08.139158+0000","last_undegraded":"2023-11-28T18:23:08.139158+0000","last_fullsized":"2023-11-28T18:23:08.139158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:02:13.375041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.130031+0000","last_change":"2023-11-28T18:23:08.123268+0000","last_active":"2023-11-28T18:23:08.130031+0000","last_peered":"2023-11-28T18:23:08.130031+0000","last_clean":"2023-11-28T18:23:08.130031+0000","last_became_active":"2023-11-28T18:23:08.122978+0000","last_became_peered":"2023-11-28T18:23:08.122978+0000","last_unstale":"2023-11-28T18:23:08.130031+0000","last_undegraded":"2023-11-28T18:23:08.130031+0000","last_fullsized":"2023-11-28T18:23:08.130031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:09:25.753235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.124886+0000","last_change":"2023-11-28T18:23:08.118361+0000","last_active":"2023-11-28T18:23:08.124886+0000","last_peered":"2023-11-28T18:23:08.124886+0000","last_clean":"2023-11-28T18:23:08.124886+0000","last_became_active":"2023-11-28T18:23:08.117973+0000","last_became_peered":"2023-11-28T18:23:08.117973+0000","last_unstale":"2023-11-28T18:23:08.124886+0000","last_undegraded":"2023-11-28T18:23:08.124886+0000","last_fullsized":"2023-11-28T18:23:08.124886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:07.540205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105530+0000","last_change":"2023-11-28T18:23:08.139744+0000","last_active":"2023-11-28T18:23:09.105530+0000","last_peered":"2023-11-28T18:23:09.105530+0000","last_clean":"2023-11-28T18:23:09.105530+0000","last_became_active":"2023-11-28T18:23:08.139464+0000","last_became_peered":"2023-11-28T18:23:08.139464+0000","last_unstale":"2023-11-28T18:23:09.105530+0000","last_undegraded":"2023-11-28T18:23:09.105530+0000","last_fullsized":"2023-11-28T18:23:09.105530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:19:34.780979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113036+0000","last_change":"2023-11-28T18:23:08.119103+0000","last_active":"2023-11-28T18:23:09.113036+0000","last_peered":"2023-11-28T18:23:09.113036+0000","last_clean":"2023-11-28T18:23:09.113036+0000","last_became_active":"2023-11-28T18:23:08.116080+0000","last_became_peered":"2023-11-28T18:23:08.116080+0000","last_unstale":"2023-11-28T18:23:09.113036+0000","last_undegraded":"2023-11-28T18:23:09.113036+0000","last_fullsized":"2023-11-28T18:23:09.113036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:44.279413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113688+0000","last_change":"2023-11-28T18:23:08.126817+0000","last_active":"2023-11-28T18:23:09.113688+0000","last_peered":"2023-11-28T18:23:09.113688+0000","last_clean":"2023-11-28T18:23:09.113688+0000","last_became_active":"2023-11-28T18:23:08.126540+0000","last_became_peered":"2023-11-28T18:23:08.126540+0000","last_unstale":"2023-11-28T18:23:09.113688+0000","last_undegraded":"2023-11-28T18:23:09.113688+0000","last_fullsized":"2023-11-28T18:23:09.113688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:08.807847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.124678+0000","last_change":"2023-11-28T18:23:08.121775+0000","last_active":"2023-11-28T18:23:08.124678+0000","last_peered":"2023-11-28T18:23:08.124678+0000","last_clean":"2023-11-28T18:23:08.124678+0000","last_became_active":"2023-11-28T18:23:08.121466+0000","last_became_peered":"2023-11-28T18:23:08.121466+0000","last_unstale":"2023-11-28T18:23:08.124678+0000","last_undegraded":"2023-11-28T18:23:08.124678+0000","last_fullsized":"2023-11-28T18:23:08.124678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:58.099267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.143459+0000","last_change":"2023-11-28T18:23:08.138282+0000","last_active":"2023-11-28T18:23:08.143459+0000","last_peered":"2023-11-28T18:23:08.143459+0000","last_clean":"2023-11-28T18:23:08.143459+0000","last_became_active":"2023-11-28T18:23:08.137171+0000","last_became_peered":"2023-11-28T18:23:08.137171+0000","last_unstale":"2023-11-28T18:23:08.143459+0000","last_undegraded":"2023-11-28T18:23:08.143459+0000","last_fullsized":"2023-11-28T18:23:08.143459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:30:18.669500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110124+0000","last_change":"2023-11-28T18:23:08.133521+0000","last_active":"2023-11-28T18:23:09.110124+0000","last_peered":"2023-11-28T18:23:09.110124+0000","last_clean":"2023-11-28T18:23:09.110124+0000","last_became_active":"2023-11-28T18:23:08.133248+0000","last_became_peered":"2023-11-28T18:23:08.133248+0000","last_unstale":"2023-11-28T18:23:09.110124+0000","last_undegraded":"2023-11-28T18:23:09.110124+0000","last_fullsized":"2023-11-28T18:23:09.110124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:17:34.585808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131127+0000","last_change":"2023-11-28T18:23:08.129150+0000","last_active":"2023-11-28T18:23:08.131127+0000","last_peered":"2023-11-28T18:23:08.131127+0000","last_clean":"2023-11-28T18:23:08.131127+0000","last_became_active":"2023-11-28T18:23:08.128622+0000","last_became_peered":"2023-11-28T18:23:08.128622+0000","last_unstale":"2023-11-28T18:23:08.131127+0000","last_undegraded":"2023-11-28T18:23:08.131127+0000","last_fullsized":"2023-11-28T18:23:08.131127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:56.050282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113274+0000","last_change":"2023-11-28T18:23:08.131333+0000","last_active":"2023-11-28T18:23:09.113274+0000","last_peered":"2023-11-28T18:23:09.113274+0000","last_clean":"2023-11-28T18:23:09.113274+0000","last_became_active":"2023-11-28T18:23:08.130917+0000","last_became_peered":"2023-11-28T18:23:08.130917+0000","last_unstale":"2023-11-28T18:23:09.113274+0000","last_undegraded":"2023-11-28T18:23:09.113274+0000","last_fullsized":"2023-11-28T18:23:09.113274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:04:48.114417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.133526+0000","last_change":"2023-11-28T18:23:08.132355+0000","last_active":"2023-11-28T18:23:08.133526+0000","last_peered":"2023-11-28T18:23:08.133526+0000","last_clean":"2023-11-28T18:23:08.133526+0000","last_became_active":"2023-11-28T18:23:08.132016+0000","last_became_peered":"2023-11-28T18:23:08.132016+0000","last_unstale":"2023-11-28T18:23:08.133526+0000","last_undegraded":"2023-11-28T18:23:08.133526+0000","last_fullsized":"2023-11-28T18:23:08.133526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:31:32.314253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113066+0000","last_change":"2023-11-28T18:23:08.119935+0000","last_active":"2023-11-28T18:23:09.113066+0000","last_peered":"2023-11-28T18:23:09.113066+0000","last_clean":"2023-11-28T18:23:09.113066+0000","last_became_active":"2023-11-28T18:23:08.118372+0000","last_became_peered":"2023-11-28T18:23:08.118372+0000","last_unstale":"2023-11-28T18:23:09.113066+0000","last_undegraded":"2023-11-28T18:23:09.113066+0000","last_fullsized":"2023-11-28T18:23:09.113066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:52.339024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.133518+0000","last_change":"2023-11-28T18:23:08.132272+0000","last_active":"2023-11-28T18:23:08.133518+0000","last_peered":"2023-11-28T18:23:08.133518+0000","last_clean":"2023-11-28T18:23:08.133518+0000","last_became_active":"2023-11-28T18:23:08.131985+0000","last_became_peered":"2023-11-28T18:23:08.131985+0000","last_unstale":"2023-11-28T18:23:08.133518+0000","last_undegraded":"2023-11-28T18:23:08.133518+0000","last_fullsized":"2023-11-28T18:23:08.133518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:00:03.297320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110084+0000","last_change":"2023-11-28T18:23:08.127816+0000","last_active":"2023-11-28T18:23:09.110084+0000","last_peered":"2023-11-28T18:23:09.110084+0000","last_clean":"2023-11-28T18:23:09.110084+0000","last_became_active":"2023-11-28T18:23:08.127380+0000","last_became_peered":"2023-11-28T18:23:08.127380+0000","last_unstale":"2023-11-28T18:23:09.110084+0000","last_undegraded":"2023-11-28T18:23:09.110084+0000","last_fullsized":"2023-11-28T18:23:09.110084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:31.242361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105706+0000","last_change":"2023-11-28T18:23:08.128188+0000","last_active":"2023-11-28T18:23:09.105706+0000","last_peered":"2023-11-28T18:23:09.105706+0000","last_clean":"2023-11-28T18:23:09.105706+0000","last_became_active":"2023-11-28T18:23:08.127860+0000","last_became_peered":"2023-11-28T18:23:08.127860+0000","last_unstale":"2023-11-28T18:23:09.105706+0000","last_undegraded":"2023-11-28T18:23:09.105706+0000","last_fullsized":"2023-11-28T18:23:09.105706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:12.095768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.133547+0000","last_change":"2023-11-28T18:23:08.132381+0000","last_active":"2023-11-28T18:23:08.133547+0000","last_peered":"2023-11-28T18:23:08.133547+0000","last_clean":"2023-11-28T18:23:08.133547+0000","last_became_active":"2023-11-28T18:23:08.132072+0000","last_became_peered":"2023-11-28T18:23:08.132072+0000","last_unstale":"2023-11-28T18:23:08.133547+0000","last_undegraded":"2023-11-28T18:23:08.133547+0000","last_fullsized":"2023-11-28T18:23:08.133547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:27:13.360596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117010+0000","last_change":"2023-11-28T18:23:08.115379+0000","last_active":"2023-11-28T18:23:08.117010+0000","last_peered":"2023-11-28T18:23:08.117010+0000","last_clean":"2023-11-28T18:23:08.117010+0000","last_became_active":"2023-11-28T18:23:08.115236+0000","last_became_peered":"2023-11-28T18:23:08.115236+0000","last_unstale":"2023-11-28T18:23:08.117010+0000","last_undegraded":"2023-11-28T18:23:08.117010+0000","last_fullsized":"2023-11-28T18:23:08.117010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:27.176600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113293+0000","last_change":"2023-11-28T18:23:08.121402+0000","last_active":"2023-11-28T18:23:09.113293+0000","last_peered":"2023-11-28T18:23:09.113293+0000","last_clean":"2023-11-28T18:23:09.113293+0000","last_became_active":"2023-11-28T18:23:08.121129+0000","last_became_peered":"2023-11-28T18:23:08.121129+0000","last_unstale":"2023-11-28T18:23:09.113293+0000","last_undegraded":"2023-11-28T18:23:09.113293+0000","last_fullsized":"2023-11-28T18:23:09.113293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:02.927891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.124746+0000","last_change":"2023-11-28T18:23:08.121594+0000","last_active":"2023-11-28T18:23:08.124746+0000","last_peered":"2023-11-28T18:23:08.124746+0000","last_clean":"2023-11-28T18:23:08.124746+0000","last_became_active":"2023-11-28T18:23:08.119009+0000","last_became_peered":"2023-11-28T18:23:08.119009+0000","last_unstale":"2023-11-28T18:23:08.124746+0000","last_undegraded":"2023-11-28T18:23:08.124746+0000","last_fullsized":"2023-11-28T18:23:08.124746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:03.581437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109676+0000","last_change":"2023-11-28T18:23:08.115403+0000","last_active":"2023-11-28T18:23:09.109676+0000","last_peered":"2023-11-28T18:23:09.109676+0000","last_clean":"2023-11-28T18:23:09.109676+0000","last_became_active":"2023-11-28T18:23:08.114955+0000","last_became_peered":"2023-11-28T18:23:08.114955+0000","last_unstale":"2023-11-28T18:23:09.109676+0000","last_undegraded":"2023-11-28T18:23:09.109676+0000","last_fullsized":"2023-11-28T18:23:09.109676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:41.242830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.138939+0000","last_change":"2023-11-28T18:23:08.129745+0000","last_active":"2023-11-28T18:23:08.138939+0000","last_peered":"2023-11-28T18:23:08.138939+0000","last_clean":"2023-11-28T18:23:08.138939+0000","last_became_active":"2023-11-28T18:23:08.129388+0000","last_became_peered":"2023-11-28T18:23:08.129388+0000","last_unstale":"2023-11-28T18:23:08.138939+0000","last_undegraded":"2023-11-28T18:23:08.138939+0000","last_fullsized":"2023-11-28T18:23:08.138939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:44:10.418671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.109812+0000","last_change":"2023-11-28T18:23:07.664773+0000","last_active":"2023-11-28T18:23:08.109812+0000","last_peered":"2023-11-28T18:23:08.109812+0000","last_clean":"2023-11-28T18:23:08.109812+0000","last_became_active":"2023-11-28T18:23:07.664478+0000","last_became_peered":"2023-11-28T18:23:07.664478+0000","last_unstale":"2023-11-28T18:23:08.109812+0000","last_undegraded":"2023-11-28T18:23:08.109812+0000","last_fullsized":"2023-11-28T18:23:08.109812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:28.322454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131144+0000","last_change":"2023-11-28T18:23:08.129441+0000","last_active":"2023-11-28T18:23:08.131144+0000","last_peered":"2023-11-28T18:23:08.131144+0000","last_clean":"2023-11-28T18:23:08.131144+0000","last_became_active":"2023-11-28T18:23:08.129143+0000","last_became_peered":"2023-11-28T18:23:08.129143+0000","last_unstale":"2023-11-28T18:23:08.131144+0000","last_undegraded":"2023-11-28T18:23:08.131144+0000","last_fullsized":"2023-11-28T18:23:08.131144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:01:48.849011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109569+0000","last_change":"2023-11-28T18:23:07.668192+0000","last_active":"2023-11-28T18:23:09.109569+0000","last_peered":"2023-11-28T18:23:09.109569+0000","last_clean":"2023-11-28T18:23:09.109569+0000","last_became_active":"2023-11-28T18:23:07.667780+0000","last_became_peered":"2023-11-28T18:23:07.667780+0000","last_unstale":"2023-11-28T18:23:09.109569+0000","last_undegraded":"2023-11-28T18:23:09.109569+0000","last_fullsized":"2023-11-28T18:23:09.109569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.109926+0000","last_change":"2023-11-28T18:23:07.665608+0000","last_active":"2023-11-28T18:23:08.109926+0000","last_peered":"2023-11-28T18:23:08.109926+0000","last_clean":"2023-11-28T18:23:08.109926+0000","last_became_active":"2023-11-28T18:23:07.665361+0000","last_became_peered":"2023-11-28T18:23:07.665361+0000","last_unstale":"2023-11-28T18:23:08.109926+0000","last_undegraded":"2023-11-28T18:23:08.109926+0000","last_fullsized":"2023-11-28T18:23:08.109926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:17.279597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.126580+0000","last_change":"2023-11-28T18:23:08.118091+0000","last_active":"2023-11-28T18:23:08.126580+0000","last_peered":"2023-11-28T18:23:08.126580+0000","last_clean":"2023-11-28T18:23:08.126580+0000","last_became_active":"2023-11-28T18:23:08.117524+0000","last_became_peered":"2023-11-28T18:23:08.117524+0000","last_unstale":"2023-11-28T18:23:08.126580+0000","last_undegraded":"2023-11-28T18:23:08.126580+0000","last_fullsized":"2023-11-28T18:23:08.126580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:19:55.437131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113169+0000","last_change":"2023-11-28T18:23:07.667847+0000","last_active":"2023-11-28T18:23:09.113169+0000","last_peered":"2023-11-28T18:23:09.113169+0000","last_clean":"2023-11-28T18:23:09.113169+0000","last_became_active":"2023-11-28T18:23:07.667466+0000","last_became_peered":"2023-11-28T18:23:07.667466+0000","last_unstale":"2023-11-28T18:23:09.113169+0000","last_undegraded":"2023-11-28T18:23:09.113169+0000","last_fullsized":"2023-11-28T18:23:09.113169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:06.807146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.127497+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.127216+0000","last_became_peered":"2023-11-28T18:23:08.127216+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:03:14.999484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109883+0000","last_change":"2023-11-28T18:23:07.681698+0000","last_active":"2023-11-28T18:23:09.109883+0000","last_peered":"2023-11-28T18:23:09.109883+0000","last_clean":"2023-11-28T18:23:09.109883+0000","last_became_active":"2023-11-28T18:23:07.681365+0000","last_became_peered":"2023-11-28T18:23:07.681365+0000","last_unstale":"2023-11-28T18:23:09.109883+0000","last_undegraded":"2023-11-28T18:23:09.109883+0000","last_fullsized":"2023-11-28T18:23:09.109883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:21.985029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113361+0000","last_change":"2023-11-28T18:23:08.132797+0000","last_active":"2023-11-28T18:23:09.113361+0000","last_peered":"2023-11-28T18:23:09.113361+0000","last_clean":"2023-11-28T18:23:09.113361+0000","last_became_active":"2023-11-28T18:23:08.132525+0000","last_became_peered":"2023-11-28T18:23:08.132525+0000","last_unstale":"2023-11-28T18:23:09.113361+0000","last_undegraded":"2023-11-28T18:23:09.113361+0000","last_fullsized":"2023-11-28T18:23:09.113361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:12:10.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113468+0000","last_change":"2023-11-28T18:23:07.672337+0000","last_active":"2023-11-28T18:23:09.113468+0000","last_peered":"2023-11-28T18:23:09.113468+0000","last_clean":"2023-11-28T18:23:09.113468+0000","last_became_active":"2023-11-28T18:23:07.671922+0000","last_became_peered":"2023-11-28T18:23:07.671922+0000","last_unstale":"2023-11-28T18:23:09.113468+0000","last_undegraded":"2023-11-28T18:23:09.113468+0000","last_fullsized":"2023-11-28T18:23:09.113468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:22.936770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.130231+0000","last_change":"2023-11-28T18:23:08.127919+0000","last_active":"2023-11-28T18:23:08.130231+0000","last_peered":"2023-11-28T18:23:08.130231+0000","last_clean":"2023-11-28T18:23:08.130231+0000","last_became_active":"2023-11-28T18:23:08.127602+0000","last_became_peered":"2023-11-28T18:23:08.127602+0000","last_unstale":"2023-11-28T18:23:08.130231+0000","last_undegraded":"2023-11-28T18:23:08.130231+0000","last_fullsized":"2023-11-28T18:23:08.130231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:09:42.431643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105765+0000","last_change":"2023-11-28T18:23:07.679219+0000","last_active":"2023-11-28T18:23:09.105765+0000","last_peered":"2023-11-28T18:23:09.105765+0000","last_clean":"2023-11-28T18:23:09.105765+0000","last_became_active":"2023-11-28T18:23:07.678920+0000","last_became_peered":"2023-11-28T18:23:07.678920+0000","last_unstale":"2023-11-28T18:23:09.105765+0000","last_undegraded":"2023-11-28T18:23:09.105765+0000","last_fullsized":"2023-11-28T18:23:09.105765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:02.332690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.140486+0000","last_change":"2023-11-28T18:23:08.132429+0000","last_active":"2023-11-28T18:23:08.140486+0000","last_peered":"2023-11-28T18:23:08.140486+0000","last_clean":"2023-11-28T18:23:08.140486+0000","last_became_active":"2023-11-28T18:23:08.132106+0000","last_became_peered":"2023-11-28T18:23:08.132106+0000","last_unstale":"2023-11-28T18:23:08.140486+0000","last_undegraded":"2023-11-28T18:23:08.140486+0000","last_fullsized":"2023-11-28T18:23:08.140486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:43:59.775353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:07.667479+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:07.665310+0000","last_became_peered":"2023-11-28T18:23:07.665310+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:03.174425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142790+0000","last_change":"2023-11-28T18:23:08.136812+0000","last_active":"2023-11-28T18:23:08.142790+0000","last_peered":"2023-11-28T18:23:08.142790+0000","last_clean":"2023-11-28T18:23:08.142790+0000","last_became_active":"2023-11-28T18:23:08.136525+0000","last_became_peered":"2023-11-28T18:23:08.136525+0000","last_unstale":"2023-11-28T18:23:08.142790+0000","last_undegraded":"2023-11-28T18:23:08.142790+0000","last_fullsized":"2023-11-28T18:23:08.142790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:09:17.327330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.104827+0000","last_change":"2023-11-28T18:23:07.660519+0000","last_active":"2023-11-28T18:23:08.104827+0000","last_peered":"2023-11-28T18:23:08.104827+0000","last_clean":"2023-11-28T18:23:08.104827+0000","last_became_active":"2023-11-28T18:23:07.660205+0000","last_became_peered":"2023-11-28T18:23:07.660205+0000","last_unstale":"2023-11-28T18:23:08.104827+0000","last_undegraded":"2023-11-28T18:23:08.104827+0000","last_fullsized":"2023-11-28T18:23:08.104827+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:04:54.125737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.130234+0000","last_change":"2023-11-28T18:23:08.127921+0000","last_active":"2023-11-28T18:23:08.130234+0000","last_peered":"2023-11-28T18:23:08.130234+0000","last_clean":"2023-11-28T18:23:08.130234+0000","last_became_active":"2023-11-28T18:23:08.127678+0000","last_became_peered":"2023-11-28T18:23:08.127678+0000","last_unstale":"2023-11-28T18:23:08.130234+0000","last_undegraded":"2023-11-28T18:23:08.130234+0000","last_fullsized":"2023-11-28T18:23:08.130234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:56:02.281565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113417+0000","last_change":"2023-11-28T18:23:07.671589+0000","last_active":"2023-11-28T18:23:09.113417+0000","last_peered":"2023-11-28T18:23:09.113417+0000","last_clean":"2023-11-28T18:23:09.113417+0000","last_became_active":"2023-11-28T18:23:07.671291+0000","last_became_peered":"2023-11-28T18:23:07.671291+0000","last_unstale":"2023-11-28T18:23:09.113417+0000","last_undegraded":"2023-11-28T18:23:09.113417+0000","last_fullsized":"2023-11-28T18:23:09.113417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:23.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.143552+0000","last_change":"2023-11-28T18:23:08.141316+0000","last_active":"2023-11-28T18:23:08.143552+0000","last_peered":"2023-11-28T18:23:08.143552+0000","last_clean":"2023-11-28T18:23:08.143552+0000","last_became_active":"2023-11-28T18:23:08.141011+0000","last_became_peered":"2023-11-28T18:23:08.141011+0000","last_unstale":"2023-11-28T18:23:08.143552+0000","last_undegraded":"2023-11-28T18:23:08.143552+0000","last_fullsized":"2023-11-28T18:23:08.143552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:05:09.149956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109447+0000","last_change":"2023-11-28T18:23:07.671192+0000","last_active":"2023-11-28T18:23:09.109447+0000","last_peered":"2023-11-28T18:23:09.109447+0000","last_clean":"2023-11-28T18:23:09.109447+0000","last_became_active":"2023-11-28T18:23:07.670812+0000","last_became_peered":"2023-11-28T18:23:07.670812+0000","last_unstale":"2023-11-28T18:23:09.109447+0000","last_undegraded":"2023-11-28T18:23:09.109447+0000","last_fullsized":"2023-11-28T18:23:09.109447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:28.806061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109506+0000","last_change":"2023-11-28T18:23:08.115229+0000","last_active":"2023-11-28T18:23:09.109506+0000","last_peered":"2023-11-28T18:23:09.109506+0000","last_clean":"2023-11-28T18:23:09.109506+0000","last_became_active":"2023-11-28T18:23:08.114541+0000","last_became_peered":"2023-11-28T18:23:08.114541+0000","last_unstale":"2023-11-28T18:23:09.109506+0000","last_undegraded":"2023-11-28T18:23:09.109506+0000","last_fullsized":"2023-11-28T18:23:09.109506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:38.816731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.106715+0000","last_change":"2023-11-28T18:23:07.673554+0000","last_active":"2023-11-28T18:23:08.106715+0000","last_peered":"2023-11-28T18:23:08.106715+0000","last_clean":"2023-11-28T18:23:08.106715+0000","last_became_active":"2023-11-28T18:23:07.672923+0000","last_became_peered":"2023-11-28T18:23:07.672923+0000","last_unstale":"2023-11-28T18:23:08.106715+0000","last_undegraded":"2023-11-28T18:23:08.106715+0000","last_fullsized":"2023-11-28T18:23:08.106715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:49.805572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.143546+0000","last_change":"2023-11-28T18:23:08.141169+0000","last_active":"2023-11-28T18:23:08.143546+0000","last_peered":"2023-11-28T18:23:08.143546+0000","last_clean":"2023-11-28T18:23:08.143546+0000","last_became_active":"2023-11-28T18:23:08.140801+0000","last_became_peered":"2023-11-28T18:23:08.140801+0000","last_unstale":"2023-11-28T18:23:08.143546+0000","last_undegraded":"2023-11-28T18:23:08.143546+0000","last_fullsized":"2023-11-28T18:23:08.143546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:53:42.320999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.110940+0000","last_change":"2023-11-28T18:23:07.680094+0000","last_active":"2023-11-28T18:23:08.110940+0000","last_peered":"2023-11-28T18:23:08.110940+0000","last_clean":"2023-11-28T18:23:08.110940+0000","last_became_active":"2023-11-28T18:23:07.679801+0000","last_became_peered":"2023-11-28T18:23:07.679801+0000","last_unstale":"2023-11-28T18:23:08.110940+0000","last_undegraded":"2023-11-28T18:23:08.110940+0000","last_fullsized":"2023-11-28T18:23:08.110940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:02.916155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142832+0000","last_change":"2023-11-28T18:23:08.137825+0000","last_active":"2023-11-28T18:23:08.142832+0000","last_peered":"2023-11-28T18:23:08.142832+0000","last_clean":"2023-11-28T18:23:08.142832+0000","last_became_active":"2023-11-28T18:23:08.137499+0000","last_became_peered":"2023-11-28T18:23:08.137499+0000","last_unstale":"2023-11-28T18:23:08.142832+0000","last_undegraded":"2023-11-28T18:23:08.142832+0000","last_fullsized":"2023-11-28T18:23:08.142832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:54.482524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.106617+0000","last_change":"2023-11-28T18:23:07.670359+0000","last_active":"2023-11-28T18:23:08.106617+0000","last_peered":"2023-11-28T18:23:08.106617+0000","last_clean":"2023-11-28T18:23:08.106617+0000","last_became_active":"2023-11-28T18:23:07.669687+0000","last_became_peered":"2023-11-28T18:23:07.669687+0000","last_unstale":"2023-11-28T18:23:08.106617+0000","last_undegraded":"2023-11-28T18:23:08.106617+0000","last_fullsized":"2023-11-28T18:23:08.106617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:45.450623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.143538+0000","last_change":"2023-11-28T18:23:08.141300+0000","last_active":"2023-11-28T18:23:08.143538+0000","last_peered":"2023-11-28T18:23:08.143538+0000","last_clean":"2023-11-28T18:23:08.143538+0000","last_became_active":"2023-11-28T18:23:08.141037+0000","last_became_peered":"2023-11-28T18:23:08.141037+0000","last_unstale":"2023-11-28T18:23:08.143538+0000","last_undegraded":"2023-11-28T18:23:08.143538+0000","last_fullsized":"2023-11-28T18:23:08.143538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:19:41.709261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105793+0000","last_change":"2023-11-28T18:23:07.673086+0000","last_active":"2023-11-28T18:23:09.105793+0000","last_peered":"2023-11-28T18:23:09.105793+0000","last_clean":"2023-11-28T18:23:09.105793+0000","last_became_active":"2023-11-28T18:23:07.672773+0000","last_became_peered":"2023-11-28T18:23:07.672773+0000","last_unstale":"2023-11-28T18:23:09.105793+0000","last_undegraded":"2023-11-28T18:23:09.105793+0000","last_fullsized":"2023-11-28T18:23:09.105793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:34.962428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113586+0000","last_change":"2023-11-28T18:23:08.121987+0000","last_active":"2023-11-28T18:23:09.113586+0000","last_peered":"2023-11-28T18:23:09.113586+0000","last_clean":"2023-11-28T18:23:09.113586+0000","last_became_active":"2023-11-28T18:23:08.121279+0000","last_became_peered":"2023-11-28T18:23:08.121279+0000","last_unstale":"2023-11-28T18:23:09.113586+0000","last_undegraded":"2023-11-28T18:23:09.113586+0000","last_fullsized":"2023-11-28T18:23:09.113586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:47.973973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.106199+0000","last_change":"2023-11-28T18:23:07.681312+0000","last_active":"2023-11-28T18:23:08.106199+0000","last_peered":"2023-11-28T18:23:08.106199+0000","last_clean":"2023-11-28T18:23:08.106199+0000","last_became_active":"2023-11-28T18:23:07.680806+0000","last_became_peered":"2023-11-28T18:23:07.680806+0000","last_unstale":"2023-11-28T18:23:08.106199+0000","last_undegraded":"2023-11-28T18:23:08.106199+0000","last_fullsized":"2023-11-28T18:23:08.106199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:58:41.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.142828+0000","last_change":"2023-11-28T18:23:08.137874+0000","last_active":"2023-11-28T18:23:08.142828+0000","last_peered":"2023-11-28T18:23:08.142828+0000","last_clean":"2023-11-28T18:23:08.142828+0000","last_became_active":"2023-11-28T18:23:08.137104+0000","last_became_peered":"2023-11-28T18:23:08.137104+0000","last_unstale":"2023-11-28T18:23:08.142828+0000","last_undegraded":"2023-11-28T18:23:08.142828+0000","last_fullsized":"2023-11-28T18:23:08.142828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:28.565657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-11-28T18:23:07.667413+0000","last_change":"2023-11-28T18:22:56.774057+0000","last_active":"2023-11-28T18:23:07.667413+0000","last_peered":"2023-11-28T18:23:07.667413+0000","last_clean":"2023-11-28T18:23:07.667413+0000","last_became_active":"2023-11-28T18:22:56.772763+0000","last_became_peered":"2023-11-28T18:22:56.772763+0000","last_unstale":"2023-11-28T18:23:07.667413+0000","last_undegraded":"2023-11-28T18:23:07.667413+0000","last_fullsized":"2023-11-28T18:23:07.667413+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_clean_scrub_stamp":"2023-11-28T18:22:55.591288+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:05.470408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113059+0000","last_change":"2023-11-28T18:23:07.674072+0000","last_active":"2023-11-28T18:23:09.113059+0000","last_peered":"2023-11-28T18:23:09.113059+0000","last_clean":"2023-11-28T18:23:09.113059+0000","last_became_active":"2023-11-28T18:23:07.673220+0000","last_became_peered":"2023-11-28T18:23:07.673220+0000","last_unstale":"2023-11-28T18:23:09.113059+0000","last_undegraded":"2023-11-28T18:23:09.113059+0000","last_fullsized":"2023-11-28T18:23:09.113059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:58.524112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.132519+0000","last_change":"2023-11-28T18:23:08.130989+0000","last_active":"2023-11-28T18:23:08.132519+0000","last_peered":"2023-11-28T18:23:08.132519+0000","last_clean":"2023-11-28T18:23:08.132519+0000","last_became_active":"2023-11-28T18:23:08.130646+0000","last_became_peered":"2023-11-28T18:23:08.130646+0000","last_unstale":"2023-11-28T18:23:08.132519+0000","last_undegraded":"2023-11-28T18:23:08.132519+0000","last_fullsized":"2023-11-28T18:23:08.132519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:06.796275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.116946+0000","last_change":"2023-11-28T18:23:07.686046+0000","last_active":"2023-11-28T18:23:08.116946+0000","last_peered":"2023-11-28T18:23:08.116946+0000","last_clean":"2023-11-28T18:23:08.116946+0000","last_became_active":"2023-11-28T18:23:07.685739+0000","last_became_peered":"2023-11-28T18:23:07.685739+0000","last_unstale":"2023-11-28T18:23:08.116946+0000","last_undegraded":"2023-11-28T18:23:08.116946+0000","last_fullsized":"2023-11-28T18:23:08.116946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:06.485631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131851+0000","last_change":"2023-11-28T18:23:08.130355+0000","last_active":"2023-11-28T18:23:08.131851+0000","last_peered":"2023-11-28T18:23:08.131851+0000","last_clean":"2023-11-28T18:23:08.131851+0000","last_became_active":"2023-11-28T18:23:08.129946+0000","last_became_peered":"2023-11-28T18:23:08.129946+0000","last_unstale":"2023-11-28T18:23:08.131851+0000","last_undegraded":"2023-11-28T18:23:08.131851+0000","last_fullsized":"2023-11-28T18:23:08.131851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:05:01.564614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.106082+0000","last_change":"2023-11-28T18:23:07.664827+0000","last_active":"2023-11-28T18:23:08.106082+0000","last_peered":"2023-11-28T18:23:08.106082+0000","last_clean":"2023-11-28T18:23:08.106082+0000","last_became_active":"2023-11-28T18:23:07.664503+0000","last_became_peered":"2023-11-28T18:23:07.664503+0000","last_unstale":"2023-11-28T18:23:08.106082+0000","last_undegraded":"2023-11-28T18:23:08.106082+0000","last_fullsized":"2023-11-28T18:23:08.106082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.238694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.123498+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.123197+0000","last_became_peered":"2023-11-28T18:23:08.123197+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:08:42.817848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113555+0000","last_change":"2023-11-28T18:23:07.673345+0000","last_active":"2023-11-28T18:23:09.113555+0000","last_peered":"2023-11-28T18:23:09.113555+0000","last_clean":"2023-11-28T18:23:09.113555+0000","last_became_active":"2023-11-28T18:23:07.672982+0000","last_became_peered":"2023-11-28T18:23:07.672982+0000","last_unstale":"2023-11-28T18:23:09.113555+0000","last_undegraded":"2023-11-28T18:23:09.113555+0000","last_fullsized":"2023-11-28T18:23:09.113555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:26.594232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.130392+0000","last_change":"2023-11-28T18:23:08.129091+0000","last_active":"2023-11-28T18:23:08.130392+0000","last_peered":"2023-11-28T18:23:08.130392+0000","last_clean":"2023-11-28T18:23:08.130392+0000","last_became_active":"2023-11-28T18:23:08.128589+0000","last_became_peered":"2023-11-28T18:23:08.128589+0000","last_unstale":"2023-11-28T18:23:08.130392+0000","last_undegraded":"2023-11-28T18:23:08.130392+0000","last_fullsized":"2023-11-28T18:23:08.130392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:29:48.729543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105665+0000","last_change":"2023-11-28T18:23:07.671874+0000","last_active":"2023-11-28T18:23:09.105665+0000","last_peered":"2023-11-28T18:23:09.105665+0000","last_clean":"2023-11-28T18:23:09.105665+0000","last_became_active":"2023-11-28T18:23:07.671500+0000","last_became_peered":"2023-11-28T18:23:07.671500+0000","last_unstale":"2023-11-28T18:23:09.105665+0000","last_undegraded":"2023-11-28T18:23:09.105665+0000","last_fullsized":"2023-11-28T18:23:09.105665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:42.922770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113612+0000","last_change":"2023-11-28T18:23:08.127419+0000","last_active":"2023-11-28T18:23:09.113612+0000","last_peered":"2023-11-28T18:23:09.113612+0000","last_clean":"2023-11-28T18:23:09.113612+0000","last_became_active":"2023-11-28T18:23:08.127157+0000","last_became_peered":"2023-11-28T18:23:08.127157+0000","last_unstale":"2023-11-28T18:23:09.113612+0000","last_undegraded":"2023-11-28T18:23:09.113612+0000","last_fullsized":"2023-11-28T18:23:09.113612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:08.473882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.120536+0000","last_change":"2023-11-28T18:23:07.684725+0000","last_active":"2023-11-28T18:23:08.120536+0000","last_peered":"2023-11-28T18:23:08.120536+0000","last_clean":"2023-11-28T18:23:08.120536+0000","last_became_active":"2023-11-28T18:23:07.684363+0000","last_became_peered":"2023-11-28T18:23:07.684363+0000","last_unstale":"2023-11-28T18:23:08.120536+0000","last_undegraded":"2023-11-28T18:23:08.120536+0000","last_fullsized":"2023-11-28T18:23:08.120536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:16.380789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105690+0000","last_change":"2023-11-28T18:23:08.120741+0000","last_active":"2023-11-28T18:23:09.105690+0000","last_peered":"2023-11-28T18:23:09.105690+0000","last_clean":"2023-11-28T18:23:09.105690+0000","last_became_active":"2023-11-28T18:23:08.120421+0000","last_became_peered":"2023-11-28T18:23:08.120421+0000","last_unstale":"2023-11-28T18:23:09.105690+0000","last_undegraded":"2023-11-28T18:23:09.105690+0000","last_fullsized":"2023-11-28T18:23:09.105690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:53:27.131631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105889+0000","last_change":"2023-11-28T18:23:07.672505+0000","last_active":"2023-11-28T18:23:09.105889+0000","last_peered":"2023-11-28T18:23:09.105889+0000","last_clean":"2023-11-28T18:23:09.105889+0000","last_became_active":"2023-11-28T18:23:07.672204+0000","last_became_peered":"2023-11-28T18:23:07.672204+0000","last_unstale":"2023-11-28T18:23:09.105889+0000","last_undegraded":"2023-11-28T18:23:09.105889+0000","last_fullsized":"2023-11-28T18:23:09.105889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:01.341643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131034+0000","last_change":"2023-11-28T18:23:08.126198+0000","last_active":"2023-11-28T18:23:08.131034+0000","last_peered":"2023-11-28T18:23:08.131034+0000","last_clean":"2023-11-28T18:23:08.131034+0000","last_became_active":"2023-11-28T18:23:08.125935+0000","last_became_peered":"2023-11-28T18:23:08.125935+0000","last_unstale":"2023-11-28T18:23:08.131034+0000","last_undegraded":"2023-11-28T18:23:08.131034+0000","last_fullsized":"2023-11-28T18:23:08.131034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:13:34.328584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109909+0000","last_change":"2023-11-28T18:23:07.672170+0000","last_active":"2023-11-28T18:23:09.109909+0000","last_peered":"2023-11-28T18:23:09.109909+0000","last_clean":"2023-11-28T18:23:09.109909+0000","last_became_active":"2023-11-28T18:23:07.671683+0000","last_became_peered":"2023-11-28T18:23:07.671683+0000","last_unstale":"2023-11-28T18:23:09.109909+0000","last_undegraded":"2023-11-28T18:23:09.109909+0000","last_fullsized":"2023-11-28T18:23:09.109909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:41.860341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.125078+0000","last_change":"2023-11-28T18:23:08.119844+0000","last_active":"2023-11-28T18:23:08.125078+0000","last_peered":"2023-11-28T18:23:08.125078+0000","last_clean":"2023-11-28T18:23:08.125078+0000","last_became_active":"2023-11-28T18:23:08.119525+0000","last_became_peered":"2023-11-28T18:23:08.119525+0000","last_unstale":"2023-11-28T18:23:08.125078+0000","last_undegraded":"2023-11-28T18:23:08.125078+0000","last_fullsized":"2023-11-28T18:23:08.125078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:15.677195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.116787+0000","last_change":"2023-11-28T18:23:07.685651+0000","last_active":"2023-11-28T18:23:08.116787+0000","last_peered":"2023-11-28T18:23:08.116787+0000","last_clean":"2023-11-28T18:23:08.116787+0000","last_became_active":"2023-11-28T18:23:07.683869+0000","last_became_peered":"2023-11-28T18:23:07.683869+0000","last_unstale":"2023-11-28T18:23:08.116787+0000","last_undegraded":"2023-11-28T18:23:08.116787+0000","last_fullsized":"2023-11-28T18:23:08.116787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:35.383852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.139176+0000","last_change":"2023-11-28T18:23:08.130016+0000","last_active":"2023-11-28T18:23:08.139176+0000","last_peered":"2023-11-28T18:23:08.139176+0000","last_clean":"2023-11-28T18:23:08.139176+0000","last_became_active":"2023-11-28T18:23:08.129586+0000","last_became_peered":"2023-11-28T18:23:08.129586+0000","last_unstale":"2023-11-28T18:23:08.139176+0000","last_undegraded":"2023-11-28T18:23:08.139176+0000","last_fullsized":"2023-11-28T18:23:08.139176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:26:27.495717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105620+0000","last_change":"2023-11-28T18:23:07.671773+0000","last_active":"2023-11-28T18:23:09.105620+0000","last_peered":"2023-11-28T18:23:09.105620+0000","last_clean":"2023-11-28T18:23:09.105620+0000","last_became_active":"2023-11-28T18:23:07.671465+0000","last_became_peered":"2023-11-28T18:23:07.671465+0000","last_unstale":"2023-11-28T18:23:09.105620+0000","last_undegraded":"2023-11-28T18:23:09.105620+0000","last_fullsized":"2023-11-28T18:23:09.105620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:44.862780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.126598+0000","last_change":"2023-11-28T18:23:08.118146+0000","last_active":"2023-11-28T18:23:08.126598+0000","last_peered":"2023-11-28T18:23:08.126598+0000","last_clean":"2023-11-28T18:23:08.126598+0000","last_became_active":"2023-11-28T18:23:08.117848+0000","last_became_peered":"2023-11-28T18:23:08.117848+0000","last_unstale":"2023-11-28T18:23:08.126598+0000","last_undegraded":"2023-11-28T18:23:08.126598+0000","last_fullsized":"2023-11-28T18:23:08.126598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:25.367755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105627+0000","last_change":"2023-11-28T18:23:07.676709+0000","last_active":"2023-11-28T18:23:09.105627+0000","last_peered":"2023-11-28T18:23:09.105627+0000","last_clean":"2023-11-28T18:23:09.105627+0000","last_became_active":"2023-11-28T18:23:07.676414+0000","last_became_peered":"2023-11-28T18:23:07.676414+0000","last_unstale":"2023-11-28T18:23:09.105627+0000","last_undegraded":"2023-11-28T18:23:09.105627+0000","last_fullsized":"2023-11-28T18:23:09.105627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.282005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.114651+0000","last_change":"2023-11-28T18:23:08.112410+0000","last_active":"2023-11-28T18:23:08.114651+0000","last_peered":"2023-11-28T18:23:08.114651+0000","last_clean":"2023-11-28T18:23:08.114651+0000","last_became_active":"2023-11-28T18:23:08.112097+0000","last_became_peered":"2023-11-28T18:23:08.112097+0000","last_unstale":"2023-11-28T18:23:08.114651+0000","last_undegraded":"2023-11-28T18:23:08.114651+0000","last_fullsized":"2023-11-28T18:23:08.114651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:03:43.223257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.124291+0000","last_change":"2023-11-28T18:23:07.684818+0000","last_active":"2023-11-28T18:23:08.124291+0000","last_peered":"2023-11-28T18:23:08.124291+0000","last_clean":"2023-11-28T18:23:08.124291+0000","last_became_active":"2023-11-28T18:23:07.684502+0000","last_became_peered":"2023-11-28T18:23:07.684502+0000","last_unstale":"2023-11-28T18:23:08.124291+0000","last_undegraded":"2023-11-28T18:23:08.124291+0000","last_fullsized":"2023-11-28T18:23:08.124291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:16:21.223266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109527+0000","last_change":"2023-11-28T18:23:08.133972+0000","last_active":"2023-11-28T18:23:09.109527+0000","last_peered":"2023-11-28T18:23:09.109527+0000","last_clean":"2023-11-28T18:23:09.109527+0000","last_became_active":"2023-11-28T18:23:08.133747+0000","last_became_peered":"2023-11-28T18:23:08.133747+0000","last_unstale":"2023-11-28T18:23:09.109527+0000","last_undegraded":"2023-11-28T18:23:09.109527+0000","last_fullsized":"2023-11-28T18:23:09.109527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:24.968241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113538+0000","last_change":"2023-11-28T18:23:08.134117+0000","last_active":"2023-11-28T18:23:09.113538+0000","last_peered":"2023-11-28T18:23:09.113538+0000","last_clean":"2023-11-28T18:23:09.113538+0000","last_became_active":"2023-11-28T18:23:08.133801+0000","last_became_peered":"2023-11-28T18:23:08.133801+0000","last_unstale":"2023-11-28T18:23:09.113538+0000","last_undegraded":"2023-11-28T18:23:09.113538+0000","last_fullsized":"2023-11-28T18:23:09.113538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:02.370862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117018+0000","last_change":"2023-11-28T18:23:07.686399+0000","last_active":"2023-11-28T18:23:08.117018+0000","last_peered":"2023-11-28T18:23:08.117018+0000","last_clean":"2023-11-28T18:23:08.117018+0000","last_became_active":"2023-11-28T18:23:07.686051+0000","last_became_peered":"2023-11-28T18:23:07.686051+0000","last_unstale":"2023-11-28T18:23:08.117018+0000","last_undegraded":"2023-11-28T18:23:08.117018+0000","last_fullsized":"2023-11-28T18:23:08.117018+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:11.797516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113317+0000","last_change":"2023-11-28T18:23:08.134870+0000","last_active":"2023-11-28T18:23:09.113317+0000","last_peered":"2023-11-28T18:23:09.113317+0000","last_clean":"2023-11-28T18:23:09.113317+0000","last_became_active":"2023-11-28T18:23:08.133732+0000","last_became_peered":"2023-11-28T18:23:08.133732+0000","last_unstale":"2023-11-28T18:23:09.113317+0000","last_undegraded":"2023-11-28T18:23:09.113317+0000","last_fullsized":"2023-11-28T18:23:09.113317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:01.364242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.120013+0000","last_change":"2023-11-28T18:23:07.682230+0000","last_active":"2023-11-28T18:23:08.120013+0000","last_peered":"2023-11-28T18:23:08.120013+0000","last_clean":"2023-11-28T18:23:08.120013+0000","last_became_active":"2023-11-28T18:23:07.681715+0000","last_became_peered":"2023-11-28T18:23:07.681715+0000","last_unstale":"2023-11-28T18:23:08.120013+0000","last_undegraded":"2023-11-28T18:23:08.120013+0000","last_fullsized":"2023-11-28T18:23:08.120013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:02.343996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131084+0000","last_change":"2023-11-28T18:23:08.128927+0000","last_active":"2023-11-28T18:23:08.131084+0000","last_peered":"2023-11-28T18:23:08.131084+0000","last_clean":"2023-11-28T18:23:08.131084+0000","last_became_active":"2023-11-28T18:23:08.127775+0000","last_became_peered":"2023-11-28T18:23:08.127775+0000","last_unstale":"2023-11-28T18:23:08.131084+0000","last_undegraded":"2023-11-28T18:23:08.131084+0000","last_fullsized":"2023-11-28T18:23:08.131084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:03.993918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.109224+0000","last_change":"2023-11-28T18:23:07.667929+0000","last_active":"2023-11-28T18:23:08.109224+0000","last_peered":"2023-11-28T18:23:08.109224+0000","last_clean":"2023-11-28T18:23:08.109224+0000","last_became_active":"2023-11-28T18:23:07.667666+0000","last_became_peered":"2023-11-28T18:23:07.667666+0000","last_unstale":"2023-11-28T18:23:08.109224+0000","last_undegraded":"2023-11-28T18:23:08.109224+0000","last_fullsized":"2023-11-28T18:23:08.109224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:32.132092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.131083+0000","last_change":"2023-11-28T18:23:08.126404+0000","last_active":"2023-11-28T18:23:08.131083+0000","last_peered":"2023-11-28T18:23:08.131083+0000","last_clean":"2023-11-28T18:23:08.131083+0000","last_became_active":"2023-11-28T18:23:08.125315+0000","last_became_peered":"2023-11-28T18:23:08.125315+0000","last_unstale":"2023-11-28T18:23:08.131083+0000","last_undegraded":"2023-11-28T18:23:08.131083+0000","last_fullsized":"2023-11-28T18:23:08.131083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:44.799540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.104198+0000","last_change":"2023-11-28T18:23:07.677758+0000","last_active":"2023-11-28T18:23:08.104198+0000","last_peered":"2023-11-28T18:23:08.104198+0000","last_clean":"2023-11-28T18:23:08.104198+0000","last_became_active":"2023-11-28T18:23:07.676902+0000","last_became_peered":"2023-11-28T18:23:07.676902+0000","last_unstale":"2023-11-28T18:23:08.104198+0000","last_undegraded":"2023-11-28T18:23:08.104198+0000","last_fullsized":"2023-11-28T18:23:08.104198+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:50.953647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":654978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":654978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T18:23:15.175 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T18:23:15.176 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-28T18:23:15.176 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-28T18:23:15.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-28T18:23:15.177 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-28T18:23:15.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe24bafe640 1 Processor -- start 2023-11-28T18:23:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe24bafe640 1 -- start start 2023-11-28T18:23:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe24bafe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2440c6ca0 0x7fe244161f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe24bafe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe24415e300 0x7fe2441624d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe24bafe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2440c2970 con 0x7fe2440c68c0 2023-11-28T18:23:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe24bafe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24415e820 con 0x7fe2440c6ca0 2023-11-28T18:23:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe24bafe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24415eb00 con 0x7fe24415e300 2023-11-28T18:23:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe249099640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2440c6ca0 0x7fe244161f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe249099640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2440c6ca0 0x7fe244161f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59352/0 (socket says 172.21.15.7:59352) 2023-11-28T18:23:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe249099640 1 -- 172.21.15.7:0/1810409487 learned_addr learned my addr 172.21.15.7:0/1810409487 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe24a09b640 1 --2- 172.21.15.7:0/1810409487 >> v2:172.21.15.193:3300/0 conn(0x7fe24415e300 0x7fe2441624d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.348+0000 7fe249099640 1 -- 172.21.15.7:0/1810409487 >> v2:172.21.15.193:3300/0 conn(0x7fe24415e300 msgr2=0x7fe2441624d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe249099640 1 --2- 172.21.15.7:0/1810409487 >> v2:172.21.15.193:3300/0 conn(0x7fe24415e300 0x7fe2441624d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe249099640 1 -- 172.21.15.7:0/1810409487 >> v1:172.21.15.193:6789/0 conn(0x7fe2440c68c0 legacy=0x7fe2440c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe249099640 1 -- 172.21.15.7:0/1810409487 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24415ede0 con 0x7fe2440c6ca0 2023-11-28T18:23:15.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe24a09b640 1 --2- 172.21.15.7:0/1810409487 >> v2:172.21.15.193:3300/0 conn(0x7fe24415e300 0x7fe2441624d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:15.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe249099640 1 --2- 172.21.15.7:0/1810409487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe244161f90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe2440c7160 tx=0x7fe23401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e884e03d39794 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:15.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe248898640 1 -- 172.21.15.7:0/1810409487 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340a89f0 con 0x7fe2440c6ca0 2023-11-28T18:23:15.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe248898640 1 -- 172.21.15.7:0/1810409487 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2340a8b50 con 0x7fe2440c6ca0 2023-11-28T18:23:15.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe248898640 1 -- 172.21.15.7:0/1810409487 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340b0440 con 0x7fe2440c6ca0 2023-11-28T18:23:15.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe24bafe640 1 -- 172.21.15.7:0/1810409487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 msgr2=0x7fe244161f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe24bafe640 1 --2- 172.21.15.7:0/1810409487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe244161f90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe2440c7160 tx=0x7fe23401ecf0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe24bafe640 1 -- 172.21.15.7:0/1810409487 shutdown_connections 2023-11-28T18:23:15.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe24bafe640 1 --2- 172.21.15.7:0/1810409487 >> v2:172.21.15.193:3300/0 conn(0x7fe24415e300 0x7fe2441624d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe24bafe640 1 --2- 172.21.15.7:0/1810409487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe244161f90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.352+0000 7fe24bafe640 1 -- 172.21.15.7:0/1810409487 >> 172.21.15.7:0/1810409487 conn(0x7fe2440662e0 msgr2=0x7fe2440666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:15.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 -- 172.21.15.7:0/1810409487 shutdown_connections 2023-11-28T18:23:15.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 -- 172.21.15.7:0/1810409487 wait complete. 2023-11-28T18:23:15.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 Processor -- start 2023-11-28T18:23:15.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 -- start start 2023-11-28T18:23:15.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe2440c68c0 0x7fe244156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe2440ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe24415e300 0x7fe2440c3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe244157230 con 0x7fe2440c6ca0 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe244157480 con 0x7fe24415e300 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24bafe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe244157760 con 0x7fe2440c68c0 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe2440ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe2440ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59358/0 (socket says 172.21.15.7:59358) 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 -- 172.21.15.7:0/3141261455 learned_addr learned my addr 172.21.15.7:0/3141261455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24a09b640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe24415e300 0x7fe2440c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24989a640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe2440c68c0 0x7fe244156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 -- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe2440c68c0 msgr2=0x7fe244156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe2440c68c0 0x7fe244156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 -- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe24415e300 msgr2=0x7fe2440c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe24415e300 0x7fe2440c3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 -- 172.21.15.7:0/3141261455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2340094d0 con 0x7fe2440c6ca0 2023-11-28T18:23:15.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe24a09b640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe24415e300 0x7fe2440c3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:15.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe249099640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe2440ca0c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe23401b900 tx=0x7fe23401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:15.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.356+0000 7fe23a7fc640 1 -- 172.21.15.7:0/3141261455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340ba600 con 0x7fe2440c6ca0 2023-11-28T18:23:15.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe23a7fc640 1 -- 172.21.15.7:0/3141261455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2340ba780 con 0x7fe2440c6ca0 2023-11-28T18:23:15.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe23a7fc640 1 -- 172.21.15.7:0/3141261455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340bab30 con 0x7fe2440c6ca0 2023-11-28T18:23:15.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2440c95e0 con 0x7fe2440c6ca0 2023-11-28T18:23:15.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2440c9ba0 con 0x7fe2440c6ca0 2023-11-28T18:23:15.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24404e890 con 0x7fe2440c6ca0 2023-11-28T18:23:15.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe23a7fc640 1 -- 172.21.15.7:0/3141261455 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe2340bac90 con 0x7fe2440c6ca0 2023-11-28T18:23:15.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe23a7fc640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe2280668d0 0x7fe228068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe23a7fc640 1 -- 172.21.15.7:0/3141261455 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe234122c00 con 0x7fe2440c6ca0 2023-11-28T18:23:15.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe24989a640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe2280668d0 0x7fe228068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.360+0000 7fe24989a640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe2280668d0 0x7fe228068d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe240000f70 tx=0x7fe240009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:15.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.364+0000 7fe23a7fc640 1 -- 172.21.15.7:0/3141261455 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe2340efec0 con 0x7fe2440c6ca0 2023-11-28T18:23:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.560+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe2440c71c0 con 0x7fe2440c6ca0 2023-11-28T18:23:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe23a7fc640 1 -- 172.21.15.7:0/3141261455 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164958 (secure 0 0 0) 0x7fe2340b5090 con 0x7fe2440c6ca0 2023-11-28T18:23:15.571 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe2280668d0 msgr2=0x7fe228068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe2280668d0 0x7fe228068d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe240000f70 tx=0x7fe240009040 comp rx=0 tx=0).stop 2023-11-28T18:23:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 msgr2=0x7fe2440ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe2440ca0c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe23401b900 tx=0x7fe23401ba90 comp rx=0 tx=0).stop 2023-11-28T18:23:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 shutdown_connections 2023-11-28T18:23:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe2280668d0 0x7fe228068d90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe24415e300 0x7fe2440c3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2440c6ca0 0x7fe2440ca0c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 --2- 172.21.15.7:0/3141261455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe2440c68c0 0x7fe244156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.564+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 >> 172.21.15.7:0/3141261455 conn(0x7fe2440662e0 msgr2=0x7fe24406d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.568+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 shutdown_connections 2023-11-28T18:23:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.568+0000 7fe24bafe640 1 -- 172.21.15.7:0/3141261455 wait complete. 2023-11-28T18:23:15.590 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"active_gid":4102,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":264069339},{"type":"v1","addr":"172.21.15.7:6833","nonce":264069339}]},"active_addr":"172.21.15.7:6833/264069339","active_change":"2023-11-28T18:22:47.445778+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":956443581}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3298981666}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":253708196}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2085927756}]}]} 2023-11-28T18:23:15.594 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-28T18:23:15.595 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-28T18:23:15.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed772a640 1 Processor -- start 2023-11-28T18:23:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed772a640 1 -- start start 2023-11-28T18:23:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed772a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbed0150d80 0x7fbed015db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed772a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed015a310 con 0x7fbed0159e60 2023-11-28T18:23:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed772a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed015a5f0 con 0x7fbed015e050 2023-11-28T18:23:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed772a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed015ae90 con 0x7fbed0150d80 2023-11-28T18:23:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed4cc5640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbed0159e60 0x7fbed00cd5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55152/0 (socket says 172.21.15.7:55152) 2023-11-28T18:23:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed4cc5640 1 -- 172.21.15.7:0/2096997088 learned_addr learned my addr 172.21.15.7:0/2096997088 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2096997088 >> v2:172.21.15.193:3300/0 conn(0x7fbed0150d80 0x7fbed015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646448106 0 0) 0x7fbed015a310 con 0x7fbed0159e60 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbebc003530 con 0x7fbed0159e60 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897268533 0 0) 0x7fbed015a5f0 con 0x7fbed015e050 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed015a310 con 0x7fbed015e050 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436572662 0 0) 0x7fbebc003530 con 0x7fbed0159e60 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 >> v2:172.21.15.193:3300/0 conn(0x7fbed0150d80 msgr2=0x7fbed015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 --2- 172.21.15.7:0/2096997088 >> v2:172.21.15.193:3300/0 conn(0x7fbed0150d80 0x7fbed015db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 >> v1:172.21.15.193:6789/0 conn(0x7fbed015e050 legacy=0x7fbed00729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed015a8d0 con 0x7fbed0159e60 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fbec0001680 con 0x7fbed0159e60 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2096997088 >> v2:172.21.15.193:3300/0 conn(0x7fbed0150d80 0x7fbed015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 >> v1:172.21.15.7:6789/0 conn(0x7fbed0159e60 legacy=0x7fbed00cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbebc003b10 0x7fbebc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbebc0047d0 0x7fbebc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc005190 0x7fbebc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed015ae90 con 0x7fbebc003b10 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc00bf80 con 0x7fbebc0047d0 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc00c140 con 0x7fbebc005190 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbebc003b10 0x7fbebc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed5cc7640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc005190 0x7fbebc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.736+0000 7fbed4cc5640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbebc0047d0 0x7fbebc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed54c6640 1 -- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc005190 msgr2=0x7fbebc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc005190 0x7fbebc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed54c6640 1 -- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbebc0047d0 msgr2=0x7fbebc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbebc0047d0 0x7fbebc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed54c6640 1 -- 172.21.15.7:0/2096997088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc003530 con 0x7fbebc003b10 2023-11-28T18:23:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed4cc5640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbebc0047d0 0x7fbebc004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed5cc7640 1 -- 172.21.15.7:0/2096997088 reap_dead start 2023-11-28T18:23:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbebc003b10 0x7fbebc003f10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009c80 tx=0x7fbecc01cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc10161a981a594 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0a5740 con 0x7fbebc003b10 2023-11-28T18:23:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc0a58a0 con 0x7fbebc003b10 2023-11-28T18:23:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbec7fff640 1 -- 172.21.15.7:0/2096997088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0a5c50 con 0x7fbebc003b10 2023-11-28T18:23:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed772a640 1 -- 172.21.15.7:0/2096997088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbebc003b10 msgr2=0x7fbebc003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed772a640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbebc003b10 0x7fbebc003f10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009c80 tx=0x7fbecc01cb90 comp rx=0 tx=0).stop 2023-11-28T18:23:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed772a640 1 -- 172.21.15.7:0/2096997088 shutdown_connections 2023-11-28T18:23:15.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed772a640 1 --2- 172.21.15.7:0/2096997088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbebc003b10 0x7fbebc003f10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.740+0000 7fbed772a640 1 -- 172.21.15.7:0/2096997088 >> 172.21.15.7:0/2096997088 conn(0x7fbed006ff10 msgr2=0x7fbed0070320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:15.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 -- 172.21.15.7:0/2096997088 shutdown_connections 2023-11-28T18:23:15.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 -- 172.21.15.7:0/2096997088 wait complete. 2023-11-28T18:23:15.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 Processor -- start 2023-11-28T18:23:15.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 -- start start 2023-11-28T18:23:15.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc003b10 0x7fbed014fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed00cad50 0x7fbed0149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbed0148ed0 0x7fbed0143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed015b020 con 0x7fbed0148ed0 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed015b2e0 con 0x7fbed00cad50 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed772a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed015b5c0 con 0x7fbebc003b10 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbed0148ed0 0x7fbed0143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbed0148ed0 0x7fbed0143f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59366/0 (socket says 172.21.15.7:59366) 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 -- 172.21.15.7:0/2166011926 learned_addr learned my addr 172.21.15.7:0/2166011926 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed4cc5640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed00cad50 0x7fbed0149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc003b10 0x7fbed014fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 -- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc003b10 msgr2=0x7fbed014fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc003b10 0x7fbed014fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 -- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed00cad50 msgr2=0x7fbed0149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed00cad50 0x7fbed0149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 -- 172.21.15.7:0/2166011926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed015b8a0 con 0x7fbed0148ed0 2023-11-28T18:23:15.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed4cc5640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed00cad50 0x7fbed0149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:15.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.744+0000 7fbed5cc7640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbed0148ed0 0x7fbed0143f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbed006e120 tx=0x7fbec8012d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:15.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbec5ffb640 1 -- 172.21.15.7:0/2166011926 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec80073e0 con 0x7fbed0148ed0 2023-11-28T18:23:15.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbec5ffb640 1 -- 172.21.15.7:0/2166011926 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec8010b20 con 0x7fbed0148ed0 2023-11-28T18:23:15.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed01497c0 con 0x7fbed0148ed0 2023-11-28T18:23:15.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbec5ffb640 1 -- 172.21.15.7:0/2166011926 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8010eb0 con 0x7fbed0148ed0 2023-11-28T18:23:15.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed015bb80 con 0x7fbed0148ed0 2023-11-28T18:23:15.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed015d590 con 0x7fbed0148ed0 2023-11-28T18:23:15.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbec5ffb640 1 -- 172.21.15.7:0/2166011926 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbec8016480 con 0x7fbed0148ed0 2023-11-28T18:23:15.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbec5ffb640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbeb40668d0 0x7fbeb4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:15.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbec5ffb640 1 -- 172.21.15.7:0/2166011926 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbec8087c30 con 0x7fbed0148ed0 2023-11-28T18:23:15.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbeb40668d0 0x7fbeb4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:15.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.748+0000 7fbed54c6640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbeb40668d0 0x7fbeb4068d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009c80 tx=0x7fbecc0a7040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:15.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.752+0000 7fbec5ffb640 1 -- 172.21.15.7:0/2166011926 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbec8054dd0 con 0x7fbed0148ed0 2023-11-28T18:23:15.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.912+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbed004e890 con 0x7fbed0148ed0 2023-11-28T18:23:15.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.912+0000 7fbec5ffb640 1 -- 172.21.15.7:0/2166011926 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fbec80583f0 con 0x7fbed0148ed0 2023-11-28T18:23:15.918 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:15.918 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbeb40668d0 msgr2=0x7fbeb4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbeb40668d0 0x7fbeb4068d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009c80 tx=0x7fbecc0a7040 comp rx=0 tx=0).stop 2023-11-28T18:23:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbed0148ed0 msgr2=0x7fbed0143f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbed0148ed0 0x7fbed0143f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbed006e120 tx=0x7fbec8012d20 comp rx=0 tx=0).stop 2023-11-28T18:23:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 shutdown_connections 2023-11-28T18:23:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbeb40668d0 0x7fbeb4068d90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbed0148ed0 0x7fbed0143f80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed00cad50 0x7fbed0149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 --2- 172.21.15.7:0/2166011926 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbebc003b10 0x7fbed014fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 >> 172.21.15.7:0/2166011926 conn(0x7fbed006ff10 msgr2=0x7fbed014dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 shutdown_connections 2023-11-28T18:23:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:15.916+0000 7fbed772a640 1 -- 172.21.15.7:0/2166011926 wait complete. 2023-11-28T18:23:15.939 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-28T18:23:15.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:16.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.076+0000 7f4653054640 1 Processor -- start 2023-11-28T18:23:16.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.076+0000 7f4653054640 1 -- start start 2023-11-28T18:23:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f4653054640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c0ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f4653054640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c15d4e0 con 0x7f464c0cb220 2023-11-28T18:23:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f4653054640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c0717f0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f4653054640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c15d730 con 0x7f464c0cae40 2023-11-28T18:23:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f464bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f464c0cb220 0x7f464c070f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55158/0 (socket says 172.21.15.7:55158) 2023-11-28T18:23:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f464bfff640 1 -- 172.21.15.7:0/2141709212 learned_addr learned my addr 172.21.15.7:0/2141709212 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f4650df0640 1 --2- 172.21.15.7:0/2141709212 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c0ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f464b7fe640 1 -- 172.21.15.7:0/2141709212 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816655281 0 0) 0x7f464c15d4e0 con 0x7f464c0cb220 2023-11-28T18:23:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f464b7fe640 1 -- 172.21.15.7:0/2141709212 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f463c003530 con 0x7f464c0cb220 2023-11-28T18:23:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f464b7fe640 1 -- 172.21.15.7:0/2141709212 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152352148 0 0) 0x7f464c0717f0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f464b7fe640 1 -- 172.21.15.7:0/2141709212 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f464c15d4e0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f4650df0640 1 -- 172.21.15.7:0/2141709212 >> v1:172.21.15.193:6789/0 conn(0x7f464c0cf9f0 legacy=0x7f464c0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f4650df0640 1 -- 172.21.15.7:0/2141709212 >> v1:172.21.15.7:6789/0 conn(0x7f464c0cb220 legacy=0x7f464c070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.080+0000 7f4650df0640 1 -- 172.21.15.7:0/2141709212 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c15da10 con 0x7f464c0cae40 2023-11-28T18:23:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4650df0640 1 --2- 172.21.15.7:0/2141709212 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c0ca4a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f46400097f0 tx=0x7f464001ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=691b87197818cdee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f464b7fe640 1 -- 172.21.15.7:0/2141709212 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46400a89b0 con 0x7f464c0cae40 2023-11-28T18:23:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f464b7fe640 1 -- 172.21.15.7:0/2141709212 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46400a8b10 con 0x7f464c0cae40 2023-11-28T18:23:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f464b7fe640 1 -- 172.21.15.7:0/2141709212 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46400a8ea0 con 0x7f464c0cae40 2023-11-28T18:23:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4653054640 1 -- 172.21.15.7:0/2141709212 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 msgr2=0x7f464c0ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4653054640 1 --2- 172.21.15.7:0/2141709212 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c0ca4a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f46400097f0 tx=0x7f464001ecb0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4653054640 1 -- 172.21.15.7:0/2141709212 shutdown_connections 2023-11-28T18:23:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4653054640 1 --2- 172.21.15.7:0/2141709212 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c0ca4a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4653054640 1 -- 172.21.15.7:0/2141709212 >> 172.21.15.7:0/2141709212 conn(0x7f464c072310 msgr2=0x7f464c072720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4653054640 1 -- 172.21.15.7:0/2141709212 shutdown_connections 2023-11-28T18:23:16.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4653054640 1 -- 172.21.15.7:0/2141709212 wait complete. 2023-11-28T18:23:16.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.084+0000 7f4653054640 1 Processor -- start 2023-11-28T18:23:16.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 -- start start 2023-11-28T18:23:16.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f464c0cb220 0x7f464c14be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f464c0cf9f0 0x7f464c14c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c0c9d10 con 0x7f464c0cf9f0 2023-11-28T18:23:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c155320 con 0x7f464c0cb220 2023-11-28T18:23:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c15d260 con 0x7f464c0cae40 2023-11-28T18:23:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f464c0cf9f0 0x7f464c14c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f464c0cf9f0 0x7f464c14c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59374/0 (socket says 172.21.15.7:59374) 2023-11-28T18:23:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 -- 172.21.15.7:0/4015090441 learned_addr learned my addr 172.21.15.7:0/4015090441 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4650df0640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f464bfff640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f464c0cb220 0x7f464c14be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 -- 172.21.15.7:0/4015090441 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 msgr2=0x7f464c14e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c14e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 -- 172.21.15.7:0/4015090441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f464c0cb220 msgr2=0x7f464c14be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f464c0cb220 0x7f464c14be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 -- 172.21.15.7:0/4015090441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4640009420 con 0x7f464c0cf9f0 2023-11-28T18:23:16.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46515f1640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f464c0cf9f0 0x7f464c14c3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f464400c730 tx=0x7f464400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46497fa640 1 -- 172.21.15.7:0/4015090441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644003b80 con 0x7f464c0cf9f0 2023-11-28T18:23:16.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46497fa640 1 -- 172.21.15.7:0/4015090441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4644003ce0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c1ef900 con 0x7f464c0cf9f0 2023-11-28T18:23:16.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f46497fa640 1 -- 172.21.15.7:0/4015090441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464400aa90 con 0x7f464c0cf9f0 2023-11-28T18:23:16.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c1efec0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.088+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c15dfb0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.092+0000 7f46497fa640 1 -- 172.21.15.7:0/4015090441 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f464400abf0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.092+0000 7f46497fa640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46300668d0 0x7f4630068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.092+0000 7f46497fa640 1 -- 172.21.15.7:0/4015090441 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46440842c0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.092+0000 7f4650df0640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46300668d0 0x7f4630068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.092+0000 7f4650df0640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46300668d0 0x7f4630068d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46400b2000 tx=0x7f4640008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.096+0000 7f46497fa640 1 -- 172.21.15.7:0/4015090441 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f464c15dfb0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.256+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f464c15dfb0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f46497fa640 1 -- 172.21.15.7:0/4015090441 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f464c15dfb0 con 0x7f464c0cf9f0 2023-11-28T18:23:16.264 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:16.264 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:16.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46300668d0 msgr2=0x7f4630068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46300668d0 0x7f4630068d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46400b2000 tx=0x7f4640008040 comp rx=0 tx=0).stop 2023-11-28T18:23:16.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f464c0cf9f0 msgr2=0x7f464c14c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f464c0cf9f0 0x7f464c14c3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f464400c730 tx=0x7f464400cbf0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 shutdown_connections 2023-11-28T18:23:16.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46300668d0 0x7f4630068d90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f464c0cf9f0 0x7f464c14c3c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f464c0cb220 0x7f464c14be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 --2- 172.21.15.7:0/4015090441 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f464c0cae40 0x7f464c14e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 >> 172.21.15.7:0/4015090441 conn(0x7f464c072310 msgr2=0x7f464c0ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 shutdown_connections 2023-11-28T18:23:16.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.260+0000 7f4653054640 1 -- 172.21.15.7:0/4015090441 wait complete. 2023-11-28T18:23:16.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-28T18:23:16.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-28T18:23:16.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-28T18:23:16.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-28T18:23:16.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-28T18:23:16.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-28T18:23:16.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-28T18:23:16.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-28T18:23:16.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.532+0000 7f5828cbb640 1 Processor -- start 2023-11-28T18:23:16.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.532+0000 7f5828cbb640 1 -- start start 2023-11-28T18:23:16.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.532+0000 7f5828cbb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5824158480 0x7f5824072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.532+0000 7f5828cbb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f58240c5d70 0x7f58240c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.532+0000 7f5828cbb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582415e010 con 0x7f58240c7be0 2023-11-28T18:23:16.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.532+0000 7f5828cbb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58240c80f0 con 0x7f58240c5d70 2023-11-28T18:23:16.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.532+0000 7f5828cbb640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58240c83d0 con 0x7f5824158480 2023-11-28T18:23:16.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f5822d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f58240c5d70 0x7f58240c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f5821d9b640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f58240c7be0 0x7f5824161600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35580/0 (socket says 172.21.15.7:35580) 2023-11-28T18:23:16.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f5821d9b640 1 -- 172.21.15.7:0/2202383940 learned_addr learned my addr 172.21.15.7:0/2202383940 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f582259c640 1 --2- 172.21.15.7:0/2202383940 >> v2:172.21.15.193:3300/0 conn(0x7f5824158480 0x7f5824072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f582159a640 1 -- 172.21.15.7:0/2202383940 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100161863 0 0) 0x7f582415e010 con 0x7f58240c7be0 2023-11-28T18:23:16.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f582159a640 1 -- 172.21.15.7:0/2202383940 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580c003530 con 0x7f58240c7be0 2023-11-28T18:23:16.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f5822d9d640 1 -- 172.21.15.7:0/2202383940 >> v2:172.21.15.193:3300/0 conn(0x7f5824158480 msgr2=0x7f5824072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f5822d9d640 1 --2- 172.21.15.7:0/2202383940 >> v2:172.21.15.193:3300/0 conn(0x7f5824158480 0x7f5824072c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f5822d9d640 1 -- 172.21.15.7:0/2202383940 >> v1:172.21.15.193:6789/0 conn(0x7f58240c7be0 legacy=0x7f5824161600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7f5822d9d640 1 -- 172.21.15.7:0/2202383940 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582415e2d0 con 0x7f58240c5d70 2023-11-28T18:23:16.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7fa79731d640 1 Processor -- start 2023-11-28T18:23:16.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.536+0000 7fa79731d640 1 -- start start 2023-11-28T18:23:16.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa79731d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa79014fef0 0x7fa790159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f995ab3e640 1 Processor -- start 2023-11-28T18:23:16.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f995ab3e640 1 -- start start 2023-11-28T18:23:16.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa79731d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa790155f10 con 0x7fa7900c6170 2023-11-28T18:23:16.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa79731d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7901561d0 con 0x7fa7900c3090 2023-11-28T18:23:16.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa79731d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa790155d20 con 0x7fa79014fef0 2023-11-28T18:23:16.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa7950b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa79014fef0 0x7fa790159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa7950b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa79014fef0 0x7fa790159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59388/0 (socket says 172.21.15.7:59388) 2023-11-28T18:23:16.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa7950b9640 1 -- 172.21.15.7:0/1058346623 learned_addr learned my addr 172.21.15.7:0/1058346623 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f995ab3e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f99540ca2c0 0x7f99540ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898271105 0 0) 0x7fa7901561d0 con 0x7fa7900c3090 2023-11-28T18:23:16.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77c003530 con 0x7fa7900c3090 2023-11-28T18:23:16.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f99588da640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f995415ee20 0x7f99540cb040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35606/0 (socket says 172.21.15.7:35606) 2023-11-28T18:23:16.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f99588da640 1 -- 172.21.15.7:0/4267913204 learned_addr learned my addr 172.21.15.7:0/4267913204 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f9953fff640 1 --2- 172.21.15.7:0/4267913204 >> v2:172.21.15.193:3300/0 conn(0x7f99540ca2c0 0x7f99540ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f995ab3e640 1 -- 172.21.15.7:0/4267913204 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99540cac50 con 0x7f995415fc40 2023-11-28T18:23:16.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f995ab3e640 1 -- 172.21.15.7:0/4267913204 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9954161ae0 con 0x7f995415ee20 2023-11-28T18:23:16.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7f995ab3e640 1 -- 172.21.15.7:0/4267913204 --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9954161dc0 con 0x7f99540ca2c0 2023-11-28T18:23:16.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3224489429 0 0) 0x7fa77c003530 con 0x7fa7900c3090 2023-11-28T18:23:16.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.540+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 >> v2:172.21.15.7:3300/0 conn(0x7fa79014fef0 msgr2=0x7fa790159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 --2- 172.21.15.7:0/1058346623 >> v2:172.21.15.7:3300/0 conn(0x7fa79014fef0 0x7fa790159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 >> v1:172.21.15.7:6789/0 conn(0x7fa7900c6170 legacy=0x7fa7900c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f99537fe640 1 -- 172.21.15.7:0/4267913204 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343317394 0 0) 0x7f9954161ae0 con 0x7f995415ee20 2023-11-28T18:23:16.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f99537fe640 1 -- 172.21.15.7:0/4267913204 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9940003530 con 0x7f995415ee20 2023-11-28T18:23:16.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f9953fff640 1 -- 172.21.15.7:0/4267913204 >> v1:172.21.15.193:6789/0 conn(0x7f995415ee20 legacy=0x7f99540cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f9953fff640 1 -- 172.21.15.7:0/4267913204 >> v1:172.21.15.7:6789/0 conn(0x7f995415fc40 legacy=0x7f9954160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f9953fff640 1 -- 172.21.15.7:0/4267913204 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99541620a0 con 0x7f99540ca2c0 2023-11-28T18:23:16.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e6b8d640 1 Processor -- start 2023-11-28T18:23:16.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e6b8d640 1 -- start start 2023-11-28T18:23:16.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7901564b0 con 0x7fa7900c3090 2023-11-28T18:23:16.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f5822d9d640 1 --2- 172.21.15.7:0/2202383940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 0x7f58240c61c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5824066d30 tx=0x7f581401c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8531971ae14daa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fa788001a00 con 0x7fa7900c3090 2023-11-28T18:23:16.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 >> v1:172.21.15.193:6789/0 conn(0x7fa7900c3090 legacy=0x7fa7900c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa77c0039d0 0x7fa77c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa77c0047d0 0x7fa77c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa77c00fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c005960 con 0x7fa77c0039d0 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c005ad0 con 0x7fa77c0051e0 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c005c90 con 0x7fa77c0047d0 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f582159a640 1 -- 172.21.15.7:0/2202383940 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58140ad640 con 0x7f58240c5d70 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f582159a640 1 -- 172.21.15.7:0/2202383940 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58140ad7a0 con 0x7f58240c5d70 2023-11-28T18:23:16.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7f582159a640 1 -- 172.21.15.7:0/2202383940 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58140adb30 con 0x7f58240c5d70 2023-11-28T18:23:16.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa7950b9640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa77c0039d0 0x7fa77c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e6b8d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e00ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e4929640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd7e0154140 0x7fd7e0159ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35616/0 (socket says 172.21.15.7:35616) 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e4929640 1 -- 172.21.15.7:0/1525649089 learned_addr learned my addr 172.21.15.7:0/1525649089 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1525649089 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e00c4b10 con 0x7fd7e00cb420 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e512a640 1 --2- 172.21.15.7:0/1525649089 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e00ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1525649089 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e015a770 con 0x7fd7e0154140 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1525649089 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e015aa50 con 0x7fd7e00cb890 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa7958ba640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa77c00fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa7948b8640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa77c0047d0 0x7fa77c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa7958ba640 1 -- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa77c0047d0 msgr2=0x7fa77c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa7958ba640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa77c0047d0 0x7fa77c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa7958ba640 1 -- 172.21.15.7:0/1058346623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa77c0039d0 msgr2=0x7fa77c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa7958ba640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa77c0039d0 0x7fa77c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fa7958ba640 1 -- 172.21.15.7:0/1058346623 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c003530 con 0x7fa77c0051e0 2023-11-28T18:23:16.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7df7fe640 1 -- 172.21.15.7:0/1525649089 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719122278 0 0) 0x7fd7e015a770 con 0x7fd7e0154140 2023-11-28T18:23:16.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e512a640 1 -- 172.21.15.7:0/1525649089 >> v1:172.21.15.193:6789/0 conn(0x7fd7e0154140 legacy=0x7fd7e0159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e512a640 1 -- 172.21.15.7:0/1525649089 >> v1:172.21.15.7:6789/0 conn(0x7fd7e00cb420 legacy=0x7fd7e0072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.544+0000 7fd7e512a640 1 -- 172.21.15.7:0/1525649089 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e015ad30 con 0x7fd7e00cb890 2023-11-28T18:23:16.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf2d654640 1 Processor -- start 2023-11-28T18:23:16.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f5828cbb640 1 -- 172.21.15.7:0/2202383940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 msgr2=0x7f58240c61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f5828cbb640 1 --2- 172.21.15.7:0/2202383940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 0x7f58240c61c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5824066d30 tx=0x7f581401c390 comp rx=0 tx=0).stop 2023-11-28T18:23:16.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf2d654640 1 -- start start 2023-11-28T18:23:16.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fa7950b9640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa77c0039d0 0x7fa77c003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:16.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f5828cbb640 1 -- 172.21.15.7:0/2202383940 shutdown_connections 2023-11-28T18:23:16.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f5828cbb640 1 --2- 172.21.15.7:0/2202383940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 0x7f58240c61c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f5828cbb640 1 --2- 172.21.15.7:0/2202383940 >> v2:172.21.15.193:3300/0 conn(0x7f5824158480 0x7f5824072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f5828cbb640 1 -- 172.21.15.7:0/2202383940 >> 172.21.15.7:0/2202383940 conn(0x7f582406c2c0 msgr2=0x7f58240662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f9953fff640 1 --2- 172.21.15.7:0/4267913204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99540ca2c0 0x7f99540ca710 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f99440097c0 tx=0x7f994401ebf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75b3827361a690dd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f5828cbb640 1 -- 172.21.15.7:0/2202383940 shutdown_connections 2023-11-28T18:23:16.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f5828cbb640 1 -- 172.21.15.7:0/2202383940 wait complete. 2023-11-28T18:23:16.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf2d654640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcf2004f900 0x7fcf20008db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf2d654640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf20001470 0x7fcf2004ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf2d654640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2004f6b0 con 0x7fcf20001090 2023-11-28T18:23:16.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf2d654640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf20001d10 con 0x7fcf2004f900 2023-11-28T18:23:16.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf2d654640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2004f130 con 0x7fcf20001470 2023-11-28T18:23:16.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf26ffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fcf20001090 0x7fcf20048c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35620/0 (socket says 172.21.15.7:35620) 2023-11-28T18:23:16.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf26ffd640 1 -- 172.21.15.7:0/1630899701 learned_addr learned my addr 172.21.15.7:0/1630899701 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f99537fe640 1 -- 172.21.15.7:0/4267913204 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99440a88f0 con 0x7f99540ca2c0 2023-11-28T18:23:16.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f99537fe640 1 -- 172.21.15.7:0/4267913204 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99440a8a50 con 0x7f99540ca2c0 2023-11-28T18:23:16.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f99537fe640 1 -- 172.21.15.7:0/4267913204 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99440a8de0 con 0x7f99540ca2c0 2023-11-28T18:23:16.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f995ab3e640 1 -- 172.21.15.7:0/4267913204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99540ca2c0 msgr2=0x7f99540ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7f995ab3e640 1 --2- 172.21.15.7:0/4267913204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99540ca2c0 0x7f99540ca710 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f99440097c0 tx=0x7f994401ebf0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f995ab3e640 1 -- 172.21.15.7:0/4267913204 shutdown_connections 2023-11-28T18:23:16.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f995ab3e640 1 --2- 172.21.15.7:0/4267913204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99540ca2c0 0x7f99540ca710 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f995ab3e640 1 -- 172.21.15.7:0/4267913204 >> 172.21.15.7:0/4267913204 conn(0x7f99540662e0 msgr2=0x7f99540666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fa7958ba640 1 -- 172.21.15.7:0/1058346623 reap_dead start 2023-11-28T18:23:16.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf267fc640 1 --2- 172.21.15.7:0/1630899701 >> v2:172.21.15.7:3300/0 conn(0x7fcf2004f900 0x7fcf20008db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf27fff640 1 --2- 172.21.15.7:0/1630899701 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf20001470 0x7fcf2004ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf267fc640 1 -- 172.21.15.7:0/1630899701 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf20001470 msgr2=0x7fcf2004ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf267fc640 1 --2- 172.21.15.7:0/1630899701 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf20001470 0x7fcf2004ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf267fc640 1 -- 172.21.15.7:0/1630899701 >> v1:172.21.15.193:6789/0 conn(0x7fcf20001090 legacy=0x7fcf20048c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.548+0000 7fcf267fc640 1 -- 172.21.15.7:0/1630899701 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2004a790 con 0x7fcf2004f900 2023-11-28T18:23:16.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5334cd6640 1 Processor -- start 2023-11-28T18:23:16.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f995ab3e640 1 -- 172.21.15.7:0/4267913204 shutdown_connections 2023-11-28T18:23:16.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f995ab3e640 1 -- 172.21.15.7:0/4267913204 wait complete. 2023-11-28T18:23:16.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7e512a640 1 --2- 172.21.15.7:0/1525649089 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e00ca500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009dc0 tx=0x7fd7d801f140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4034acbc839fac72 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5334cd6640 1 -- start start 2023-11-28T18:23:16.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5828cbb640 1 Processor -- start 2023-11-28T18:23:16.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7df7fe640 1 -- 172.21.15.7:0/1525649089 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80af600 con 0x7fd7e00cb890 2023-11-28T18:23:16.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7df7fe640 1 -- 172.21.15.7:0/1525649089 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d80ab050 con 0x7fd7e00cb890 2023-11-28T18:23:16.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7df7fe640 1 -- 172.21.15.7:0/1525649089 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80af980 con 0x7fd7e00cb890 2023-11-28T18:23:16.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5828cbb640 1 -- start start 2023-11-28T18:23:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fa7958ba640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa77c00fda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa788007fd0 tx=0x7fa78801cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba5c37d50f51f62 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880a5c70 con 0x7fa77c0051e0 2023-11-28T18:23:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7880a5dd0 con 0x7fa77c0051e0 2023-11-28T18:23:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fa787fff640 1 -- 172.21.15.7:0/1058346623 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880ad680 con 0x7fa77c0051e0 2023-11-28T18:23:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fa79731d640 1 -- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 msgr2=0x7fa77c00fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fa79731d640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa77c00fda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa788007fd0 tx=0x7fa78801cde0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1525649089 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 msgr2=0x7fd7e00ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7e6b8d640 1 --2- 172.21.15.7:0/1525649089 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e00ca500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009dc0 tx=0x7fd7d801f140 comp rx=0 tx=0).stop 2023-11-28T18:23:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1525649089 shutdown_connections 2023-11-28T18:23:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7e6b8d640 1 --2- 172.21.15.7:0/1525649089 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e00ca500 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1525649089 >> 172.21.15.7:0/1525649089 conn(0x7fd7e00cfaa0 msgr2=0x7fd7e00cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fbe2b0fe640 1 Processor -- start 2023-11-28T18:23:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5828cbb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 0x7f58241f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5828cbb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58240c7be0 0x7f582414e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5828cbb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5824158480 0x7f5824153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5828cbb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58241fa390 con 0x7f58240c5d70 2023-11-28T18:23:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5828cbb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5824153c40 con 0x7f58240c7be0 2023-11-28T18:23:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7fbe2b0fe640 1 -- start start 2023-11-28T18:23:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.552+0000 7f5828cbb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582414df10 con 0x7f5824158480 2023-11-28T18:23:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1525649089 shutdown_connections 2023-11-28T18:23:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f995ab3e640 1 Processor -- start 2023-11-28T18:23:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5821d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58240c7be0 0x7f582414e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5822d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5824158480 0x7f5824153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f582259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 0x7f58241f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5822d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5824158480 0x7f5824153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38632/0 (socket says 172.21.15.7:38632) 2023-11-28T18:23:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5822d9d640 1 -- 172.21.15.7:0/2095916829 learned_addr learned my addr 172.21.15.7:0/2095916829 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1525649089 wait complete. 2023-11-28T18:23:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f995ab3e640 1 -- start start 2023-11-28T18:23:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5334cd6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f5330072620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5334cd6640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f53300cb890 0x7f53300ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5334cd6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53300c4b10 con 0x7f5330154140 2023-11-28T18:23:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5334cd6640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533015a770 con 0x7f53300cb890 2023-11-28T18:23:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5334cd6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533015aa50 con 0x7f53300cb420 2023-11-28T18:23:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fcf267fc640 1 --2- 172.21.15.7:0/1630899701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf2004f900 0x7fcf20008db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0097c0 tx=0x7fcf1c01ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd73687f51e0faf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f532ed9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f53300cb890 0x7f53300ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f532e59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5330154140 0x7f5330159ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55192/0 (socket says 172.21.15.7:55192) 2023-11-28T18:23:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f532e59c640 1 -- 172.21.15.7:0/2540762232 learned_addr learned my addr 172.21.15.7:0/2540762232 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f532dd9b640 1 --2- 172.21.15.7:0/2540762232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f5330072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe2b0fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe24154140 0x7fbe24159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe2b0fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbe240cb860 0x7fbe240ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe2b0fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe240c4b10 con 0x7fbe240cb420 2023-11-28T18:23:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe2b0fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2415a770 con 0x7fbe240cb860 2023-11-28T18:23:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe2b0fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2415aa50 con 0x7fbe24154140 2023-11-28T18:23:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fcf25ffb640 1 -- 172.21.15.7:0/1630899701 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c0a87b0 con 0x7fcf2004f900 2023-11-28T18:23:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fcf25ffb640 1 -- 172.21.15.7:0/1630899701 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c0a8910 con 0x7fcf2004f900 2023-11-28T18:23:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fcf25ffb640 1 -- 172.21.15.7:0/1630899701 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c0a8ca0 con 0x7fcf2004f900 2023-11-28T18:23:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f532dd9b640 1 -- 172.21.15.7:0/2540762232 >> v2:172.21.15.193:3300/0 conn(0x7f53300cb890 msgr2=0x7f53300ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f532dd9b640 1 --2- 172.21.15.7:0/2540762232 >> v2:172.21.15.193:3300/0 conn(0x7f53300cb890 0x7f53300ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f532dd9b640 1 -- 172.21.15.7:0/2540762232 >> v1:172.21.15.7:6789/0 conn(0x7f5330154140 legacy=0x7f5330159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe23fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fbe240cb420 0x7fbe24072620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35634/0 (socket says 172.21.15.7:35634) 2023-11-28T18:23:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe23fff640 1 -- 172.21.15.7:0/2934643042 learned_addr learned my addr 172.21.15.7:0/2934643042 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe28e9a640 1 --2- 172.21.15.7:0/2934643042 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe24154140 0x7fbe24159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fcf2d654640 1 -- 172.21.15.7:0/1630899701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf2004f900 msgr2=0x7fcf20008db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fcf2d654640 1 --2- 172.21.15.7:0/1630899701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf2004f900 0x7fcf20008db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0097c0 tx=0x7fcf1c01ebd0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5821d9b640 1 -- 172.21.15.7:0/2095916829 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5824158480 msgr2=0x7f5824153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5821d9b640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5824158480 0x7f5824153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5821d9b640 1 -- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 msgr2=0x7f58241f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f995ab3e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99540ca2c0 0x7f99540bf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f995ab3e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995415ee20 0x7f99540bfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f995ab3e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f995415fc40 0x7f99540c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f995ab3e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99540c4450 con 0x7f995415fc40 2023-11-28T18:23:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f995ab3e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99540c17c0 con 0x7f995415ee20 2023-11-28T18:23:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f995ab3e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9954070760 con 0x7f99540ca2c0 2023-11-28T18:23:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fd7e6b8d640 1 Processor -- start 2023-11-28T18:23:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe28e9a640 1 -- 172.21.15.7:0/2934643042 >> v2:172.21.15.193:3300/0 conn(0x7fbe240cb860 msgr2=0x7fbe240ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe28e9a640 1 --2- 172.21.15.7:0/2934643042 >> v2:172.21.15.193:3300/0 conn(0x7fbe240cb860 0x7fbe240ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fa79731d640 1 -- 172.21.15.7:0/1058346623 shutdown_connections 2023-11-28T18:23:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5821d9b640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 0x7f58241f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5821d9b640 1 -- 172.21.15.7:0/2095916829 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5814009420 con 0x7f58240c7be0 2023-11-28T18:23:16.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f582259c640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58240c5d70 0x7f58241f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:16.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fa79731d640 1 --2- 172.21.15.7:0/1058346623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa77c00fda0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f99588da640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99540ca2c0 0x7f99540bf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f99588da640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99540ca2c0 0x7f99540bf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38660/0 (socket says 172.21.15.7:38660) 2023-11-28T18:23:16.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f99588da640 1 -- 172.21.15.7:0/2785711566 learned_addr learned my addr 172.21.15.7:0/2785711566 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f9953fff640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995415ee20 0x7f99540bfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fa79731d640 1 -- 172.21.15.7:0/1058346623 >> 172.21.15.7:0/1058346623 conn(0x7fa79006c2c0 msgr2=0x7fa7900662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f99590db640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f995415fc40 0x7f99540c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f5821d9b640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58240c7be0 0x7f582414e3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f581c00c7e0 tx=0x7f581c00cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c00a640 con 0x7f58240c7be0 2023-11-28T18:23:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c00a7a0 con 0x7f58240c7be0 2023-11-28T18:23:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c00ab10 con 0x7f58240c7be0 2023-11-28T18:23:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fa79731d640 1 -- 172.21.15.7:0/1058346623 shutdown_connections 2023-11-28T18:23:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99588da640 1 -- 172.21.15.7:0/2785711566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995415ee20 msgr2=0x7f99540bfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99588da640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995415ee20 0x7f99540bfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fa79731d640 1 -- 172.21.15.7:0/1058346623 wait complete. 2023-11-28T18:23:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99588da640 1 -- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f995415fc40 msgr2=0x7f99540c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99588da640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f995415fc40 0x7f99540c4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99588da640 1 -- 172.21.15.7:0/2785711566 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9944009420 con 0x7f99540ca2c0 2023-11-28T18:23:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58241f6570 con 0x7f58240c7be0 2023-11-28T18:23:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58241f6810 con 0x7f58240c7be0 2023-11-28T18:23:16.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f9953fff640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995415ee20 0x7f99540bfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:16.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99590db640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f995415fc40 0x7f99540c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:16.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99588da640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99540ca2c0 0x7f99540bf6f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f993c00c520 tx=0x7f993c00c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58240c7780 con 0x7f58240c7be0 2023-11-28T18:23:16.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7f532dd9b640 1 -- 172.21.15.7:0/2540762232 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533015ad30 con 0x7f53300cb420 2023-11-28T18:23:16.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0074e0 con 0x7f99540ca2c0 2023-11-28T18:23:16.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c007640 con 0x7f99540ca2c0 2023-11-28T18:23:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0074e0 con 0x7f99540ca2c0 2023-11-28T18:23:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99540c1200 con 0x7f99540ca2c0 2023-11-28T18:23:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fa79731d640 1 Processor -- start 2023-11-28T18:23:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99540c4070 con 0x7f99540ca2c0 2023-11-28T18:23:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fa79731d640 1 -- start start 2023-11-28T18:23:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe28e9a640 1 -- 172.21.15.7:0/2934643042 >> v1:172.21.15.193:6789/0 conn(0x7fbe240cb420 legacy=0x7fbe24072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fbe28e9a640 1 -- 172.21.15.7:0/2934643042 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2415ad30 con 0x7fbe24154140 2023-11-28T18:23:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f99540ca0e0 con 0x7f99540ca2c0 2023-11-28T18:23:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.556+0000 7fd7e6b8d640 1 -- start start 2023-11-28T18:23:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e6b8d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e00cb420 0x7fd7e01551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e6b8d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e0157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e6b8d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e0154140 0x7fd7e00ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e6b8d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e0156450 con 0x7fd7e0154140 2023-11-28T18:23:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e6b8d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e015a170 con 0x7fd7e00cb420 2023-11-28T18:23:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e6b8d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e01598f0 con 0x7fd7e00cb890 2023-11-28T18:23:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e0157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e4929640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e00cb420 0x7fd7e01551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fcf2d654640 1 -- 172.21.15.7:0/1630899701 shutdown_connections 2023-11-28T18:23:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fcf2d654640 1 --2- 172.21.15.7:0/1630899701 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf20001470 0x7fcf2004ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fcf2d654640 1 --2- 172.21.15.7:0/1630899701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf2004f900 0x7fcf20008db0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fcf2d654640 1 -- 172.21.15.7:0/1630899701 >> 172.21.15.7:0/1630899701 conn(0x7fcf20050730 msgr2=0x7fcf20050b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e0157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38664/0 (socket says 172.21.15.7:38664) 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 -- 172.21.15.7:0/1455031909 learned_addr learned my addr 172.21.15.7:0/1455031909 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e512a640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e0154140 0x7fd7e00ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 -- 172.21.15.7:0/1455031909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e00cb420 msgr2=0x7fd7e01551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e00cb420 0x7fd7e01551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 -- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e0154140 msgr2=0x7fd7e00ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e0154140 0x7fd7e00ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 -- 172.21.15.7:0/1455031909 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d8009a70 con 0x7fd7e00cb890 2023-11-28T18:23:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7e512a640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e0154140 0x7fd7e00ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fd7dffff640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e0157120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00c480 tx=0x7fd7cc00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.560+0000 7fbe28e9a640 1 --2- 172.21.15.7:0/2934643042 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe24154140 0x7fbe24159ec0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbe100097c0 tx=0x7fbe1001ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a151ab1cec2dbe66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fcf2d654640 1 -- 172.21.15.7:0/1630899701 shutdown_connections 2023-11-28T18:23:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f532dd9b640 1 --2- 172.21.15.7:0/2540762232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f5330072620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5328009d90 tx=0x7f532801c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c255da761b9edff0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f581c00ac70 con 0x7f58240c7be0 2023-11-28T18:23:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe237fe640 1 -- 172.21.15.7:0/2934643042 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe100a8970 con 0x7fbe24154140 2023-11-28T18:23:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe237fe640 1 -- 172.21.15.7:0/2934643042 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe100a8ad0 con 0x7fbe24154140 2023-11-28T18:23:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe237fe640 1 -- 172.21.15.7:0/2934643042 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe100a8e60 con 0x7fbe24154140 2023-11-28T18:23:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa79731d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa7900c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa79731d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa7900688c0 0x7fa7901f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa79731d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa790153dd0 0x7fa790154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa79731d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa790158630 con 0x7fa790153dd0 2023-11-28T18:23:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa79731d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7900ca040 con 0x7fa77c0051e0 2023-11-28T18:23:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa79731d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901599b0 con 0x7fa7900688c0 2023-11-28T18:23:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7948b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa7900688c0 0x7fa7901f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc011790 con 0x7fd7e00cb890 2023-11-28T18:23:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7cc011910 con 0x7fd7e00cb890 2023-11-28T18:23:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc011cc0 con 0x7fd7e00cb890 2023-11-28T18:23:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e0159bb0 con 0x7fd7e00cb890 2023-11-28T18:23:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e0156640 con 0x7fd7e00cb890 2023-11-28T18:23:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fcf2d654640 1 -- 172.21.15.7:0/1630899701 wait complete. 2023-11-28T18:23:16.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f532d59a640 1 -- 172.21.15.7:0/2540762232 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53280a5890 con 0x7f53300cb420 2023-11-28T18:23:16.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f532d59a640 1 -- 172.21.15.7:0/2540762232 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53280a59f0 con 0x7f53300cb420 2023-11-28T18:23:16.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fb763432640 1 Processor -- start 2023-11-28T18:23:16.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fb763432640 1 -- start start 2023-11-28T18:23:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5812ffd640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5804066900 0x7f5804068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f532d59a640 1 -- 172.21.15.7:0/2540762232 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53280a5d80 con 0x7f53300cb420 2023-11-28T18:23:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/2934643042 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe24154140 msgr2=0x7fbe24159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe2b0fe640 1 --2- 172.21.15.7:0/2934643042 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe24154140 0x7fbe24159ec0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbe100097c0 tx=0x7fbe1001ecb0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7948b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa7900688c0 0x7fa7901f1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38668/0 (socket says 172.21.15.7:38668) 2023-11-28T18:23:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7948b8640 1 -- 172.21.15.7:0/3290607307 learned_addr learned my addr 172.21.15.7:0/3290607307 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7950b9640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa7900c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5334cd6640 1 -- 172.21.15.7:0/2540762232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 msgr2=0x7f5330072620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5334cd6640 1 --2- 172.21.15.7:0/2540762232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f5330072620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5328009d90 tx=0x7f532801c870 comp rx=0 tx=0).stop 2023-11-28T18:23:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f581c0839b0 con 0x7f58240c7be0 2023-11-28T18:23:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/2934643042 shutdown_connections 2023-11-28T18:23:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe2b0fe640 1 --2- 172.21.15.7:0/2934643042 >> v2:172.21.15.193:3300/0 conn(0x7fbe240cb860 0x7fbe240ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe2b0fe640 1 --2- 172.21.15.7:0/2934643042 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe24154140 0x7fbe24159ec0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/2934643042 >> 172.21.15.7:0/2934643042 conn(0x7fbe240cfaa0 msgr2=0x7fbe240cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/2934643042 shutdown_connections 2023-11-28T18:23:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7950b9640 1 -- 172.21.15.7:0/3290607307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa7900688c0 msgr2=0x7fa7901f1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7950b9640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa7900688c0 0x7fa7901f1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7950b9640 1 -- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa790153dd0 msgr2=0x7fa790154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7950b9640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa790153dd0 0x7fa790154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7950b9640 1 -- 172.21.15.7:0/3290607307 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7900cad30 con 0x7fa77c0051e0 2023-11-28T18:23:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5812ffd640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6810/1682271586,v1:172.21.15.7:6812/1682271586] conn(0x7f580406eb20 0x7f5804070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 --> [v2:172.21.15.7:6810/1682271586,v1:172.21.15.7:6812/1682271586] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58040715f0 con 0x7f580406eb20 2023-11-28T18:23:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f581c0882a0 con 0x7f58240c7be0 2023-11-28T18:23:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f582259c640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5804066900 0x7f5804068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa7950b9640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa7900c8f70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa780011ca0 tx=0x7fa78000f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f993c0047f0 con 0x7f99540ca2c0 2023-11-28T18:23:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fcf2d654640 1 Processor -- start 2023-11-28T18:23:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fcf2d654640 1 -- start start 2023-11-28T18:23:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5334cd6640 1 -- 172.21.15.7:0/2540762232 shutdown_connections 2023-11-28T18:23:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5334cd6640 1 --2- 172.21.15.7:0/2540762232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f5330072620 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5334cd6640 1 --2- 172.21.15.7:0/2540762232 >> v2:172.21.15.193:3300/0 conn(0x7f53300cb890 0x7f53300ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5334cd6640 1 -- 172.21.15.7:0/2540762232 >> 172.21.15.7:0/2540762232 conn(0x7f53300cfaa0 msgr2=0x7f53300cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7f5822d9d640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6810/1682271586,v1:172.21.15.7:6812/1682271586] conn(0x7f580406eb20 0x7f5804070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780009e60 con 0x7fa77c0051e0 2023-11-28T18:23:16.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78000fcc0 con 0x7fa77c0051e0 2023-11-28T18:23:16.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780004020 con 0x7fa77c0051e0 2023-11-28T18:23:16.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7901f29a0 con 0x7fa77c0051e0 2023-11-28T18:23:16.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f582259c640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5804066900 0x7f5804068dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5818009eb0 tx=0x7f5818009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7900ca530 con 0x7fa77c0051e0 2023-11-28T18:23:16.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf2d654640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf20001090 0x7fcf200d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf2d654640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf20001470 0x7fcf20037040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf2d654640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf2004f900 0x7fcf200cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/2934643042 wait complete. 2023-11-28T18:23:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf2d654640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2004d3a0 con 0x7fcf20001090 2023-11-28T18:23:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf2d654640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2004b590 con 0x7fcf20001470 2023-11-28T18:23:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf2d654640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2004b0a0 con 0x7fcf2004f900 2023-11-28T18:23:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.564+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7e0072100 con 0x7fd7e00cb890 2023-11-28T18:23:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f5822d9d640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6810/1682271586,v1:172.21.15.7:6812/1682271586] conn(0x7f580406eb20 0x7f5804070f40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f99517fa640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9934066990 0x7f9934068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf2004f900 0x7fcf200cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf2004f900 0x7fcf200cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38670/0 (socket says 172.21.15.7:38670) 2023-11-28T18:23:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 -- 172.21.15.7:0/2249315208 learned_addr learned my addr 172.21.15.7:0/2249315208 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f993c054730 con 0x7f99540ca2c0 2023-11-28T18:23:16.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa79004e890 con 0x7fa77c0051e0 2023-11-28T18:23:16.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f5334cd6640 1 -- 172.21.15.7:0/2540762232 shutdown_connections 2023-11-28T18:23:16.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f99517fa640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6808/2282553034,v1:172.21.15.7:6809/2282553034] conn(0x7f993406f790 0x7f9934071c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 --> [v2:172.21.15.7:6808/2282553034,v1:172.21.15.7:6809/2282553034] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9934072300 con 0x7f993406f790 2023-11-28T18:23:16.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f993c0074e0 con 0x7f99540ca2c0 2023-11-28T18:23:16.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f9953fff640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9934066990 0x7f9934068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fb763432640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 0x7fb75c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fb763432640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fb75c0c8060 0x7fb75c0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fb763432640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c0c8e80 con 0x7fb75c0c9990 2023-11-28T18:23:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fb763432640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb75c0c8610 con 0x7fb75c0c8060 2023-11-28T18:23:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fb763432640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c0c8890 con 0x7fb75c151f20 2023-11-28T18:23:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 -- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf20001470 msgr2=0x7fcf20037040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T18:23:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf20001470 0x7fcf20037040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 -- 172.21.15.7:0/2249315208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf20001090 msgr2=0x7fcf200d1a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T18:23:16.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fbe2b0fe640 1 Processor -- start 2023-11-28T18:23:16.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f99590db640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6808/2282553034,v1:172.21.15.7:6809/2282553034] conn(0x7f993406f790 0x7f9934071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf20001090 0x7fcf200d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 -- 172.21.15.7:0/2249315208 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c009420 con 0x7fcf2004f900 2023-11-28T18:23:16.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd7cc03e460 con 0x7fd7e00cb890 2023-11-28T18:23:16.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 <== osd.0 v2:172.21.15.7:6810/1682271586 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f58040715f0 con 0x7f580406eb20 2023-11-28T18:23:16.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f9953fff640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9934066990 0x7f9934068e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f99440b2000 tx=0x7f99440aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fcf27fff640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf2004f900 0x7fcf200cfd00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf2806a140 tx=0x7fcf28068290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fb7609cd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb75c0c9990 0x7fb75c0cd5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55196/0 (socket says 172.21.15.7:55196) 2023-11-28T18:23:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fb7609cd640 1 -- 172.21.15.7:0/4117997604 learned_addr learned my addr 172.21.15.7:0/4117997604 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fbe2b0fe640 1 -- start start 2023-11-28T18:23:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28073690 con 0x7fcf2004f900 2023-11-28T18:23:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2806b970 con 0x7fcf2004f900 2023-11-28T18:23:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2806bd20 con 0x7fcf2004f900 2023-11-28T18:23:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2004b830 con 0x7fcf2004f900 2023-11-28T18:23:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2004bdf0 con 0x7fcf2004f900 2023-11-28T18:23:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7f5334cd6640 1 -- 172.21.15.7:0/2540762232 wait complete. 2023-11-28T18:23:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fd7dd7fa640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7c40669d0 0x7fd7c4068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7f5334cd6640 1 Processor -- start 2023-11-28T18:23:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7cc0a6350 con 0x7fd7e00cb890 2023-11-28T18:23:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200154906 0 0) 0x7fb75c0c8e80 con 0x7fb75c0c9990 2023-11-28T18:23:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.568+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa780004180 con 0x7fa77c0051e0 2023-11-28T18:23:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fa785ffb640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa7740669a0 0x7fa774068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa780084980 con 0x7fa77c0051e0 2023-11-28T18:23:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fa785ffb640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6824/927014245,v1:172.21.15.7:6825/927014245] conn(0x7fa77406ebc0 0x7fa774070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 --> [v2:172.21.15.7:6824/927014245,v1:172.21.15.7:6825/927014245] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa774071690 con 0x7fa77406ebc0 2023-11-28T18:23:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7800892a0 con 0x7fa77c0051e0 2023-11-28T18:23:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7f99590db640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6808/2282553034,v1:172.21.15.7:6809/2282553034] conn(0x7f993406f790 0x7f9934071c30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fd7dd7fa640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6800/2659836262,v1:172.21.15.7:6801/2659836262] conn(0x7fd7c406eb70 0x7fd7c4070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 --> [v2:172.21.15.7:6800/2659836262,v1:172.21.15.7:6801/2659836262] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c4071640 con 0x7fd7c406eb70 2023-11-28T18:23:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf20001fb0 con 0x7fcf2004f900 2023-11-28T18:23:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe2b0fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe240cb420 0x7fbe241f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe2b0fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe240cb860 0x7fbe2415b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe2b0fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe24154140 0x7fbe240cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe2b0fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2415a2c0 con 0x7fbe24154140 2023-11-28T18:23:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb748003530 con 0x7fb75c0c9990 2023-11-28T18:23:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb7619cf640 1 --2- 172.21.15.7:0/4117997604 >> v2:172.21.15.193:3300/0 conn(0x7fb75c0c8060 0x7fb75c0c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe23fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe240cb860 0x7fbe2415b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe23fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe240cb860 0x7fbe2415b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38676/0 (socket says 172.21.15.7:38676) 2023-11-28T18:23:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe23fff640 1 -- 172.21.15.7:0/1549987575 learned_addr learned my addr 172.21.15.7:0/1549987575 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 <== osd.2 v2:172.21.15.7:6808/2282553034 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9934072300 con 0x7f993406f790 2023-11-28T18:23:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fcf280685e0 con 0x7fcf2004f900 2023-11-28T18:23:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe28e9a640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe240cb420 0x7fbe241f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe241f4270 con 0x7fbe240cb420 2023-11-28T18:23:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe241f3c80 con 0x7fbe240cb860 2023-11-28T18:23:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe2969b640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe24154140 0x7fbe240cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb7611ce640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 0x7fb75c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2333012272 0 0) 0x7fb748003530 con 0x7fb75c0c9990 2023-11-28T18:23:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 msgr2=0x7fb75c15b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 0x7fb75c15b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 >> v2:172.21.15.193:3300/0 conn(0x7fb75c0c8060 msgr2=0x7fb75c0c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 --2- 172.21.15.7:0/4117997604 >> v2:172.21.15.193:3300/0 conn(0x7fb75c0c8060 0x7fb75c0c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c072b60 con 0x7fb75c0c9990 2023-11-28T18:23:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fb74c001680 con 0x7fb75c0c9990 2023-11-28T18:23:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 >> v1:172.21.15.7:6789/0 conn(0x7fb75c0c9990 legacy=0x7fb75c0cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb748003980 0x7fb748003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 0x7fb7480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb748004d10 0x7fb748005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c0c8610 con 0x7fb748004d10 2023-11-28T18:23:16.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c0c8890 con 0x7fb748003980 2023-11-28T18:23:16.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74800bac0 con 0x7fb75c151f20 2023-11-28T18:23:16.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7cc0a4c20 con 0x7fd7e00cb890 2023-11-28T18:23:16.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fd7e4929640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7c40669d0 0x7fd7c4068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe23fff640 1 -- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe240cb420 msgr2=0x7fbe241f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe23fff640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe240cb420 0x7fbe241f4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe23fff640 1 -- 172.21.15.7:0/1549987575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe24154140 msgr2=0x7fbe240cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe23fff640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe24154140 0x7fbe240cee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fbe23fff640 1 -- 172.21.15.7:0/1549987575 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe10009420 con 0x7fbe240cb860 2023-11-28T18:23:16.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fbe23fff640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe240cb860 0x7fbe2415b0f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800c530 tx=0x7fbe1800c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fb7609cd640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 0x7fb7480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fa7948b8640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa7740669a0 0x7fa774068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.572+0000 7fa7958ba640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6824/927014245,v1:172.21.15.7:6825/927014245] conn(0x7fa77406ebc0 0x7fa774070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fa7948b8640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa7740669a0 0x7fa774068e60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa79013dc30 tx=0x7fa78c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18005cf0 con 0x7fbe240cb860 2023-11-28T18:23:16.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fb7611ce640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb748003980 0x7fb748003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fa7958ba640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6824/927014245,v1:172.21.15.7:6825/927014245] conn(0x7fa77406ebc0 0x7fa774070fe0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fd7e512a640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6800/2659836262,v1:172.21.15.7:6801/2659836262] conn(0x7fd7c406eb70 0x7fd7c4070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe18005e50 con 0x7fbe240cb860 2023-11-28T18:23:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe241597d0 con 0x7fbe240cb860 2023-11-28T18:23:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18007270 con 0x7fbe240cb860 2023-11-28T18:23:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe24159d00 con 0x7fbe240cb860 2023-11-28T18:23:16.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fb7609cd640 1 -- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb748003980 msgr2=0x7fb748003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fb7609cd640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb748003980 0x7fb748003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fb7609cd640 1 -- 172.21.15.7:0/4117997604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb748004d10 msgr2=0x7fb748005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fb7609cd640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb748004d10 0x7fb748005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fb7609cd640 1 -- 172.21.15.7:0/4117997604 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c072b60 con 0x7fb75c151f20 2023-11-28T18:23:16.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f5334cd6640 1 -- start start 2023-11-28T18:23:16.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fcf0f7fe640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf14066920 0x7fcf14068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f5334cd6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f53301480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f5334cd6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53300cb890 0x7f5330148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbe24072100 con 0x7fbe240cb860 2023-11-28T18:23:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fd7e4929640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7c40669d0 0x7fd7c4068e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4000f70 tx=0x7fd7d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fd7e512a640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6800/2659836262,v1:172.21.15.7:6801/2659836262] conn(0x7fd7c406eb70 0x7fd7c4070f90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f5334cd6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5330154140 0x7f53301fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f5334cd6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53301fcb20 con 0x7f5330154140 2023-11-28T18:23:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f5334cd6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330147a30 con 0x7f53300cb890 2023-11-28T18:23:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f5334cd6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53301fc4f0 con 0x7f53300cb420 2023-11-28T18:23:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf280ff5b0 con 0x7fcf2004f900 2023-11-28T18:23:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fcf26ffd640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf14066920 0x7fcf14068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532e59c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f53301480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532e59c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f53301480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38688/0 (socket says 172.21.15.7:38688) 2023-11-28T18:23:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532e59c640 1 -- 172.21.15.7:0/854053888 learned_addr learned my addr 172.21.15.7:0/854053888 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532dd9b640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53300cb890 0x7f5330148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532ed9d640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5330154140 0x7f53301fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532e59c640 1 -- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53300cb890 msgr2=0x7f5330148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532e59c640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53300cb890 0x7f5330148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532e59c640 1 -- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5330154140 msgr2=0x7f53301fcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532e59c640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5330154140 0x7f53301fcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532e59c640 1 -- 172.21.15.7:0/854053888 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5328009a70 con 0x7f53300cb420 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7f532ed9d640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5330154140 0x7f53301fcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fcf0f7fe640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:6817/2413083632,v1:172.21.15.193:6819/2413083632] conn(0x7fcf1406eb40 0x7fcf14070f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 --> [v2:172.21.15.193:6817/2413083632,v1:172.21.15.193:6819/2413083632] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf14071610 con 0x7fcf1406eb40 2023-11-28T18:23:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf28101b90 con 0x7fcf2004f900 2023-11-28T18:23:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 <== osd.3 v2:172.21.15.7:6800/2659836262 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd7c4071640 con 0x7fd7c406eb70 2023-11-28T18:23:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.576+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbe180073d0 con 0x7fbe240cb860 2023-11-28T18:23:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 <== osd.1 v2:172.21.15.7:6824/927014245 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa774071690 con 0x7fa77406ebc0 2023-11-28T18:23:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7f532e59c640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f53301480c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5324009e20 tx=0x7f532400c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53240078f0 con 0x7f53300cb420 2023-11-28T18:23:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5324007a50 con 0x7f53300cb420 2023-11-28T18:23:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324007e20 con 0x7f53300cb420 2023-11-28T18:23:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fbe217fa640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe080668d0 0x7fbe08068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe180838b0 con 0x7fbe240cb860 2023-11-28T18:23:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fbe217fa640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:6816/667378636,v1:172.21.15.193:6818/667378636] conn(0x7fbe0806ea70 0x7fbe08070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 --> [v2:172.21.15.193:6816/667378636,v1:172.21.15.193:6818/667378636] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe08071540 con 0x7fbe0806ea70 2023-11-28T18:23:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5330200510 con 0x7f53300cb420 2023-11-28T18:23:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53301fc7b0 con 0x7f53300cb420 2023-11-28T18:23:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fcf267fc640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:6817/2413083632,v1:172.21.15.193:6819/2413083632] conn(0x7fcf1406eb40 0x7fcf14070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fbe28e9a640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe080668d0 0x7fbe08068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.580+0000 7fbe2969b640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:6816/667378636,v1:172.21.15.193:6818/667378636] conn(0x7fbe0806ea70 0x7fbe08070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fcf267fc640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:6817/2413083632,v1:172.21.15.193:6819/2413083632] conn(0x7fcf1406eb40 0x7fcf14070f60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fcf26ffd640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf14066920 0x7fcf14068de0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf18001560 tx=0x7fcf18007a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 <== osd.4 v2:172.21.15.193:6817/2413083632 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcf14071610 con 0x7fcf1406eb40 2023-11-28T18:23:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fbe2969b640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:6816/667378636,v1:172.21.15.193:6818/667378636] conn(0x7fbe0806ea70 0x7fbe08070e90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fbe28e9a640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe080668d0 0x7fbe08068d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe100b2000 tx=0x7fbe100aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe18050b90 con 0x7fbe240cb860 2023-11-28T18:23:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5330072100 con 0x7f53300cb420 2023-11-28T18:23:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5324009070 con 0x7f53300cb420 2023-11-28T18:23:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 <== osd.6 v2:172.21.15.193:6816/667378636 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fbe08071540 con 0x7fbe0806ea70 2023-11-28T18:23:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fb7609cd640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 0x7fb7480047d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c260 tx=0x7fb74c01cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6f8b74ee9b5d6d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c0a5960 con 0x7fb75c151f20 2023-11-28T18:23:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c0a5ac0 con 0x7fb75c151f20 2023-11-28T18:23:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.584+0000 7f531effd640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5310066950 0x7f5310068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7fb753fff640 1 -- 172.21.15.7:0/4117997604 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c0a5960 con 0x7fb75c151f20 2023-11-28T18:23:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7fb763432640 1 -- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 msgr2=0x7fb7480047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7fb763432640 1 --2- 172.21.15.7:0/4117997604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c151f20 0x7fb7480047d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c260 tx=0x7fb74c01cc60 comp rx=0 tx=0).stop 2023-11-28T18:23:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7fb7619cf640 1 -- 172.21.15.7:0/4117997604 reap_dead start 2023-11-28T18:23:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7fb763432640 1 -- 172.21.15.7:0/4117997604 shutdown_connections 2023-11-28T18:23:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7fb763432640 1 -- 172.21.15.7:0/4117997604 >> 172.21.15.7:0/4117997604 conn(0x7fb75c06ff10 msgr2=0x7fb75c070320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7fb763432640 1 -- 172.21.15.7:0/4117997604 shutdown_connections 2023-11-28T18:23:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7fb763432640 1 -- 172.21.15.7:0/4117997604 wait complete. 2023-11-28T18:23:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5324057690 con 0x7f53300cb420 2023-11-28T18:23:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7f531effd640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:6800/4136487123,v1:172.21.15.193:6801/4136487123] conn(0x7f531006eb70 0x7f5310070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7f532dd9b640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5310066950 0x7f5310068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 --> [v2:172.21.15.193:6800/4136487123,v1:172.21.15.193:6801/4136487123] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5310071640 con 0x7f531006eb70 2023-11-28T18:23:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5324009bf0 con 0x7f53300cb420 2023-11-28T18:23:16.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.588+0000 7f532ed9d640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:6800/4136487123,v1:172.21.15.193:6801/4136487123] conn(0x7f531006eb70 0x7f5310070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb763432640 1 Processor -- start 2023-11-28T18:23:16.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb763432640 1 -- start start 2023-11-28T18:23:16.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 --> [v2:172.21.15.7:6810/1682271586,v1:172.21.15.7:6812/1682271586] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f582404e890 con 0x7f580406eb20 2023-11-28T18:23:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb763432640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb75c151f20 0x7fb75c14a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb763432640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb75c148a80 0x7fb75c153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7f532dd9b640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5310066950 0x7f5310068e10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f53280af000 tx=0x7f53280a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb763432640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c154280 0x7fb75c154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb763432640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c13eb30 con 0x7fb75c151f20 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb763432640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c154700 con 0x7fb75c148a80 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb763432640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c13d700 con 0x7fb75c154280 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb75c148a80 0x7fb75c153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb75c148a80 0x7fb75c153d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50386/0 (socket says 172.21.15.7:50386) 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 -- 172.21.15.7:0/1426304861 learned_addr learned my addr 172.21.15.7:0/1426304861 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7619cf640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c154280 0x7fb75c154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7611ce640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb75c151f20 0x7fb75c14a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7f532ed9d640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:6800/4136487123,v1:172.21.15.193:6801/4136487123] conn(0x7f531006eb70 0x7f5310070f90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 -- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c154280 msgr2=0x7fb75c154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb75c154280 0x7fb75c154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 -- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb75c151f20 msgr2=0x7fb75c14a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb75c151f20 0x7fb75c14a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 -- 172.21.15.7:0/1426304861 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c13d9a0 con 0x7fb75c148a80 2023-11-28T18:23:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7611ce640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb75c151f20 0x7fb75c14a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 <== osd.5 v2:172.21.15.193:6800/4136487123 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5310071640 con 0x7f531006eb70 2023-11-28T18:23:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.592+0000 7fb7609cd640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb75c148a80 0x7fb75c153d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb74c01ca90 tx=0x7fb748005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c0a5960 con 0x7fb75c148a80 2023-11-28T18:23:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c0070d0 con 0x7fb75c148a80 2023-11-28T18:23:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c007440 con 0x7fb75c148a80 2023-11-28T18:23:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c13dc40 con 0x7fb75c148a80 2023-11-28T18:23:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c13e200 con 0x7fb75c148a80 2023-11-28T18:23:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb75c073240 con 0x7fb75c148a80 2023-11-28T18:23:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7f5812ffd640 1 -- 172.21.15.7:0/2095916829 <== osd.0 v2:172.21.15.7:6810/1682271586 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f582404e890 con 0x7f580406eb20 2023-11-28T18:23:16.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6810/1682271586,v1:172.21.15.7:6812/1682271586] conn(0x7f580406eb20 msgr2=0x7f5804070f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.596+0000 7f5828cbb640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6810/1682271586,v1:172.21.15.7:6812/1682271586] conn(0x7f580406eb20 0x7f5804070f40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5804066900 msgr2=0x7f5804068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5828cbb640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5804066900 0x7f5804068dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5818009eb0 tx=0x7f5818009190 comp rx=0 tx=0).stop 2023-11-28T18:23:16.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58240c7be0 msgr2=0x7f582414e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5828cbb640 1 --2- 172.21.15.7:0/2095916829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58240c7be0 0x7f582414e3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f581c00c7e0 tx=0x7f581c00cca0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5822d9d640 1 -- 172.21.15.7:0/2095916829 reap_dead start 2023-11-28T18:23:16.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 shutdown_connections 2023-11-28T18:23:16.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 >> 172.21.15.7:0/2095916829 conn(0x7f582406c2c0 msgr2=0x7f582406df40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 shutdown_connections 2023-11-28T18:23:16.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7f5828cbb640 1 -- 172.21.15.7:0/2095916829 wait complete. 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb74c0075a0 con 0x7fb75c148a80 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7fb751ffb640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb740066920 0x7fb740068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb74c11ef30 con 0x7fb75c148a80 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7fb7611ce640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb740066920 0x7fb740068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7fb751ffb640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:6802/1530886252,v1:172.21.15.193:6804/1530886252] conn(0x7fb74006eb40 0x7fb740070f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 --> [v2:172.21.15.193:6802/1530886252,v1:172.21.15.193:6804/1530886252] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb740071610 con 0x7fb74006eb40 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.600+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb74c11f2a0 con 0x7fb75c148a80 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.604+0000 7fb7619cf640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:6802/1530886252,v1:172.21.15.193:6804/1530886252] conn(0x7fb74006eb40 0x7fb740070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.604+0000 7fb7611ce640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb740066920 0x7fb740068de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb7580039b0 tx=0x7fb758008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.604+0000 7fb7619cf640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:6802/1530886252,v1:172.21.15.193:6804/1530886252] conn(0x7fb74006eb40 0x7fb740070f60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.604+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 <== osd.7 v2:172.21.15.193:6802/1530886252 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb740071610 con 0x7fb74006eb40 2023-11-28T18:23:16.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.604+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 --> [v2:172.21.15.7:6808/2282553034,v1:172.21.15.7:6809/2282553034] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9954162660 con 0x7f993406f790 2023-11-28T18:23:16.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7f99517fa640 1 -- 172.21.15.7:0/2785711566 <== osd.2 v2:172.21.15.7:6808/2282553034 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f994c013040 con 0x7f993406f790 2023-11-28T18:23:16.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6808/2282553034,v1:172.21.15.7:6809/2282553034] conn(0x7f993406f790 msgr2=0x7f9934071c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 --> [v2:172.21.15.7:6800/2659836262,v1:172.21.15.7:6801/2659836262] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd7e0071e20 con 0x7fd7c406eb70 2023-11-28T18:23:16.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7f995ab3e640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6808/2282553034,v1:172.21.15.7:6809/2282553034] conn(0x7f993406f790 0x7f9934071c30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 --> [v2:172.21.15.7:6824/927014245,v1:172.21.15.7:6825/927014245] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa7900c7b40 con 0x7fa77406ebc0 2023-11-28T18:23:16.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7dd7fa640 1 -- 172.21.15.7:0/1455031909 <== osd.3 v2:172.21.15.7:6800/2659836262 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd7d80af880 con 0x7fd7c406eb70 2023-11-28T18:23:16.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fa785ffb640 1 -- 172.21.15.7:0/3290607307 <== osd.1 v2:172.21.15.7:6824/927014245 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa7900c7b40 con 0x7fa77406ebc0 2023-11-28T18:23:16.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6800/2659836262,v1:172.21.15.7:6801/2659836262] conn(0x7fd7c406eb70 msgr2=0x7fd7c4070f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7e6b8d640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6800/2659836262,v1:172.21.15.7:6801/2659836262] conn(0x7fd7c406eb70 0x7fd7c4070f90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7c40669d0 msgr2=0x7fd7c4068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7e6b8d640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7c40669d0 0x7fd7c4068e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4000f70 tx=0x7fd7d4009040 comp rx=0 tx=0).stop 2023-11-28T18:23:16.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 msgr2=0x7fd7e0157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7e6b8d640 1 --2- 172.21.15.7:0/1455031909 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd7e00cb890 0x7fd7e0157120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00c480 tx=0x7fd7cc00c940 comp rx=0 tx=0).stop 2023-11-28T18:23:16.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.608+0000 7fd7e512a640 1 -- 172.21.15.7:0/1455031909 reap_dead start 2023-11-28T18:23:16.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6824/927014245,v1:172.21.15.7:6825/927014245] conn(0x7fa77406ebc0 msgr2=0x7fa774070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6824/927014245,v1:172.21.15.7:6825/927014245] conn(0x7fa77406ebc0 0x7fa774070fe0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9934066990 msgr2=0x7f9934068e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f995ab3e640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9934066990 0x7f9934068e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f99440b2000 tx=0x7f99440aa070 comp rx=0 tx=0).stop 2023-11-28T18:23:16.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99540ca2c0 msgr2=0x7f99540bf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f995ab3e640 1 --2- 172.21.15.7:0/2785711566 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99540ca2c0 0x7f99540bf6f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f993c00c520 tx=0x7f993c00c9e0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f99590db640 1 -- 172.21.15.7:0/2785711566 reap_dead start 2023-11-28T18:23:16.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 shutdown_connections 2023-11-28T18:23:16.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 >> 172.21.15.7:0/2785711566 conn(0x7f99540662e0 msgr2=0x7f99540c22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 shutdown_connections 2023-11-28T18:23:16.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7f995ab3e640 1 -- 172.21.15.7:0/2785711566 wait complete. 2023-11-28T18:23:16.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 shutdown_connections 2023-11-28T18:23:16.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 >> 172.21.15.7:0/1455031909 conn(0x7fd7e00cfaa0 msgr2=0x7fd7e00ca900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa7740669a0 msgr2=0x7fa774068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa7740669a0 0x7fa774068e60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa79013dc30 tx=0x7fa78c009040 comp rx=0 tx=0).stop 2023-11-28T18:23:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 msgr2=0x7fa7900c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 --2- 172.21.15.7:0/3290607307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77c0051e0 0x7fa7900c8f70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa780011ca0 tx=0x7fa78000f610 comp rx=0 tx=0).stop 2023-11-28T18:23:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa7958ba640 1 -- 172.21.15.7:0/3290607307 reap_dead start 2023-11-28T18:23:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 shutdown_connections 2023-11-28T18:23:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 >> 172.21.15.7:0/3290607307 conn(0x7fa79006c2c0 msgr2=0x7fa79006d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 shutdown_connections 2023-11-28T18:23:16.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fd7e6b8d640 1 -- 172.21.15.7:0/1455031909 wait complete. 2023-11-28T18:23:16.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 shutdown_connections 2023-11-28T18:23:16.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fa79731d640 1 -- 172.21.15.7:0/3290607307 wait complete. 2023-11-28T18:23:16.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 --> [v2:172.21.15.193:6817/2413083632,v1:172.21.15.193:6819/2413083632] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcf2004aa50 con 0x7fcf1406eb40 2023-11-28T18:23:16.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.612+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 --> [v2:172.21.15.193:6816/667378636,v1:172.21.15.193:6818/667378636] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbe24071e20 con 0x7fbe0806ea70 2023-11-28T18:23:16.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf0f7fe640 1 -- 172.21.15.7:0/2249315208 <== osd.4 v2:172.21.15.193:6817/2413083632 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcf1c0a8a80 con 0x7fcf1406eb40 2023-11-28T18:23:16.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe217fa640 1 -- 172.21.15.7:0/1549987575 <== osd.6 v2:172.21.15.193:6816/667378636 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbe1c009a70 con 0x7fbe0806ea70 2023-11-28T18:23:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:6817/2413083632,v1:172.21.15.193:6819/2413083632] conn(0x7fcf1406eb40 msgr2=0x7fcf14070f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:6817/2413083632,v1:172.21.15.193:6819/2413083632] conn(0x7fcf1406eb40 0x7fcf14070f60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf14066920 msgr2=0x7fcf14068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf14066920 0x7fcf14068de0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf18001560 tx=0x7fcf18007a50 comp rx=0 tx=0).stop 2023-11-28T18:23:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf2004f900 msgr2=0x7fcf200cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 --2- 172.21.15.7:0/2249315208 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcf2004f900 0x7fcf200cfd00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf2806a140 tx=0x7fcf28068290 comp rx=0 tx=0).stop 2023-11-28T18:23:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf27fff640 1 -- 172.21.15.7:0/2249315208 reap_dead start 2023-11-28T18:23:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 shutdown_connections 2023-11-28T18:23:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 >> 172.21.15.7:0/2249315208 conn(0x7fcf20050730 msgr2=0x7fcf200360c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 shutdown_connections 2023-11-28T18:23:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:6816/667378636,v1:172.21.15.193:6818/667378636] conn(0x7fbe0806ea70 msgr2=0x7fbe08070e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:6816/667378636,v1:172.21.15.193:6818/667378636] conn(0x7fbe0806ea70 0x7fbe08070e90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fcf2d654640 1 -- 172.21.15.7:0/2249315208 wait complete. 2023-11-28T18:23:16.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe080668d0 msgr2=0x7fbe08068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe080668d0 0x7fbe08068d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe100b2000 tx=0x7fbe100aa070 comp rx=0 tx=0).stop 2023-11-28T18:23:16.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe240cb860 msgr2=0x7fbe2415b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 --2- 172.21.15.7:0/1549987575 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe240cb860 0x7fbe2415b0f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800c530 tx=0x7fbe1800c9f0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2969b640 1 -- 172.21.15.7:0/1549987575 reap_dead start 2023-11-28T18:23:16.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 shutdown_connections 2023-11-28T18:23:16.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 >> 172.21.15.7:0/1549987575 conn(0x7fbe240cfaa0 msgr2=0x7fbe240c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 shutdown_connections 2023-11-28T18:23:16.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.616+0000 7fbe2b0fe640 1 -- 172.21.15.7:0/1549987575 wait complete. 2023-11-28T18:23:16.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.624+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 --> [v2:172.21.15.193:6800/4136487123,v1:172.21.15.193:6801/4136487123] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5330071e20 con 0x7f531006eb70 2023-11-28T18:23:16.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.624+0000 7f531effd640 1 -- 172.21.15.7:0/854053888 <== osd.5 v2:172.21.15.193:6800/4136487123 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5320009750 con 0x7f531006eb70 2023-11-28T18:23:16.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:6800/4136487123,v1:172.21.15.193:6801/4136487123] conn(0x7f531006eb70 msgr2=0x7f5310070f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:6800/4136487123,v1:172.21.15.193:6801/4136487123] conn(0x7f531006eb70 0x7f5310070f90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5310066950 msgr2=0x7f5310068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5310066950 0x7f5310068e10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f53280af000 tx=0x7f53280a7070 comp rx=0 tx=0).stop 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 msgr2=0x7f53301480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 --2- 172.21.15.7:0/854053888 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f53300cb420 0x7f53301480c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5324009e20 tx=0x7f532400c430 comp rx=0 tx=0).stop 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f532ed9d640 1 -- 172.21.15.7:0/854053888 reap_dead start 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 shutdown_connections 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 >> 172.21.15.7:0/854053888 conn(0x7f53300cfaa0 msgr2=0x7f53300c5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 shutdown_connections 2023-11-28T18:23:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.628+0000 7f5334cd6640 1 -- 172.21.15.7:0/854053888 wait complete. 2023-11-28T18:23:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.632+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 --> [v2:172.21.15.193:6802/1530886252,v1:172.21.15.193:6804/1530886252] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb75c0c8b10 con 0x7fb74006eb40 2023-11-28T18:23:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.632+0000 7fb751ffb640 1 -- 172.21.15.7:0/1426304861 <== osd.7 v2:172.21.15.193:6802/1530886252 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb754009c90 con 0x7fb74006eb40 2023-11-28T18:23:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.632+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:6802/1530886252,v1:172.21.15.193:6804/1530886252] conn(0x7fb74006eb40 msgr2=0x7fb740070f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.632+0000 7fb763432640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:6802/1530886252,v1:172.21.15.193:6804/1530886252] conn(0x7fb74006eb40 0x7fb740070f60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.632+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb740066920 msgr2=0x7fb740068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.632+0000 7fb763432640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb740066920 0x7fb740068de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb7580039b0 tx=0x7fb758008040 comp rx=0 tx=0).stop 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.632+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb75c148a80 msgr2=0x7fb75c153d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.632+0000 7fb763432640 1 --2- 172.21.15.7:0/1426304861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb75c148a80 0x7fb75c153d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb74c01ca90 tx=0x7fb748005800 comp rx=0 tx=0).stop 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.636+0000 7fb7619cf640 1 -- 172.21.15.7:0/1426304861 reap_dead start 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.636+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 shutdown_connections 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.636+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 >> 172.21.15.7:0/1426304861 conn(0x7fb75c06ff10 msgr2=0x7fb75c150720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.640+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 shutdown_connections 2023-11-28T18:23:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.640+0000 7fb763432640 1 -- 172.21.15.7:0/1426304861 wait complete. 2023-11-28T18:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:16.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T18:23:16.672 INFO:teuthology.orchestra.run.smithi007.stdout:55834574856 2023-11-28T18:23:16.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T18:23:16.673 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:16.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T18:23:16.674 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2023-11-28T18:23:16.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T18:23:16.675 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:16.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T18:23:16.676 INFO:teuthology.orchestra.run.smithi007.stdout:55834574856 2023-11-28T18:23:16.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T18:23:16.677 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:16.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T18:23:16.679 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:16.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T18:23:16.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.948+0000 7f3c19cd6640 1 Processor -- start 2023-11-28T18:23:16.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.948+0000 7f3c19cd6640 1 -- start start 2023-11-28T18:23:16.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.948+0000 7f3c19cd6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3c140c3090 0x7f3c14159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.948+0000 7f3c19cd6640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3c140c6170 0x7f3c140c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.948+0000 7f3c19cd6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c14155f10 con 0x7f3c1414fef0 2023-11-28T18:23:16.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.948+0000 7f3c19cd6640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c14155d20 con 0x7f3c140c3090 2023-11-28T18:23:16.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.948+0000 7f3c19cd6640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c141561d0 con 0x7f3c140c6170 2023-11-28T18:23:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f3c13fff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3c140c6170 0x7f3c140c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f3c137fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f3c1414fef0 0x7f3c140c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35642/0 (socket says 172.21.15.7:35642) 2023-11-28T18:23:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f3c137fe640 1 -- 172.21.15.7:0/1490076400 learned_addr learned my addr 172.21.15.7:0/1490076400 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f3c12ffd640 1 --2- 172.21.15.7:0/1490076400 >> v2:172.21.15.7:3300/0 conn(0x7f3c140c3090 0x7f3c14159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f3c12ffd640 1 -- 172.21.15.7:0/1490076400 >> v2:172.21.15.193:3300/0 conn(0x7f3c140c6170 msgr2=0x7f3c140c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f3c12ffd640 1 --2- 172.21.15.7:0/1490076400 >> v2:172.21.15.193:3300/0 conn(0x7f3c140c6170 0x7f3c140c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f3c12ffd640 1 -- 172.21.15.7:0/1490076400 >> v1:172.21.15.193:6789/0 conn(0x7f3c1414fef0 legacy=0x7f3c140c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f3c12ffd640 1 -- 172.21.15.7:0/1490076400 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c141564b0 con 0x7f3c140c3090 2023-11-28T18:23:16.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f4cd9e09640 1 Processor -- start 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.952+0000 7f4cd9e09640 1 -- start start 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd9e09640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4ccc0e71c0 0x7f4ccc046270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd9e09640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ccc0467b0 0x7f4ccc04ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd9e09640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ccc04c660 con 0x7f4ccc0dd290 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd9e09640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ccc046be0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd9e09640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ccc0454b0 con 0x7f4ccc0e71c0 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ccc0467b0 0x7f4ccc04ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f4ccc0dd290 0x7f4ccc0e66c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35644/0 (socket says 172.21.15.7:35644) 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd37fe640 1 -- 172.21.15.7:0/3043934519 learned_addr learned my addr 172.21.15.7:0/3043934519 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd2ffd640 1 --2- 172.21.15.7:0/3043934519 >> v2:172.21.15.193:3300/0 conn(0x7f4ccc0e71c0 0x7f4ccc046270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd3fff640 1 -- 172.21.15.7:0/3043934519 >> v2:172.21.15.193:3300/0 conn(0x7f4ccc0e71c0 msgr2=0x7f4ccc046270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd3fff640 1 --2- 172.21.15.7:0/3043934519 >> v2:172.21.15.193:3300/0 conn(0x7f4ccc0e71c0 0x7f4ccc046270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd3fff640 1 -- 172.21.15.7:0/3043934519 >> v1:172.21.15.193:6789/0 conn(0x7f4ccc0dd290 legacy=0x7f4ccc0e66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.956+0000 7f4cd3fff640 1 -- 172.21.15.7:0/3043934519 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc045790 con 0x7f4ccc0467b0 2023-11-28T18:23:16.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7f3c12ffd640 1 --2- 172.21.15.7:0/1490076400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c140c3090 0x7f3c14159cd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c14073390 tx=0x7f3c0401e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0e9d3754af36d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7f3c127fc640 1 -- 172.21.15.7:0/1490076400 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c040ad610 con 0x7f3c140c3090 2023-11-28T18:23:16.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7f3c127fc640 1 -- 172.21.15.7:0/1490076400 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c040ad770 con 0x7f3c140c3090 2023-11-28T18:23:16.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7f3c127fc640 1 -- 172.21.15.7:0/1490076400 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c040adb00 con 0x7f3c140c3090 2023-11-28T18:23:16.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7f4cd3fff640 1 --2- 172.21.15.7:0/3043934519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc04ba40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40695f0 tx=0x7f4cd409a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4575b40351fec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7f4cd27fc640 1 -- 172.21.15.7:0/3043934519 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4123dc0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7f4cd27fc640 1 -- 172.21.15.7:0/3043934519 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd412b440 con 0x7f4ccc0467b0 2023-11-28T18:23:16.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7f4cd27fc640 1 -- 172.21.15.7:0/3043934519 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4123dc0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7fae443d8640 1 Processor -- start 2023-11-28T18:23:16.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.960+0000 7fae443d8640 1 -- start start 2023-11-28T18:23:16.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 -- 172.21.15.7:0/1490076400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c140c3090 msgr2=0x7f3c14159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 --2- 172.21.15.7:0/1490076400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c140c3090 0x7f3c14159cd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c14073390 tx=0x7f3c0401e490 comp rx=0 tx=0).stop 2023-11-28T18:23:16.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 -- 172.21.15.7:0/1490076400 shutdown_connections 2023-11-28T18:23:16.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 --2- 172.21.15.7:0/1490076400 >> v2:172.21.15.193:3300/0 conn(0x7f3c140c6170 0x7f3c140c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 --2- 172.21.15.7:0/1490076400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c140c3090 0x7f3c14159cd0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c14073390 tx=0x7f3c0401e490 comp rx=0 tx=0).stop 2023-11-28T18:23:16.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 -- 172.21.15.7:0/1490076400 >> 172.21.15.7:0/1490076400 conn(0x7f3c1406c2c0 msgr2=0x7f3c140662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 -- 172.21.15.7:0/1490076400 shutdown_connections 2023-11-28T18:23:16.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 -- 172.21.15.7:0/1490076400 wait complete. 2023-11-28T18:23:16.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7fae443d8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fae34045940 0x7fae340e3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7fae443d8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae340dd8a0 con 0x7fae3404c6a0 2023-11-28T18:23:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7fae443d8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae340066c0 con 0x7fae340dd4c0 2023-11-28T18:23:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7fae443d8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae340069a0 con 0x7fae34045940 2023-11-28T18:23:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f4cd9e09640 1 -- 172.21.15.7:0/3043934519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 msgr2=0x7f4ccc04ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f4cd9e09640 1 --2- 172.21.15.7:0/3043934519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc04ba40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40695f0 tx=0x7f4cd409a120 comp rx=0 tx=0).stop 2023-11-28T18:23:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f4cd9e09640 1 -- 172.21.15.7:0/3043934519 shutdown_connections 2023-11-28T18:23:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f4cd9e09640 1 --2- 172.21.15.7:0/3043934519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc04ba40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f4cd9e09640 1 --2- 172.21.15.7:0/3043934519 >> v2:172.21.15.193:3300/0 conn(0x7f4ccc0e71c0 0x7f4ccc046270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f4cd9e09640 1 -- 172.21.15.7:0/3043934519 >> 172.21.15.7:0/3043934519 conn(0x7f4ccc050160 msgr2=0x7f4ccc050570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f4cd9e09640 1 -- 172.21.15.7:0/3043934519 shutdown_connections 2023-11-28T18:23:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f4cd9e09640 1 -- 172.21.15.7:0/3043934519 wait complete. 2023-11-28T18:23:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7f3c19cd6640 1 Processor -- start 2023-11-28T18:23:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c19cd6640 1 -- start start 2023-11-28T18:23:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7fae42174640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fae340dd4c0 0x7fae34005ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35654/0 (socket says 172.21.15.7:35654) 2023-11-28T18:23:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7fae42174640 1 -- 172.21.15.7:0/3094202011 learned_addr learned my addr 172.21.15.7:0/3094202011 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.964+0000 7fae42975640 1 --2- 172.21.15.7:0/3094202011 >> v2:172.21.15.193:3300/0 conn(0x7fae34045940 0x7fae340e3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7fae42975640 1 -- 172.21.15.7:0/3094202011 >> v1:172.21.15.193:6789/0 conn(0x7fae340dd4c0 legacy=0x7fae34005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7fae42975640 1 -- 172.21.15.7:0/3094202011 >> v1:172.21.15.7:6789/0 conn(0x7fae3404c6a0 legacy=0x7fae34045230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7fae42975640 1 -- 172.21.15.7:0/3094202011 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae34006c80 con 0x7fae34045940 2023-11-28T18:23:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd9e09640 1 Processor -- start 2023-11-28T18:23:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd9e09640 1 -- start start 2023-11-28T18:23:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c19cd6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c140c6170 0x7f3c1414a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd9e09640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd9e09640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ccc0dd290 0x7f4ccc047c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd9e09640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ccc0e71c0 0x7f4ccc0bec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd9e09640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0474e0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd9e09640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0b5850 con 0x7f4ccc0dd290 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd9e09640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0cb300 con 0x7f4ccc0e71c0 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd2ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ccc0dd290 0x7f4ccc047c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc0cadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59522/0 (socket says 172.21.15.7:59522) 2023-11-28T18:23:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd37fe640 1 -- 172.21.15.7:0/198489337 learned_addr learned my addr 172.21.15.7:0/198489337 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c19cd6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1414fef0 0x7f3c1414ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd37fe640 1 -- 172.21.15.7:0/198489337 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ccc0e71c0 msgr2=0x7f4ccc0bec40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T18:23:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c137fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c140c6170 0x7f3c1414a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c137fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c140c6170 0x7f3c1414a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38722/0 (socket says 172.21.15.7:38722) 2023-11-28T18:23:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c137fe640 1 -- 172.21.15.7:0/3782066574 learned_addr learned my addr 172.21.15.7:0/3782066574 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c19cd6640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c1414bfa0 0x7f3c141415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c19cd6640 1 -- 172.21.15.7:0/3782066574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c14072700 con 0x7f3c1414bfa0 2023-11-28T18:23:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c19cd6640 1 -- 172.21.15.7:0/3782066574 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c140c1550 con 0x7f3c1414fef0 2023-11-28T18:23:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f3c19cd6640 1 -- 172.21.15.7:0/3782066574 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c141fd490 con 0x7f3c140c6170 2023-11-28T18:23:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c12ffd640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1414fef0 0x7f3c1414ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd37fe640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ccc0e71c0 0x7f4ccc0bec40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c13fff640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c1414bfa0 0x7f3c141415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c137fe640 1 -- 172.21.15.7:0/3782066574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1414fef0 msgr2=0x7f3c1414ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c137fe640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1414fef0 0x7f3c1414ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f8372e46640 1 Processor -- start 2023-11-28T18:23:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c137fe640 1 -- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c1414bfa0 msgr2=0x7f3c141415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c137fe640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c1414bfa0 0x7f3c141415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd37fe640 1 -- 172.21.15.7:0/198489337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ccc0dd290 msgr2=0x7f4ccc047c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f8372e46640 1 -- start start 2023-11-28T18:23:16.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c137fe640 1 -- 172.21.15.7:0/3782066574 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c04009420 con 0x7f3c140c6170 2023-11-28T18:23:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c13fff640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c1414bfa0 0x7f3c141415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd37fe640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ccc0dd290 0x7f4ccc047c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.968+0000 7f4cd37fe640 1 -- 172.21.15.7:0/198489337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd4074cf0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f4cd37fe640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc0cadc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000c500 tx=0x7f4cc000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f4cbffff640 1 -- 172.21.15.7:0/198489337 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0005cf0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f4cbffff640 1 -- 172.21.15.7:0/198489337 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc0005e50 con 0x7f4ccc0467b0 2023-11-28T18:23:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae42975640 1 --2- 172.21.15.7:0/3094202011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae34045940 0x7fae340e3150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0097c0 tx=0x7fae2c01ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7173c61109d3b088 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f4cbffff640 1 -- 172.21.15.7:0/198489337 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0005cf0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f4cd9e09640 1 -- 172.21.15.7:0/198489337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc0499a0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f4cd9e09640 1 -- 172.21.15.7:0/198489337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc049f60 con 0x7f4ccc0467b0 2023-11-28T18:23:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c137fe640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c140c6170 0x7f3c1414a920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00e570 tx=0x7f3c0c00ea30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3bfbfff640 1 -- 172.21.15.7:0/3782066574 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c012870 con 0x7f3c140c6170 2023-11-28T18:23:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3bfbfff640 1 -- 172.21.15.7:0/3782066574 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c0129d0 con 0x7f3c140c6170 2023-11-28T18:23:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3bfbfff640 1 -- 172.21.15.7:0/3782066574 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c012d40 con 0x7f3c140c6170 2023-11-28T18:23:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f3c19cd6640 1 -- 172.21.15.7:0/3782066574 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c141fd770 con 0x7f3c140c6170 2023-11-28T18:23:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae41172640 1 -- 172.21.15.7:0/3094202011 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0a8930 con 0x7fae34045940 2023-11-28T18:23:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae41172640 1 -- 172.21.15.7:0/3094202011 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c0a8a90 con 0x7fae34045940 2023-11-28T18:23:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae41172640 1 -- 172.21.15.7:0/3094202011 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0a8e20 con 0x7fae34045940 2023-11-28T18:23:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae443d8640 1 -- 172.21.15.7:0/3094202011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae34045940 msgr2=0x7fae340e3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae443d8640 1 --2- 172.21.15.7:0/3094202011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae34045940 0x7fae340e3150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0097c0 tx=0x7fae2c01ec30 comp rx=0 tx=0).stop 2023-11-28T18:23:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae443d8640 1 -- 172.21.15.7:0/3094202011 shutdown_connections 2023-11-28T18:23:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae443d8640 1 --2- 172.21.15.7:0/3094202011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae34045940 0x7fae340e3150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7fae443d8640 1 -- 172.21.15.7:0/3094202011 >> 172.21.15.7:0/3094202011 conn(0x7fae34050190 msgr2=0x7fae340505a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f4cbdffb640 1 -- 172.21.15.7:0/198489337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc0052a0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f8372e46640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c066e00 0x7f836c159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f8372e46640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f836c1571d0 0x7f836c0c7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.972+0000 7f8372e46640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f836c155d40 0x7f836c14fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7fae443d8640 1 -- 172.21.15.7:0/3094202011 shutdown_connections 2023-11-28T18:23:16.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f8370be2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c066e00 0x7f836c159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f8370be2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c066e00 0x7f836c159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38726/0 (socket says 172.21.15.7:38726) 2023-11-28T18:23:16.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f836bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f836c1571d0 0x7f836c0c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f3c19cd6640 1 -- 172.21.15.7:0/3782066574 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1414a400 con 0x7f3c140c6170 2023-11-28T18:23:16.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f836bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f836c1571d0 0x7f836c0c7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59548/0 (socket says 172.21.15.7:59548) 2023-11-28T18:23:16.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f8372e46640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836c15a260 con 0x7f836c1571d0 2023-11-28T18:23:16.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f8372e46640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836c1565d0 con 0x7f836c155d40 2023-11-28T18:23:16.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f8372e46640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836c156870 con 0x7f836c066e00 2023-11-28T18:23:16.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f8370be2640 1 -- 172.21.15.7:0/3508930173 learned_addr learned my addr 172.21.15.7:0/3508930173 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7fc8f40bb640 1 Processor -- start 2023-11-28T18:23:16.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7fc8f40bb640 1 -- start start 2023-11-28T18:23:16.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f4cbffff640 1 -- 172.21.15.7:0/198489337 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4cc00070f0 con 0x7f4ccc0467b0 2023-11-28T18:23:16.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f83713e3640 1 --2- 172.21.15.7:0/3508930173 >> v2:172.21.15.193:3300/0 conn(0x7f836c155d40 0x7f836c14fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7fae443d8640 1 -- 172.21.15.7:0/3094202011 wait complete. 2023-11-28T18:23:16.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7fae443d8640 1 Processor -- start 2023-11-28T18:23:16.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f83713e3640 1 -- 172.21.15.7:0/3508930173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c066e00 msgr2=0x7f836c159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7fae443d8640 1 -- start start 2023-11-28T18:23:16.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f83713e3640 1 --2- 172.21.15.7:0/3508930173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c066e00 0x7f836c159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f83713e3640 1 -- 172.21.15.7:0/3508930173 >> v2:172.21.15.7:3300/0 conn(0x7f836c1571d0 msgr2=0x7f836c0c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f83713e3640 1 --2- 172.21.15.7:0/3508930173 >> v2:172.21.15.7:3300/0 conn(0x7f836c1571d0 0x7f836c0c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f3bfbfff640 1 -- 172.21.15.7:0/3782066574 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3c0c01c460 con 0x7f3c140c6170 2023-11-28T18:23:16.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.976+0000 7f83713e3640 1 -- 172.21.15.7:0/3508930173 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836c156b30 con 0x7f836c155d40 2023-11-28T18:23:16.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f836bfff640 1 --2- 172.21.15.7:0/3508930173 >> v2:172.21.15.7:3300/0 conn(0x7f836c1571d0 0x7f836c0c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:16.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f8370be2640 1 --2- 172.21.15.7:0/3508930173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c066e00 0x7f836c159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f4cbffff640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cb80668f0 0x7f4cb8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f4cbffff640 1 -- 172.21.15.7:0/198489337 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4cc0083560 con 0x7f4ccc0467b0 2023-11-28T18:23:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f3bf9ffb640 1 -- 172.21.15.7:0/3782066574 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c140c8bd0 con 0x7f3c140c6170 2023-11-28T18:23:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae443d8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae34045940 0x7fae34046030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f40bb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc8ec1500d0 0x7fc8ec159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f40bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec066e00 0x7fc8ec0c7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f40bb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec15a260 con 0x7fc8ec1571d0 2023-11-28T18:23:16.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f40bb640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec159290 con 0x7fc8ec1500d0 2023-11-28T18:23:16.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f40bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec1504b0 con 0x7fc8ec066e00 2023-11-28T18:23:16.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae443d8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae3404c6a0 0x7fae340d6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f4cd2ffd640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cb80668f0 0x7f4cb8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f1e57640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc8ec1500d0 0x7fc8ec159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f1e57640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc8ec1500d0 0x7fc8ec159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50472/0 (socket says 172.21.15.7:50472) 2023-11-28T18:23:16.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f1e57640 1 -- 172.21.15.7:0/1476479710 learned_addr learned my addr 172.21.15.7:0/1476479710 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f1656640 1 --2- 172.21.15.7:0/1476479710 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec066e00 0x7fc8ec0c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42174640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae34045940 0x7fae34046030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42174640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae34045940 0x7fae34046030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38738/0 (socket says 172.21.15.7:38738) 2023-11-28T18:23:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42174640 1 -- 172.21.15.7:0/3083257073 learned_addr learned my addr 172.21.15.7:0/3083257073 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae443d8640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae340dd4c0 0x7fae340d65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f1e57640 1 -- 172.21.15.7:0/1476479710 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec066e00 msgr2=0x7fc8ec0c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f1e57640 1 --2- 172.21.15.7:0/1476479710 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec066e00 0x7fc8ec0c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f1e57640 1 -- 172.21.15.7:0/1476479710 >> v1:172.21.15.7:6789/0 conn(0x7fc8ec1571d0 legacy=0x7fc8ec155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fc8f1e57640 1 -- 172.21.15.7:0/1476479710 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec1565d0 con 0x7fc8ec1500d0 2023-11-28T18:23:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae340d5760 con 0x7fae340dd4c0 2023-11-28T18:23:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae340dc680 con 0x7fae3404c6a0 2023-11-28T18:23:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae41973640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae3404c6a0 0x7fae340d6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae340d5c00 con 0x7fae34045940 2023-11-28T18:23:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f51a4432640 1 Processor -- start 2023-11-28T18:23:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f51a4432640 1 -- start start 2023-11-28T18:23:16.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42975640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae340dd4c0 0x7fae340d65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f3bfbfff640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c00066920 0x7f3c00068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42174640 1 -- 172.21.15.7:0/3083257073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae3404c6a0 msgr2=0x7fae340d6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42174640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae3404c6a0 0x7fae340d6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42174640 1 -- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae340dd4c0 msgr2=0x7fae340d65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42174640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae340dd4c0 0x7fae340d65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7f3bfbfff640 1 -- 172.21.15.7:0/3782066574 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c0c087400 con 0x7f3c140c6170 2023-11-28T18:23:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.980+0000 7fae42174640 1 -- 172.21.15.7:0/3083257073 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c009420 con 0x7fae34045940 2023-11-28T18:23:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f3c12ffd640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c00066920 0x7f3c00068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fae42174640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae34045940 0x7fae34046030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fae3800c370 tx=0x7fae3800c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f4cd2ffd640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cb80668f0 0x7f4cb8068db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc035210 tx=0x7f4cc8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f83713e3640 1 --2- 172.21.15.7:0/3508930173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c155d40 0x7f836c14fe10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f836c06ce30 tx=0x7f836401f0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84d7b3f74773be34 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f3c12ffd640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c00066920 0x7f3c00068de0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3c040b1000 tx=0x7f3c040a9000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f836b7fe640 1 -- 172.21.15.7:0/3508930173 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83640b0440 con 0x7f836c155d40 2023-11-28T18:23:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f836b7fe640 1 -- 172.21.15.7:0/3508930173 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83640b05a0 con 0x7f836c155d40 2023-11-28T18:23:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f836b7fe640 1 -- 172.21.15.7:0/3508930173 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83640b0930 con 0x7f836c155d40 2023-11-28T18:23:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f51a4432640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f51a21ce640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f51a19cd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f519c1571d0 0x7f519c0c7790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35658/0 (socket says 172.21.15.7:35658) 2023-11-28T18:23:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f51a21ce640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38752/0 (socket says 172.21.15.7:38752) 2023-11-28T18:23:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f51a19cd640 1 -- 172.21.15.7:0/3070451254 learned_addr learned my addr 172.21.15.7:0/3070451254 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fc8f1e57640 1 --2- 172.21.15.7:0/1476479710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec1500d0 0x7fc8ec159d20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009870 tx=0x7fc8e801c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e560ef7ee9f376 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f3bfbfff640 1 -- 172.21.15.7:0/3782066574 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3c0c058020 con 0x7f3c140c6170 2023-11-28T18:23:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fae32ffd640 1 -- 172.21.15.7:0/3083257073 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38005cf0 con 0x7fae34045940 2023-11-28T18:23:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 -- 172.21.15.7:0/3508930173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c155d40 msgr2=0x7f836c14fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 --2- 172.21.15.7:0/3508930173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c155d40 0x7f836c14fe10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f836c06ce30 tx=0x7f836401f0f0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 -- 172.21.15.7:0/3508930173 shutdown_connections 2023-11-28T18:23:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 --2- 172.21.15.7:0/3508930173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c155d40 0x7f836c14fe10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 --2- 172.21.15.7:0/3508930173 >> v2:172.21.15.7:3300/0 conn(0x7f836c1571d0 0x7f836c0c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 --2- 172.21.15.7:0/3508930173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c066e00 0x7f836c159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 -- 172.21.15.7:0/3508930173 >> 172.21.15.7:0/3508930173 conn(0x7f836c0662e0 msgr2=0x7f836c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fc8f0e55640 1 -- 172.21.15.7:0/1476479710 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e80ac630 con 0x7fc8ec1500d0 2023-11-28T18:23:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fc8f0e55640 1 -- 172.21.15.7:0/1476479710 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e80ac790 con 0x7fc8ec1500d0 2023-11-28T18:23:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fc8f0e55640 1 -- 172.21.15.7:0/1476479710 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e80acb20 con 0x7fc8ec1500d0 2023-11-28T18:23:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f51a4432640 1 -- 172.21.15.7:0/3070451254 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519c15a260 con 0x7f519c155d40 2023-11-28T18:23:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f51a4432640 1 -- 172.21.15.7:0/3070451254 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519c159290 con 0x7f519c1571d0 2023-11-28T18:23:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f51a4432640 1 -- 172.21.15.7:0/3070451254 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519c1504b0 con 0x7f519c1500d0 2023-11-28T18:23:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fd276417640 1 Processor -- start 2023-11-28T18:23:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 -- 172.21.15.7:0/3508930173 shutdown_connections 2023-11-28T18:23:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fae32ffd640 1 -- 172.21.15.7:0/3083257073 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae38005e50 con 0x7fae34045940 2023-11-28T18:23:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fae32ffd640 1 -- 172.21.15.7:0/3083257073 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae380072f0 con 0x7fae34045940 2023-11-28T18:23:17.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae340364d0 con 0x7fae34045940 2023-11-28T18:23:17.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae340dc0c0 con 0x7fae34045940 2023-11-28T18:23:17.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7f51a21ce640 1 -- 172.21.15.7:0/3070451254 >> v1:172.21.15.193:6789/0 conn(0x7f519c1571d0 legacy=0x7f519c0c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7f51a21ce640 1 -- 172.21.15.7:0/3070451254 >> v1:172.21.15.7:6789/0 conn(0x7f519c155d40 legacy=0x7f519c0c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7f51a21ce640 1 -- 172.21.15.7:0/3070451254 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c1565d0 con 0x7f519c1500d0 2023-11-28T18:23:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7fd276417640 1 -- start start 2023-11-28T18:23:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd276417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd2700c6880 0x7fd27015ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd276417640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd2700c21d0 0x7fd2700c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd276417640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd270161940 con 0x7fd2700c80d0 2023-11-28T18:23:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd276417640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd27015f190 con 0x7fd2700c21d0 2023-11-28T18:23:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd276417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2700c29a0 con 0x7fd2700c6880 2023-11-28T18:23:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fc8f40bb640 1 -- 172.21.15.7:0/1476479710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec1500d0 msgr2=0x7fc8ec159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/1476479710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec1500d0 0x7fc8ec159d20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009870 tx=0x7fc8e801c500 comp rx=0 tx=0).stop 2023-11-28T18:23:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fc8f40bb640 1 -- 172.21.15.7:0/1476479710 shutdown_connections 2023-11-28T18:23:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/1476479710 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec066e00 0x7fc8ec0c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/1476479710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec1500d0 0x7fc8ec159d20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fc8f40bb640 1 -- 172.21.15.7:0/1476479710 >> 172.21.15.7:0/1476479710 conn(0x7fc8ec0662e0 msgr2=0x7fc8ec0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.984+0000 7f8372e46640 1 -- 172.21.15.7:0/3508930173 wait complete. 2023-11-28T18:23:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd2749b4640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd2700c80d0 0x7fd270157e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35664/0 (socket says 172.21.15.7:35664) 2023-11-28T18:23:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7f8372e46640 1 Processor -- start 2023-11-28T18:23:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd2749b4640 1 -- 172.21.15.7:0/1553844239 learned_addr learned my addr 172.21.15.7:0/1553844239 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7f8372e46640 1 -- start start 2023-11-28T18:23:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae340d5a20 con 0x7fae34045940 2023-11-28T18:23:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182071555 0 0) 0x7fd270161940 con 0x7fd2700c80d0 2023-11-28T18:23:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd258003530 con 0x7fd2700c80d0 2023-11-28T18:23:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fc8f40bb640 1 -- 172.21.15.7:0/1476479710 shutdown_connections 2023-11-28T18:23:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2402098449 0 0) 0x7fd258003530 con 0x7fd2700c80d0 2023-11-28T18:23:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd2700c6880 msgr2=0x7fd27015ec50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T18:23:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fd26effd640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd2700c6880 0x7fd27015ec50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 >> v2:172.21.15.7:3300/0 conn(0x7fd2700c21d0 msgr2=0x7fd2700c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 --2- 172.21.15.7:0/1553844239 >> v2:172.21.15.7:3300/0 conn(0x7fd2700c21d0 0x7fd2700c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2701626c0 con 0x7fd2700c80d0 2023-11-28T18:23:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fd260001680 con 0x7fd2700c80d0 2023-11-28T18:23:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 >> v1:172.21.15.193:6789/0 conn(0x7fd2700c80d0 legacy=0x7fd270157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd258003980 0x7fd258003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2580047d0 0x7fd258004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd2700c6880 0x7fd25800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27015f190 con 0x7fd258003980 2023-11-28T18:23:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2700c29a0 con 0x7fd2580047d0 2023-11-28T18:23:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd258005600 con 0x7fd2700c6880 2023-11-28T18:23:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8372e46640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c066e00 0x7f836c13b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8372e46640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c155d40 0x7f836c0726c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8372e46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f836c1571d0 0x7f836c072c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26f7fe640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2580047d0 0x7fd258004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7fc8f40bb640 1 -- 172.21.15.7:0/1476479710 wait complete. 2023-11-28T18:23:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd2749b4640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd2700c6880 0x7fd25800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f836bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c155d40 0x7f836c0726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f836bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c155d40 0x7f836c0726c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38778/0 (socket says 172.21.15.7:38778) 2023-11-28T18:23:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8370be2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c066e00 0x7f836c13b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8370be2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c066e00 0x7f836c13b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50514/0 (socket says 172.21.15.7:50514) 2023-11-28T18:23:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f836bfff640 1 -- 172.21.15.7:0/3522520496 learned_addr learned my addr 172.21.15.7:0/3522520496 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26f7fe640 1 -- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd2700c6880 msgr2=0x7fd25800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f5c1f49a640 1 Processor -- start 2023-11-28T18:23:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26f7fe640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd2700c6880 0x7fd25800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c0707b0 con 0x7f836c1571d0 2023-11-28T18:23:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c14a9d0 con 0x7f836c066e00 2023-11-28T18:23:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836c1559b0 con 0x7f836c155d40 2023-11-28T18:23:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f40bb640 1 Processor -- start 2023-11-28T18:23:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f40bb640 1 -- start start 2023-11-28T18:23:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f5c1f49a640 1 -- start start 2023-11-28T18:23:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f83713e3640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f836c1571d0 0x7f836c072c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.988+0000 7f4cbffff640 1 -- 172.21.15.7:0/198489337 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4cc0050840 con 0x7f4ccc0467b0 2023-11-28T18:23:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fae32ffd640 1 -- 172.21.15.7:0/3083257073 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fae38007450 con 0x7fae34045940 2023-11-28T18:23:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8370be2640 1 -- 172.21.15.7:0/3522520496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c155d40 msgr2=0x7f836c0726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fae32ffd640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae200669a0 0x7fae20068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fae32ffd640 1 -- 172.21.15.7:0/3083257073 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae38083a90 con 0x7fae34045940 2023-11-28T18:23:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f40bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec066e00 0x7fc8ec154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f40bb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1500d0 0x7fc8ec0707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f40bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec1571d0 0x7fc8ec0729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f40bb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec072560 con 0x7fc8ec1500d0 2023-11-28T18:23:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f40bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec2035e0 con 0x7fc8ec066e00 2023-11-28T18:23:17.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f40bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec202a60 con 0x7fc8ec1571d0 2023-11-28T18:23:17.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8370be2640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c155d40 0x7f836c0726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8370be2640 1 -- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f836c1571d0 msgr2=0x7f836c072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8370be2640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f836c1571d0 0x7f836c072c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f2658640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec1571d0 0x7fc8ec0729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fae41973640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae200669a0 0x7fae20068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f2658640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec1571d0 0x7fc8ec0729f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38784/0 (socket says 172.21.15.7:38784) 2023-11-28T18:23:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f2658640 1 -- 172.21.15.7:0/3553869385 learned_addr learned my addr 172.21.15.7:0/3553869385 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fc8f1e57640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec066e00 0x7fc8ec154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7f8370be2640 1 -- 172.21.15.7:0/3522520496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c0094d0 con 0x7f836c066e00 2023-11-28T18:23:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f2658640 1 -- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec066e00 msgr2=0x7fc8ec154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f2658640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec066e00 0x7fc8ec154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1f49a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5c180c50d0 0x7f5c180cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1f49a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c18159f20 con 0x7f5c180c9f10 2023-11-28T18:23:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1f49a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1815a200 con 0x7f5c18153f20 2023-11-28T18:23:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1f49a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c180c45a0 con 0x7f5c180c50d0 2023-11-28T18:23:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f2658640 1 -- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1500d0 msgr2=0x7fc8ec0707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f2658640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1500d0 0x7fc8ec0707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1d236640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f5c18153f20 0x7f5c18071db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35678/0 (socket says 172.21.15.7:35678) 2023-11-28T18:23:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1d236640 1 -- 172.21.15.7:0/574481429 learned_addr learned my addr 172.21.15.7:0/574481429 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f2658640 1 -- 172.21.15.7:0/3553869385 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e80094d0 con 0x7fc8ec1571d0 2023-11-28T18:23:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f8370be2640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c066e00 0x7f836c13b000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f835c00e910 tx=0x7f835c00edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f1e57640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec066e00 0x7fc8ec154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1da37640 1 --2- 172.21.15.7:0/574481429 >> v2:172.21.15.193:3300/0 conn(0x7f5c180c50d0 0x7f5c180cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.992+0000 7fd26f7fe640 1 -- 172.21.15.7:0/1553844239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd258003980 msgr2=0x7fd258003da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:23:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fd26f7fe640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd258003980 0x7fd258003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fd26f7fe640 1 -- 172.21.15.7:0/1553844239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258003530 con 0x7fd2580047d0 2023-11-28T18:23:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f2658640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec1571d0 0x7fc8ec0729f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400ba10 tx=0x7fc8e400bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f51a21ce640 1 --2- 172.21.15.7:0/3070451254 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c159d20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f519400aed0 tx=0x7f519401dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f46138766405d66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fae41973640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae200669a0 0x7fae20068e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0695f0 tx=0x7fae3c07bd50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8e27fc640 1 -- 172.21.15.7:0/3553869385 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e400cba0 con 0x7fc8ec1571d0 2023-11-28T18:23:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8e27fc640 1 -- 172.21.15.7:0/3553869385 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e400cd00 con 0x7fc8ec1571d0 2023-11-28T18:23:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f83697fa640 1 -- 172.21.15.7:0/3522520496 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c00ca00 con 0x7f836c066e00 2023-11-28T18:23:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec202d40 con 0x7fc8ec1571d0 2023-11-28T18:23:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8e27fc640 1 -- 172.21.15.7:0/3553869385 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4014680 con 0x7fc8ec1571d0 2023-11-28T18:23:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec203300 con 0x7fc8ec1571d0 2023-11-28T18:23:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fae32ffd640 1 -- 172.21.15.7:0/3083257073 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fae38050c20 con 0x7fae34045940 2023-11-28T18:23:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f83697fa640 1 -- 172.21.15.7:0/3522520496 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835c00cb60 con 0x7f836c066e00 2023-11-28T18:23:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8364009a70 con 0x7f836c066e00 2023-11-28T18:23:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836c0721a0 con 0x7f836c066e00 2023-11-28T18:23:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f51a11cc640 1 -- 172.21.15.7:0/3070451254 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519400a2d0 con 0x7f519c1500d0 2023-11-28T18:23:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f51a11cc640 1 -- 172.21.15.7:0/3070451254 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51940a6930 con 0x7f519c1500d0 2023-11-28T18:23:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f51a11cc640 1 -- 172.21.15.7:0/3070451254 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51940a6cc0 con 0x7f519c1500d0 2023-11-28T18:23:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f83697fa640 1 -- 172.21.15.7:0/3522520496 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c00ced0 con 0x7f836c066e00 2023-11-28T18:23:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7fd2749b4640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd2700c6880 0x7fd25800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f51a4432640 1 -- 172.21.15.7:0/3070451254 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 msgr2=0x7f519c159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f51a4432640 1 --2- 172.21.15.7:0/3070451254 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c159d20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f519400aed0 tx=0x7f519401dab0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f51a4432640 1 -- 172.21.15.7:0/3070451254 shutdown_connections 2023-11-28T18:23:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c005260 con 0x7f836c066e00 2023-11-28T18:23:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f51a4432640 1 --2- 172.21.15.7:0/3070451254 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c159d20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f51a4432640 1 -- 172.21.15.7:0/3070451254 >> 172.21.15.7:0/3070451254 conn(0x7f519c0662e0 msgr2=0x7f519c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f51a4432640 1 -- 172.21.15.7:0/3070451254 shutdown_connections 2023-11-28T18:23:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7fc8e27fc640 1 -- 172.21.15.7:0/3553869385 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc8e401d3b0 con 0x7fc8ec1571d0 2023-11-28T18:23:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7fd26f7fe640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2580047d0 0x7fd258004c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd264009890 tx=0x7fd26401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d279d1532d7d576 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f83697fa640 1 -- 172.21.15.7:0/3522520496 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f835c013470 con 0x7f836c066e00 2023-11-28T18:23:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7fc8e27fc640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8d0066920 0x7fc8d0068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7fc8e27fc640 1 -- 172.21.15.7:0/3553869385 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8e4083f20 con 0x7fc8ec1571d0 2023-11-28T18:23:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1da37640 1 -- 172.21.15.7:0/574481429 >> v1:172.21.15.193:6789/0 conn(0x7f5c18153f20 legacy=0x7f5c18071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1da37640 1 -- 172.21.15.7:0/574481429 >> v1:172.21.15.7:6789/0 conn(0x7f5c180c9f10 legacy=0x7f5c180c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:16.996+0000 7f5c1da37640 1 -- 172.21.15.7:0/574481429 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1815a4e0 con 0x7f5c180c50d0 2023-11-28T18:23:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec072710 con 0x7fc8ec1571d0 2023-11-28T18:23:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f5c1da37640 1 --2- 172.21.15.7:0/574481429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c180c50d0 0x7f5c180cff30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009d90 tx=0x7f5c0801c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ace65532d8307849 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.000+0000 7f51a4432640 1 -- 172.21.15.7:0/3070451254 wait complete. 2023-11-28T18:23:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7fc8f1e57640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8d0066920 0x7fc8d0068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f83697fa640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83580669a0 0x7f8358068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f83697fa640 1 -- 172.21.15.7:0/3522520496 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f835c0848a0 con 0x7f836c066e00 2023-11-28T18:23:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a4432640 1 Processor -- start 2023-11-28T18:23:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f836bfff640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83580669a0 0x7f8358068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f836bfff640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83580669a0 0x7f8358068e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f836c1583d0 tx=0x7f8360008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a4432640 1 -- start start 2023-11-28T18:23:17.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7fc8f1e57640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8d0066920 0x7fc8d0068de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc8e80ae000 tx=0x7fc8e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a4432640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a4432640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f519c155d40 0x7f519c13eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a4432640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f519c1571d0 0x7f519c1390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a4432640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c14eec0 con 0x7f519c155d40 2023-11-28T18:23:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a4432640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c0caff0 con 0x7f519c1571d0 2023-11-28T18:23:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a4432640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519c14f180 con 0x7f519c1500d0 2023-11-28T18:23:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c144080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38796/0 (socket says 172.21.15.7:38796) 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 -- 172.21.15.7:0/4285410876 learned_addr learned my addr 172.21.15.7:0/4285410876 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a19cd640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f519c155d40 0x7f519c13eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a29cf640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f519c1571d0 0x7f519c1390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 -- 172.21.15.7:0/4285410876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f519c1571d0 msgr2=0x7f519c1390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f519c1571d0 0x7f519c1390f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 -- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f519c155d40 msgr2=0x7f519c13eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f519c155d40 0x7f519c13eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 -- 172.21.15.7:0/4285410876 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519400ab80 con 0x7f519c1500d0 2023-11-28T18:23:17.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.004+0000 7f51a21ce640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c144080 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f51940b0040 tx=0x7f519401a850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640ab600 con 0x7fd2580047d0 2023-11-28T18:23:17.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2640a7050 con 0x7fd2580047d0 2023-11-28T18:23:17.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd26effd640 1 -- 172.21.15.7:0/1553844239 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640ab600 con 0x7fd2580047d0 2023-11-28T18:23:17.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f5c17fff640 1 -- 172.21.15.7:0/574481429 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080a66c0 con 0x7f5c180c50d0 2023-11-28T18:23:17.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f5192ffd640 1 -- 172.21.15.7:0/4285410876 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51940bc070 con 0x7f519c1500d0 2023-11-28T18:23:17.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f5192ffd640 1 -- 172.21.15.7:0/4285410876 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51940ae9b0 con 0x7f519c1500d0 2023-11-28T18:23:17.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f5192ffd640 1 -- 172.21.15.7:0/4285410876 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51940aed40 con 0x7f519c1500d0 2023-11-28T18:23:17.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519c1445c0 con 0x7f519c1500d0 2023-11-28T18:23:17.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519c0cad10 con 0x7f519c1500d0 2023-11-28T18:23:17.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f83697fa640 1 -- 172.21.15.7:0/3522520496 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f835c051c00 con 0x7f836c066e00 2023-11-28T18:23:17.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fc8e27fc640 1 -- 172.21.15.7:0/3553869385 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc8ec072710 con 0x7fc8ec1571d0 2023-11-28T18:23:17.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f5c17fff640 1 -- 172.21.15.7:0/574481429 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c080a6820 con 0x7f5c180c50d0 2023-11-28T18:23:17.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f5c17fff640 1 -- 172.21.15.7:0/574481429 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080a6bb0 con 0x7f5c180c50d0 2023-11-28T18:23:17.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd276417640 1 -- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2580047d0 msgr2=0x7fd258004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd276417640 1 --2- 172.21.15.7:0/1553844239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2580047d0 0x7fd258004c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd264009890 tx=0x7fd26401c3c0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd2749b4640 1 -- 172.21.15.7:0/1553844239 reap_dead start 2023-11-28T18:23:17.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd276417640 1 -- 172.21.15.7:0/1553844239 shutdown_connections 2023-11-28T18:23:17.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd276417640 1 -- 172.21.15.7:0/1553844239 >> 172.21.15.7:0/1553844239 conn(0x7fd2700662e0 msgr2=0x7fd2700666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519c04e890 con 0x7f519c1500d0 2023-11-28T18:23:17.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd276417640 1 -- 172.21.15.7:0/1553844239 shutdown_connections 2023-11-28T18:23:17.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7fd276417640 1 -- 172.21.15.7:0/1553844239 wait complete. 2023-11-28T18:23:17.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f5c1f49a640 1 -- 172.21.15.7:0/574481429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c180c50d0 msgr2=0x7f5c180cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.008+0000 7f5c1f49a640 1 --2- 172.21.15.7:0/574481429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c180c50d0 0x7f5c180cff30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009d90 tx=0x7f5c0801c7f0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7f5c1f49a640 1 -- 172.21.15.7:0/574481429 shutdown_connections 2023-11-28T18:23:17.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7f5c1f49a640 1 --2- 172.21.15.7:0/574481429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c180c50d0 0x7f5c180cff30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7f5c1f49a640 1 -- 172.21.15.7:0/574481429 >> 172.21.15.7:0/574481429 conn(0x7f5c1806c2b0 msgr2=0x7f5c18070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7f5192ffd640 1 -- 172.21.15.7:0/4285410876 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f51940b7480 con 0x7f519c1500d0 2023-11-28T18:23:17.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7f5c1f49a640 1 -- 172.21.15.7:0/574481429 shutdown_connections 2023-11-28T18:23:17.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7f5c1f49a640 1 -- 172.21.15.7:0/574481429 wait complete. 2023-11-28T18:23:17.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7fd276417640 1 Processor -- start 2023-11-28T18:23:17.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7fd276417640 1 -- start start 2023-11-28T18:23:17.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7f5192ffd640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f51800668d0 0x7f5180068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7fd276417640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2580047d0 0x7fd27015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7f5192ffd640 1 -- 172.21.15.7:0/4285410876 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f519411f7e0 con 0x7f519c1500d0 2023-11-28T18:23:17.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f51a19cd640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f51800668d0 0x7f5180068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7fd276417640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd27015a6a0 0x7fd2700cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7fd276417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd27015f370 0x7fd270136f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7fd276417640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27015ba20 con 0x7fd2580047d0 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7fd276417640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd270158f00 con 0x7fd27015a6a0 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.012+0000 7fd276417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2701591c0 con 0x7fd27015f370 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd26ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2580047d0 0x7fd27015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd26ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2580047d0 0x7fd27015b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59602/0 (socket says 172.21.15.7:59602) 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd26ffff640 1 -- 172.21.15.7:0/680450130 learned_addr learned my addr 172.21.15.7:0/680450130 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:17.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd2749b4640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd27015f370 0x7fd270136f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f51a19cd640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f51800668d0 0x7f5180068d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f519c155b60 tx=0x7f518c00a130 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd2749b4640 1 -- 172.21.15.7:0/680450130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd27015a6a0 msgr2=0x7fd2700cb7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T18:23:17.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd2749b4640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd27015a6a0 0x7fd2700cb7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd2749b4640 1 -- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2580047d0 msgr2=0x7fd27015b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd2749b4640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2580047d0 0x7fd27015b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd2749b4640 1 -- 172.21.15.7:0/680450130 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2701594a0 con 0x7fd27015f370 2023-11-28T18:23:17.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd2749b4640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd27015f370 0x7fd270136f60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd270067090 tx=0x7fd260007420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5c1f49a640 1 Processor -- start 2023-11-28T18:23:17.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5192ffd640 1 -- 172.21.15.7:0/4285410876 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f51940ecac0 con 0x7f519c1500d0 2023-11-28T18:23:17.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5c1f49a640 1 -- start start 2023-11-28T18:23:17.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd26cff9640 1 -- 172.21.15.7:0/680450130 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260007750 con 0x7fd27015f370 2023-11-28T18:23:17.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd26cff9640 1 -- 172.21.15.7:0/680450130 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd260010720 con 0x7fd27015f370 2023-11-28T18:23:17.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd270159780 con 0x7fd27015f370 2023-11-28T18:23:17.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd26cff9640 1 -- 172.21.15.7:0/680450130 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260010ad0 con 0x7fd27015f370 2023-11-28T18:23:17.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd270159d40 con 0x7fd27015f370 2023-11-28T18:23:17.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5c1f49a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c180c50d0 0x7f5c180c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5c1f49a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5c180c9f10 0x7f5c18147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5c1f49a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c18153f20 0x7f5c180c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5c1f49a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c181f68b0 con 0x7f5c180c50d0 2023-11-28T18:23:17.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5c1f49a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c181f5cc0 con 0x7f5c18153f20 2023-11-28T18:23:17.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.016+0000 7f5c1f49a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180c9660 con 0x7f5c180c9f10 2023-11-28T18:23:17.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.020+0000 7f5c1ca35640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5c180c9f10 0x7f5c18147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.020+0000 7f5c1ca35640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5c180c9f10 0x7f5c18147af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38814/0 (socket says 172.21.15.7:38814) 2023-11-28T18:23:17.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.020+0000 7f5c1ca35640 1 -- 172.21.15.7:0/4268751122 learned_addr learned my addr 172.21.15.7:0/4268751122 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:17.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.020+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd270159f30 con 0x7fd27015f370 2023-11-28T18:23:17.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.020+0000 7fd26cff9640 1 -- 172.21.15.7:0/680450130 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd260010c30 con 0x7fd27015f370 2023-11-28T18:23:17.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.020+0000 7fd26cff9640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd25c0669a0 0x7fd25c068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.020+0000 7fd26ffff640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd25c0669a0 0x7fd25c068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.020+0000 7fd26cff9640 1 -- 172.21.15.7:0/680450130 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd260086ce0 con 0x7fd27015f370 2023-11-28T18:23:17.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1d236640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c180c50d0 0x7f5c180c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1ca35640 1 -- 172.21.15.7:0/4268751122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c18153f20 msgr2=0x7f5c180c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1da37640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c18153f20 0x7f5c180c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7fd26ffff640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd25c0669a0 0x7fd25c068e60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd268004ad0 tx=0x7fd26800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1ca35640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c18153f20 0x7f5c180c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1ca35640 1 -- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c180c50d0 msgr2=0x7f5c180c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1ca35640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c180c50d0 0x7f5c180c8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1ca35640 1 -- 172.21.15.7:0/4268751122 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c08009a70 con 0x7f5c180c9f10 2023-11-28T18:23:17.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1d236640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c180c50d0 0x7f5c180c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:17.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1ca35640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5c180c9f10 0x7f5c18147af0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c00c480 tx=0x7f5c0c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c15ffb640 1 -- 172.21.15.7:0/4268751122 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c012830 con 0x7f5c180c9f10 2023-11-28T18:23:17.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c15ffb640 1 -- 172.21.15.7:0/4268751122 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c0129b0 con 0x7f5c180c9f10 2023-11-28T18:23:17.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c15ffb640 1 -- 172.21.15.7:0/4268751122 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c012d60 con 0x7f5c180c9f10 2023-11-28T18:23:17.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7fd26cff9640 1 -- 172.21.15.7:0/680450130 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd260053f90 con 0x7fd27015f370 2023-11-28T18:23:17.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c181f5f10 con 0x7f5c180c9f10 2023-11-28T18:23:17.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.024+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c181f64d0 con 0x7f5c180c9f10 2023-11-28T18:23:17.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.028+0000 7f5c15ffb640 1 -- 172.21.15.7:0/4268751122 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5c0c01c460 con 0x7f5c180c9f10 2023-11-28T18:23:17.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.028+0000 7f5c15ffb640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5c040668d0 0x7f5c04068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:17.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.028+0000 7f5c1d236640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5c040668d0 0x7f5c04068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:17.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.032+0000 7f5c1d236640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5c040668d0 0x7f5c04068d90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5c10007fd0 tx=0x7f5c10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:17.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.032+0000 7f5c15ffb640 1 -- 172.21.15.7:0/4268751122 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c0c00e280 con 0x7f5c180c9f10 2023-11-28T18:23:17.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.032+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c181f6690 con 0x7f5c180c9f10 2023-11-28T18:23:17.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.036+0000 7f5c15ffb640 1 -- 172.21.15.7:0/4268751122 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5c0c051d20 con 0x7f5c180c9f10 2023-11-28T18:23:17.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.328+0000 7f3bf9ffb640 1 -- 172.21.15.7:0/3782066574 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3c1404e890 con 0x7f3c140c6170 2023-11-28T18:23:17.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.328+0000 7f3bfbfff640 1 -- 172.21.15.7:0/3782066574 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3c0c057bb0 con 0x7f3c140c6170 2023-11-28T18:23:17.333 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.332+0000 7f4cbdffb640 1 -- 172.21.15.7:0/198489337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4ccc0b8b40 con 0x7f4ccc0467b0 2023-11-28T18:23:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 -- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c00066920 msgr2=0x7f3c00068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c00066920 0x7f3c00068de0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3c040b1000 tx=0x7f3c040a9000 comp rx=0 tx=0).stop 2023-11-28T18:23:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f4cbffff640 1 -- 172.21.15.7:0/198489337 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4cc0053e60 con 0x7f4ccc0467b0 2023-11-28T18:23:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 -- 172.21.15.7:0/3782066574 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c140c6170 msgr2=0x7f3c1414a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.341 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2023-11-28T18:23:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c140c6170 0x7f3c1414a920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00e570 tx=0x7f3c0c00ea30 comp rx=0 tx=0).stop 2023-11-28T18:23:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 -- 172.21.15.7:0/3782066574 shutdown_connections 2023-11-28T18:23:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c00066920 0x7f3c00068de0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c1414bfa0 0x7f3c141415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1414fef0 0x7f3c1414ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 --2- 172.21.15.7:0/3782066574 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c140c6170 0x7f3c1414a920 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 -- 172.21.15.7:0/3782066574 >> 172.21.15.7:0/3782066574 conn(0x7f3c1406c2c0 msgr2=0x7f3c14155530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 -- 172.21.15.7:0/3782066574 shutdown_connections 2023-11-28T18:23:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.336+0000 7f3bf9ffb640 1 -- 172.21.15.7:0/3782066574 wait complete. 2023-11-28T18:23:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 -- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cb80668f0 msgr2=0x7f4cb8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cb80668f0 0x7f4cb8068db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc035210 tx=0x7f4cc8008040 comp rx=0 tx=0).stop 2023-11-28T18:23:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 -- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 msgr2=0x7f4ccc0cadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc0cadc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000c500 tx=0x7f4cc000c9c0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 -- 172.21.15.7:0/198489337 shutdown_connections 2023-11-28T18:23:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cb80668f0 0x7f4cb8068db0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ccc0e71c0 0x7f4ccc0bec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ccc0dd290 0x7f4ccc047c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 --2- 172.21.15.7:0/198489337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ccc0467b0 0x7f4ccc0cadc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 -- 172.21.15.7:0/198489337 >> 172.21.15.7:0/198489337 conn(0x7f4ccc050160 msgr2=0x7f4ccc050540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 -- 172.21.15.7:0/198489337 shutdown_connections 2023-11-28T18:23:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fae340052d0 con 0x7fae34045940 2023-11-28T18:23:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f834c005510 con 0x7f836c066e00 2023-11-28T18:23:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.340+0000 7f4cbdffb640 1 -- 172.21.15.7:0/198489337 wait complete. 2023-11-28T18:23:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.344+0000 7f83697fa640 1 -- 172.21.15.7:0/3522520496 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f835c055220 con 0x7f836c066e00 2023-11-28T18:23:17.348 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2023-11-28T18:23:17.349 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2023-11-28T18:23:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.344+0000 7fae32ffd640 1 -- 172.21.15.7:0/3083257073 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fae38017020 con 0x7fae34045940 2023-11-28T18:23:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.344+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc8ec072710 con 0x7fc8ec1571d0 2023-11-28T18:23:17.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.344+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83580669a0 msgr2=0x7f8358068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.344+0000 7f8372e46640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83580669a0 0x7f8358068e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f836c1583d0 tx=0x7f8360008040 comp rx=0 tx=0).stop 2023-11-28T18:23:17.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c066e00 msgr2=0x7f836c13b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c066e00 0x7f836c13b000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f835c00e910 tx=0x7f835c00edd0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fc8e27fc640 1 -- 172.21.15.7:0/3553869385 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc8ec072710 con 0x7fc8ec1571d0 2023-11-28T18:23:17.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae200669a0 msgr2=0x7fae20068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae200669a0 0x7fae20068e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0695f0 tx=0x7fae3c07bd50 comp rx=0 tx=0).stop 2023-11-28T18:23:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae34045940 msgr2=0x7fae34046030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae34045940 0x7fae34046030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fae3800c370 tx=0x7fae3800c770 comp rx=0 tx=0).stop 2023-11-28T18:23:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 shutdown_connections 2023-11-28T18:23:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae200669a0 0x7fae20068e60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae340dd4c0 0x7fae340d65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae3404c6a0 0x7fae340d6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 --2- 172.21.15.7:0/3083257073 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae34045940 0x7fae34046030 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 >> 172.21.15.7:0/3083257073 conn(0x7fae34050190 msgr2=0x7fae3404e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 shutdown_connections 2023-11-28T18:23:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f83580669a0 0x7f8358068e60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f836c1571d0 0x7f836c072c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f836c155d40 0x7f836c0726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 --2- 172.21.15.7:0/3522520496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836c066e00 0x7f836c13b000 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 >> 172.21.15.7:0/3522520496 conn(0x7f836c0662e0 msgr2=0x7f836c140bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2023-11-28T18:23:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 shutdown_connections 2023-11-28T18:23:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7f8372e46640 1 -- 172.21.15.7:0/3522520496 wait complete. 2023-11-28T18:23:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 shutdown_connections 2023-11-28T18:23:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.348+0000 7fae443d8640 1 -- 172.21.15.7:0/3083257073 wait complete. 2023-11-28T18:23:17.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.352+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8d0066920 msgr2=0x7fc8d0068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.352+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8d0066920 0x7fc8d0068de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc8e80ae000 tx=0x7fc8e8008040 comp rx=0 tx=0).stop 2023-11-28T18:23:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.352+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec1571d0 msgr2=0x7fc8ec0729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.352+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec1571d0 0x7fc8ec0729f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400ba10 tx=0x7fc8e400bed0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 shutdown_connections 2023-11-28T18:23:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8d0066920 0x7fc8d0068de0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc8ec1571d0 0x7fc8ec0729f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1500d0 0x7fc8ec0707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7fc8f40bb640 1 --2- 172.21.15.7:0/3553869385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8ec066e00 0x7fc8ec154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 >> 172.21.15.7:0/3553869385 conn(0x7fc8ec0662e0 msgr2=0x7fc8ec0666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f519c0c2280 con 0x7f519c1500d0 2023-11-28T18:23:17.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 shutdown_connections 2023-11-28T18:23:17.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7fc8f40bb640 1 -- 172.21.15.7:0/3553869385 wait complete. 2023-11-28T18:23:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.356+0000 7f5192ffd640 1 -- 172.21.15.7:0/4285410876 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f51940f00e0 con 0x7f519c1500d0 2023-11-28T18:23:17.362 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f51800668d0 msgr2=0x7f5180068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f51800668d0 0x7f5180068d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f519c155b60 tx=0x7f518c00a130 comp rx=0 tx=0).stop 2023-11-28T18:23:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 msgr2=0x7f519c144080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c144080 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f51940b0040 tx=0x7f519401a850 comp rx=0 tx=0).stop 2023-11-28T18:23:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 shutdown_connections 2023-11-28T18:23:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f51800668d0 0x7f5180068d90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f519c1571d0 0x7f519c1390f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f519c155d40 0x7f519c13eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 --2- 172.21.15.7:0/4285410876 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f519c1500d0 0x7f519c144080 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 >> 172.21.15.7:0/4285410876 conn(0x7f519c0662e0 msgr2=0x7f519c0666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.360+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 shutdown_connections 2023-11-28T18:23:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.364+0000 7f51a4432640 1 -- 172.21.15.7:0/4285410876 wait complete. 2023-11-28T18:23:17.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.364+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd27004e890 con 0x7fd27015f370 2023-11-28T18:23:17.375 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2023-11-28T18:23:17.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.368+0000 7fd26cff9640 1 -- 172.21.15.7:0/680450130 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd260019020 con 0x7fd27015f370 2023-11-28T18:23:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd25c0669a0 msgr2=0x7fd25c068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd25c0669a0 0x7fd25c068e60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd268004ad0 tx=0x7fd26800b040 comp rx=0 tx=0).stop 2023-11-28T18:23:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd27015f370 msgr2=0x7fd270136f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd27015f370 0x7fd270136f60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd270067090 tx=0x7fd260007420 comp rx=0 tx=0).stop 2023-11-28T18:23:17.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 shutdown_connections 2023-11-28T18:23:17.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd25c0669a0 0x7fd25c068e60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd27015f370 0x7fd270136f60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd27015a6a0 0x7fd2700cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 --2- 172.21.15.7:0/680450130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2580047d0 0x7fd27015b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 >> 172.21.15.7:0/680450130 conn(0x7fd2700662e0 msgr2=0x7fd270155790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 shutdown_connections 2023-11-28T18:23:17.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.372+0000 7fd276417640 1 -- 172.21.15.7:0/680450130 wait complete. 2023-11-28T18:23:17.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.376+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5c181f69f0 con 0x7f5c180c9f10 2023-11-28T18:23:17.381 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2023-11-28T18:23:17.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.376+0000 7f5c15ffb640 1 -- 172.21.15.7:0/4268751122 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5c181f69f0 con 0x7f5c180c9f10 2023-11-28T18:23:17.381 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5c040668d0 msgr2=0x7f5c04068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5c040668d0 0x7f5c04068d90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5c10007fd0 tx=0x7f5c10009040 comp rx=0 tx=0).stop 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5c180c9f10 msgr2=0x7f5c18147af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5c180c9f10 0x7f5c18147af0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c00c480 tx=0x7f5c0c00c940 comp rx=0 tx=0).stop 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 shutdown_connections 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5c040668d0 0x7f5c04068d90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c18153f20 0x7f5c180c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5c180c9f10 0x7f5c18147af0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 --2- 172.21.15.7:0/4268751122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c180c50d0 0x7f5c180c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:17.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 >> 172.21.15.7:0/4268751122 conn(0x7f5c1806c2b0 msgr2=0x7f5c180caae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:17.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 shutdown_connections 2023-11-28T18:23:17.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:17.380+0000 7f5c1f49a640 1 -- 172.21.15.7:0/4268751122 wait complete. 2023-11-28T18:23:17.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2023-11-28T18:23:17.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2023-11-28T18:23:17.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2023-11-28T18:23:17.401 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2023-11-28T18:23:17.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2023-11-28T18:23:17.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2023-11-28T18:23:17.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2023-11-28T18:23:18.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T18:23:18.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T18:23:18.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T18:23:18.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T18:23:18.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T18:23:18.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T18:23:18.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T18:23:18.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T18:23:18.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.604+0000 7f219e1c4640 1 Processor -- start 2023-11-28T18:23:18.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.608+0000 7f219e1c4640 1 -- start start 2023-11-28T18:23:18.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.608+0000 7f219e1c4640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f21980c3090 0x7f2198159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.608+0000 7f219e1c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2198155f10 con 0x7f21980c6170 2023-11-28T18:23:18.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.608+0000 7f2196ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f21980c3090 0x7f2198159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.608+0000 7f2196ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f21980c3090 0x7f2198159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50560/0 (socket says 172.21.15.7:50560) 2023-11-28T18:23:18.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.608+0000 7f2196ffd640 1 -- 172.21.15.7:0/2036389633 learned_addr learned my addr 172.21.15.7:0/2036389633 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.608+0000 7f219e1c4640 1 -- 172.21.15.7:0/2036389633 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21981561d0 con 0x7f219814fef0 2023-11-28T18:23:18.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.608+0000 7f219e1c4640 1 -- 172.21.15.7:0/2036389633 --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2198155d20 con 0x7f21980c3090 2023-11-28T18:23:18.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.612+0000 7f2196ffd640 1 -- 172.21.15.7:0/2036389633 >> v1:172.21.15.193:6789/0 conn(0x7f219814fef0 legacy=0x7f21980c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.612+0000 7f2196ffd640 1 -- 172.21.15.7:0/2036389633 >> v1:172.21.15.7:6789/0 conn(0x7f21980c6170 legacy=0x7f21980c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.612+0000 7f2196ffd640 1 -- 172.21.15.7:0/2036389633 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21981564b0 con 0x7f21980c3090 2023-11-28T18:23:18.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.616+0000 7f2196ffd640 1 --2- 172.21.15.7:0/2036389633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c3090 0x7f2198159cd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f219000c2e0 tx=0x7f219001dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bef9b08d7988427c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.616+0000 7f21967fc640 1 -- 172.21.15.7:0/2036389633 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21900a6db0 con 0x7f21980c3090 2023-11-28T18:23:18.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.616+0000 7f21967fc640 1 -- 172.21.15.7:0/2036389633 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219000a0d0 con 0x7f21980c3090 2023-11-28T18:23:18.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.616+0000 7f21967fc640 1 -- 172.21.15.7:0/2036389633 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219000a460 con 0x7f21980c3090 2023-11-28T18:23:18.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.620+0000 7f219e1c4640 1 -- 172.21.15.7:0/2036389633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c3090 msgr2=0x7f2198159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.620+0000 7f219e1c4640 1 --2- 172.21.15.7:0/2036389633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c3090 0x7f2198159cd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f219000c2e0 tx=0x7f219001dd20 comp rx=0 tx=0).stop 2023-11-28T18:23:18.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.620+0000 7f219e1c4640 1 -- 172.21.15.7:0/2036389633 shutdown_connections 2023-11-28T18:23:18.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.620+0000 7f219e1c4640 1 --2- 172.21.15.7:0/2036389633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c3090 0x7f2198159cd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.620+0000 7f219e1c4640 1 -- 172.21.15.7:0/2036389633 >> 172.21.15.7:0/2036389633 conn(0x7f219806c2c0 msgr2=0x7f21980662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.620+0000 7f219e1c4640 1 -- 172.21.15.7:0/2036389633 shutdown_connections 2023-11-28T18:23:18.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.620+0000 7f219e1c4640 1 -- 172.21.15.7:0/2036389633 wait complete. 2023-11-28T18:23:18.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.624+0000 7f219e1c4640 1 Processor -- start 2023-11-28T18:23:18.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.624+0000 7f219e1c4640 1 -- start start 2023-11-28T18:23:18.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f219e1c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21980c3090 0x7f2198150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f219e1c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c6170 0x7f2198145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f219e1c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f219814fef0 0x7f2198145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f219e1c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21981f0c30 con 0x7f21980c3090 2023-11-28T18:23:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f219e1c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21981518e0 con 0x7f21980c6170 2023-11-28T18:23:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f219e1c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2198151b50 con 0x7f219814fef0 2023-11-28T18:23:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21980c3090 0x7f2198150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21980c3090 0x7f2198150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59622/0 (socket says 172.21.15.7:59622) 2023-11-28T18:23:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 -- 172.21.15.7:0/1005780872 learned_addr learned my addr 172.21.15.7:0/1005780872 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f2196ffd640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c6170 0x7f2198145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f2197fff640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f219814fef0 0x7f2198145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 -- 172.21.15.7:0/1005780872 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f219814fef0 msgr2=0x7f2198145fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f219814fef0 0x7f2198145fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 -- 172.21.15.7:0/1005780872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c6170 msgr2=0x7f2198145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c6170 0x7f2198145a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 -- 172.21.15.7:0/1005780872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219000c050 con 0x7f21980c3090 2023-11-28T18:23:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f21977fe640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21980c3090 0x7f2198150870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f218800b3e0 tx=0x7f218800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f217ffff640 1 -- 172.21.15.7:0/1005780872 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188011830 con 0x7f21980c3090 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2198151e30 con 0x7f21980c3090 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f217ffff640 1 -- 172.21.15.7:0/1005780872 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21880119b0 con 0x7f21980c3090 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f217ffff640 1 -- 172.21.15.7:0/1005780872 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188011d60 con 0x7f21980c3090 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.628+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21981523f0 con 0x7f21980c3090 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.632+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21980c8bd0 con 0x7f21980c3090 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.640+0000 7f217ffff640 1 -- 172.21.15.7:0/1005780872 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f218801a400 con 0x7f21980c3090 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.640+0000 7f217ffff640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21840668d0 0x7f2184068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.640+0000 7f217ffff640 1 -- 172.21.15.7:0/1005780872 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2188084460 con 0x7f21980c3090 2023-11-28T18:23:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.644+0000 7f2196ffd640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21840668d0 0x7f2184068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.644+0000 7f2196ffd640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21840668d0 0x7f2184068d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f21900af000 tx=0x7f21900a8070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.644+0000 7f217ffff640 1 -- 172.21.15.7:0/1005780872 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2188051740 con 0x7f21980c3090 2023-11-28T18:23:18.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.676+0000 7fb7e5b56640 1 Processor -- start 2023-11-28T18:23:18.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.676+0000 7fb7e5b56640 1 -- start start 2023-11-28T18:23:18.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7e5b56640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e0156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7e5b56640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb7e0157230 0x7fb7e01576a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7e5b56640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e00c7360 con 0x7fb7e006d780 2023-11-28T18:23:18.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7e5b56640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e0156c70 con 0x7fb7e0157230 2023-11-28T18:23:18.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7e5b56640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e0157be0 con 0x7fb7e006d3a0 2023-11-28T18:23:18.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7dffff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fb7e006d780 0x7fb7e006e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35682/0 (socket says 172.21.15.7:35682) 2023-11-28T18:23:18.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7dffff640 1 -- 172.21.15.7:0/3565091653 learned_addr learned my addr 172.21.15.7:0/3565091653 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7e5355640 1 --2- 172.21.15.7:0/3565091653 >> v2:172.21.15.7:3300/0 conn(0x7fb7e0157230 0x7fb7e01576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.680+0000 7fb7df7fe640 1 -- 172.21.15.7:0/3565091653 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546394457 0 0) 0x7fb7e00c7360 con 0x7fb7e006d780 2023-11-28T18:23:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.684+0000 7fb7df7fe640 1 -- 172.21.15.7:0/3565091653 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c8005740 con 0x7fb7e006d780 2023-11-28T18:23:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.684+0000 7fb7e5355640 1 -- 172.21.15.7:0/3565091653 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 msgr2=0x7fb7e0156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.684+0000 7fb7e5355640 1 --2- 172.21.15.7:0/3565091653 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e0156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.684+0000 7fb7e5355640 1 -- 172.21.15.7:0/3565091653 >> v1:172.21.15.193:6789/0 conn(0x7fb7e006d780 legacy=0x7fb7e006e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.684+0000 7fb7e5355640 1 -- 172.21.15.7:0/3565091653 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e00c5d10 con 0x7fb7e0157230 2023-11-28T18:23:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.688+0000 7fb7e5355640 1 --2- 172.21.15.7:0/3565091653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7e0157230 0x7fb7e01576a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7d000abc0 tx=0x7fb7d001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6037bef056d3ac50 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.688+0000 7fb7df7fe640 1 -- 172.21.15.7:0/3565091653 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d00a59c0 con 0x7fb7e0157230 2023-11-28T18:23:18.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7df7fe640 1 -- 172.21.15.7:0/3565091653 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d00a5b20 con 0x7fb7e0157230 2023-11-28T18:23:18.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7df7fe640 1 -- 172.21.15.7:0/3565091653 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d00a5e90 con 0x7fb7e0157230 2023-11-28T18:23:18.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fe171679640 1 Processor -- start 2023-11-28T18:23:18.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b92a5a640 1 Processor -- start 2023-11-28T18:23:18.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b92a5a640 1 -- start start 2023-11-28T18:23:18.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7e5b56640 1 -- 172.21.15.7:0/3565091653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7e0157230 msgr2=0x7fb7e01576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/3565091653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7e0157230 0x7fb7e01576a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7d000abc0 tx=0x7fb7d001c960 comp rx=0 tx=0).stop 2023-11-28T18:23:18.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b92a5a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7b8c154140 0x7f7b8c159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b92a5a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b8c0cb890 0x7f7b8c0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b92a5a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8c0c4b10 con 0x7f7b8c0cb420 2023-11-28T18:23:18.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b92a5a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c15a770 con 0x7f7b8c0cb890 2023-11-28T18:23:18.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b92a5a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c15aa50 con 0x7f7b8c154140 2023-11-28T18:23:18.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7e5b56640 1 -- 172.21.15.7:0/3565091653 shutdown_connections 2023-11-28T18:23:18.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/3565091653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7e0157230 0x7fb7e01576a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/3565091653 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e0156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7e5b56640 1 -- 172.21.15.7:0/3565091653 >> 172.21.15.7:0/3565091653 conn(0x7fb7e00664b0 msgr2=0x7fb7e00668c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b90ff7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b8c0cb890 0x7f7b8c0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fe171679640 1 -- start start 2023-11-28T18:23:18.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b90ff7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b8c0cb890 0x7f7b8c0ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59646/0 (socket says 172.21.15.7:59646) 2023-11-28T18:23:18.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b90ff7640 1 -- 172.21.15.7:0/1348137259 learned_addr learned my addr 172.21.15.7:0/1348137259 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b8bfff640 1 --2- 172.21.15.7:0/1348137259 >> v2:172.21.15.193:3300/0 conn(0x7f7b8c154140 0x7f7b8c159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7e5b56640 1 -- 172.21.15.7:0/3565091653 shutdown_connections 2023-11-28T18:23:18.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7fb7e5b56640 1 -- 172.21.15.7:0/3565091653 wait complete. 2023-11-28T18:23:18.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b90ff7640 1 -- 172.21.15.7:0/1348137259 >> v2:172.21.15.193:3300/0 conn(0x7f7b8c154140 msgr2=0x7f7b8c159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b90ff7640 1 --2- 172.21.15.7:0/1348137259 >> v2:172.21.15.193:3300/0 conn(0x7f7b8c154140 0x7f7b8c159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b90ff7640 1 -- 172.21.15.7:0/1348137259 >> v1:172.21.15.7:6789/0 conn(0x7f7b8c0cb420 legacy=0x7f7b8c072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f7b90ff7640 1 -- 172.21.15.7:0/1348137259 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c15ad30 con 0x7f7b8c0cb890 2023-11-28T18:23:18.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe171679640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c154140 0x7fe16c159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe171679640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe16c0cb420 0x7fe16c072620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe16affd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c154140 0x7fe16c159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe16affd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c154140 0x7fe16c159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38842/0 (socket says 172.21.15.7:38842) 2023-11-28T18:23:18.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe16b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe16c0cb860 0x7fe16c0ca500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55256/0 (socket says 172.21.15.7:55256) 2023-11-28T18:23:18.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f076445f640 1 Processor -- start 2023-11-28T18:23:18.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.692+0000 7f076445f640 1 -- start start 2023-11-28T18:23:18.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f076445f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c5d70 0x7f075c0c6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f076445f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c15e010 con 0x7f075c158480 2023-11-28T18:23:18.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f076445f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c0c80f0 con 0x7f075c0c7be0 2023-11-28T18:23:18.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f076445f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0c83d0 con 0x7f075c0c5d70 2023-11-28T18:23:18.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f0ca6565640 1 Processor -- start 2023-11-28T18:23:18.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fb7e5b56640 1 Processor -- start 2023-11-28T18:23:18.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fb7e5b56640 1 -- start start 2023-11-28T18:23:18.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f07629fc640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c5d70 0x7f075c0c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f07629fc640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c5d70 0x7f075c0c6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38852/0 (socket says 172.21.15.7:38852) 2023-11-28T18:23:18.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f07629fc640 1 -- 172.21.15.7:0/2738998208 learned_addr learned my addr 172.21.15.7:0/2738998208 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f0ca6565640 1 -- start start 2023-11-28T18:23:18.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe16affd640 1 -- 172.21.15.7:0/2043559105 learned_addr learned my addr 172.21.15.7:0/2043559105 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe171679640 1 -- 172.21.15.7:0/2043559105 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c0c4b10 con 0x7fe16c0cb860 2023-11-28T18:23:18.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe171679640 1 -- 172.21.15.7:0/2043559105 --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c15a770 con 0x7fe16c0cb420 2023-11-28T18:23:18.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fe171679640 1 -- 172.21.15.7:0/2043559105 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c15aa50 con 0x7fe16c154140 2023-11-28T18:23:18.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f07629fc640 1 -- 172.21.15.7:0/2738998208 >> v1:172.21.15.193:6789/0 conn(0x7f075c0c7be0 legacy=0x7f075c161600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f07629fc640 1 -- 172.21.15.7:0/2738998208 >> v1:172.21.15.7:6789/0 conn(0x7f075c158480 legacy=0x7f075c072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7f07629fc640 1 -- 172.21.15.7:0/2738998208 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c15e2d0 con 0x7f075c0c5d70 2023-11-28T18:23:18.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fb7e5b56640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e02025e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e5b56640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e006d780 0x7fb7e01f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e5b56640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7e0157230 0x7fb7e00725b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e5b56640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e0203530 con 0x7fb7e0157230 2023-11-28T18:23:18.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e5b56640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e014a490 con 0x7fb7e006d780 2023-11-28T18:23:18.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e5b56640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e00c4b50 con 0x7fb7e006d3a0 2023-11-28T18:23:18.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f7b90ff7640 1 --2- 172.21.15.7:0/1348137259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 0x7f7b8c0ca500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c08d4b0 tx=0x7f7b8401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84529d5b8669a239 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.696+0000 7fc67469a640 1 Processor -- start 2023-11-28T18:23:18.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e02025e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e02025e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38862/0 (socket says 172.21.15.7:38862) 2023-11-28T18:23:18.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 -- 172.21.15.7:0/1735660830 learned_addr learned my addr 172.21.15.7:0/1735660830 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fe16affd640 1 -- 172.21.15.7:0/2043559105 >> v2:172.21.15.7:3300/0 conn(0x7fe16c0cb420 msgr2=0x7fe16c072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fc67469a640 1 -- start start 2023-11-28T18:23:18.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7dffff640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e006d780 0x7fb7e01f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fe16affd640 1 --2- 172.21.15.7:0/2043559105 >> v2:172.21.15.7:3300/0 conn(0x7fe16c0cb420 0x7fe16c072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 -- 172.21.15.7:0/1735660830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e006d780 msgr2=0x7fb7e01f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e006d780 0x7fb7e01f4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 -- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7e0157230 msgr2=0x7fb7e00725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7e0157230 0x7fb7e00725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 -- 172.21.15.7:0/1735660830 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d000a5c0 con 0x7fb7e006d3a0 2023-11-28T18:23:18.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fe16a7fc640 1 -- 172.21.15.7:0/2043559105 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701212963 0 0) 0x7fe16c0c4b10 con 0x7fe16c0cb860 2023-11-28T18:23:18.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fe16affd640 1 -- 172.21.15.7:0/2043559105 >> v1:172.21.15.7:6789/0 conn(0x7fe16c0cb860 legacy=0x7fe16c0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f0ca6565640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ca01500d0 0x7f0ca0159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f0ca6565640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca015a260 con 0x7f0ca0155d40 2023-11-28T18:23:18.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f0ca6565640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca0159290 con 0x7f0ca01571d0 2023-11-28T18:23:18.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f0ca6565640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca01504b0 con 0x7f0ca01500d0 2023-11-28T18:23:18.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7e4b54640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e02025e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800ba80 tx=0x7fb7d800bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f0c9ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ca01500d0 0x7f0ca0159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f0c9ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ca01500d0 0x7f0ca0159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59676/0 (socket says 172.21.15.7:59676) 2023-11-28T18:23:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f0c9ffff640 1 -- 172.21.15.7:0/536999235 learned_addr learned my addr 172.21.15.7:0/536999235 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7dd7fa640 1 -- 172.21.15.7:0/1735660830 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d800cc00 con 0x7fb7e006d3a0 2023-11-28T18:23:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7dd7fa640 1 -- 172.21.15.7:0/1735660830 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d800cd60 con 0x7fb7e006d3a0 2023-11-28T18:23:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fb7dd7fa640 1 -- 172.21.15.7:0/1735660830 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8014680 con 0x7fb7e006d3a0 2023-11-28T18:23:18.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fe16affd640 1 -- 172.21.15.7:0/2043559105 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c15ad30 con 0x7fe16c154140 2023-11-28T18:23:18.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7fc67469a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc66c0c8410 0x7fc66c0c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f7b8affd640 1 -- 172.21.15.7:0/1348137259 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b840a5730 con 0x7f7b8c0cb890 2023-11-28T18:23:18.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f7b8affd640 1 -- 172.21.15.7:0/1348137259 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b840a5890 con 0x7f7b8c0cb890 2023-11-28T18:23:18.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.700+0000 7f7b8affd640 1 -- 172.21.15.7:0/1348137259 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b840a5c20 con 0x7f7b8c0cb890 2023-11-28T18:23:18.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc671c35640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc66c15e160 0x7fc66c0c21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35700/0 (socket says 172.21.15.7:35700) 2023-11-28T18:23:18.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076472677 0 0) 0x7f0ca0159290 con 0x7f0ca01571d0 2023-11-28T18:23:18.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e0202c90 con 0x7fb7e006d3a0 2023-11-28T18:23:18.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e0203250 con 0x7fb7e006d3a0 2023-11-28T18:23:18.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e004e890 con 0x7fb7e006d3a0 2023-11-28T18:23:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc672436640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc66c158450 0x7fc66c161f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55288/0 (socket says 172.21.15.7:55288) 2023-11-28T18:23:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c88003530 con 0x7f0ca01571d0 2023-11-28T18:23:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 955318230 0 0) 0x7f0c88003530 con 0x7f0ca01571d0 2023-11-28T18:23:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 >> v2:172.21.15.7:3300/0 conn(0x7f0ca01500d0 msgr2=0x7f0ca0159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 --2- 172.21.15.7:0/536999235 >> v2:172.21.15.7:3300/0 conn(0x7f0ca01500d0 0x7f0ca0159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 >> v1:172.21.15.7:6789/0 conn(0x7f0ca0155d40 legacy=0x7f0ca00c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca01565d0 con 0x7f0ca01571d0 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f0c94001680 con 0x7f0ca01571d0 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 >> v1:172.21.15.193:6789/0 conn(0x7f0ca01571d0 legacy=0x7f0ca00c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c880039a0 0x7f0c88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0c880047d0 0x7f0c88004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c880051e0 0x7f0c8800fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca01504b0 con 0x7f0c880039a0 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c88010540 con 0x7f0c880051e0 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c88010700 con 0x7f0c880047d0 2023-11-28T18:23:18.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0ca4b02640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c880051e0 0x7f0c8800fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f7b92a5a640 1 -- 172.21.15.7:0/1348137259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 msgr2=0x7f7b8c0ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f7b92a5a640 1 --2- 172.21.15.7:0/1348137259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 0x7f7b8c0ca500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c08d4b0 tx=0x7f7b8401c980 comp rx=0 tx=0).stop 2023-11-28T18:23:18.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc67469a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c15e710 con 0x7fc66c158450 2023-11-28T18:23:18.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc67469a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c0c7f10 con 0x7fc66c15e160 2023-11-28T18:23:18.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc67469a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c15e9d0 con 0x7fc66c0c8410 2023-11-28T18:23:18.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc672436640 1 -- 172.21.15.7:0/3999489600 learned_addr learned my addr 172.21.15.7:0/3999489600 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:18.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc672c37640 1 --2- 172.21.15.7:0/3999489600 >> v2:172.21.15.193:3300/0 conn(0x7fc66c0c8410 0x7fc66c0c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc671434640 1 -- 172.21.15.7:0/3999489600 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594395694 0 0) 0x7fc66c0c7f10 con 0x7fc66c15e160 2023-11-28T18:23:18.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc671434640 1 -- 172.21.15.7:0/3999489600 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc658003530 con 0x7fc66c15e160 2023-11-28T18:23:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f07629fc640 1 --2- 172.21.15.7:0/2738998208 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c5d70 0x7f075c0c6200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0754009e80 tx=0x7f075401c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20cfad53c1d0ca server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7f0c9ffff640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c880039a0 0x7f0c88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f7b92a5a640 1 -- 172.21.15.7:0/1348137259 shutdown_connections 2023-11-28T18:23:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f7b92a5a640 1 --2- 172.21.15.7:0/1348137259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 0x7f7b8c0ca500 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f7b92a5a640 1 --2- 172.21.15.7:0/1348137259 >> v2:172.21.15.193:3300/0 conn(0x7f7b8c154140 0x7f7b8c159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f7b92a5a640 1 -- 172.21.15.7:0/1348137259 >> 172.21.15.7:0/1348137259 conn(0x7f7b8c0cfaa0 msgr2=0x7f7b8c0cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc672c37640 1 -- 172.21.15.7:0/3999489600 >> v1:172.21.15.193:6789/0 conn(0x7fc66c15e160 legacy=0x7fc66c0c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc672c37640 1 -- 172.21.15.7:0/3999489600 >> v1:172.21.15.7:6789/0 conn(0x7fc66c158450 legacy=0x7fc66c161f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.704+0000 7fc672c37640 1 -- 172.21.15.7:0/3999489600 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c15ecb0 con 0x7fc66c0c8410 2023-11-28T18:23:18.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7fb7dd7fa640 1 -- 172.21.15.7:0/1735660830 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb7d80147e0 con 0x7fb7e006d3a0 2023-11-28T18:23:18.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7fe16affd640 1 --2- 172.21.15.7:0/2043559105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c154140 0x7fe16c159ec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe154009870 tx=0x7fe15401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e21389a759335c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7fb7dd7fa640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7cc0669a0 0x7fb7cc068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f07611f9640 1 -- 172.21.15.7:0/2738998208 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540a57a0 con 0x7f075c0c5d70 2023-11-28T18:23:18.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f07611f9640 1 -- 172.21.15.7:0/2738998208 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07540a5900 con 0x7f075c0c5d70 2023-11-28T18:23:18.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f07611f9640 1 -- 172.21.15.7:0/2738998208 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540a5c90 con 0x7f075c0c5d70 2023-11-28T18:23:18.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f076445f640 1 -- 172.21.15.7:0/2738998208 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c5d70 msgr2=0x7f075c0c6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f076445f640 1 --2- 172.21.15.7:0/2738998208 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c5d70 0x7f075c0c6200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0754009e80 tx=0x7f075401c920 comp rx=0 tx=0).stop 2023-11-28T18:23:18.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f076445f640 1 -- 172.21.15.7:0/2738998208 shutdown_connections 2023-11-28T18:23:18.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f076445f640 1 --2- 172.21.15.7:0/2738998208 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c5d70 0x7f075c0c6200 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f076445f640 1 -- 172.21.15.7:0/2738998208 >> 172.21.15.7:0/2738998208 conn(0x7f075c06c2c0 msgr2=0x7f075c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f076445f640 1 -- 172.21.15.7:0/2738998208 shutdown_connections 2023-11-28T18:23:18.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7fb7dffff640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7cc0669a0 0x7fb7cc068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f0c9ffff640 1 -- 172.21.15.7:0/536999235 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0c880047d0 msgr2=0x7f0c88004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f0c9ffff640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0c880047d0 0x7f0c88004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f0c9ffff640 1 -- 172.21.15.7:0/536999235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c880051e0 msgr2=0x7f0c8800fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f0c9ffff640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c880051e0 0x7f0c8800fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f0c9ffff640 1 -- 172.21.15.7:0/536999235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0159290 con 0x7f0c880039a0 2023-11-28T18:23:18.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f0ca4b02640 1 -- 172.21.15.7:0/536999235 reap_dead start 2023-11-28T18:23:18.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7fb7dd7fa640 1 -- 172.21.15.7:0/1735660830 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7d8084390 con 0x7fb7e006d3a0 2023-11-28T18:23:18.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7f076445f640 1 -- 172.21.15.7:0/2738998208 wait complete. 2023-11-28T18:23:18.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7fbae25b5640 1 Processor -- start 2023-11-28T18:23:18.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fbae25b5640 1 -- start start 2023-11-28T18:23:18.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.708+0000 7fb7dffff640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7cc0669a0 0x7fb7cc068e60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00c4f70 tx=0x7fb7d4007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc672c37640 1 --2- 172.21.15.7:0/3999489600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 0x7fc66c0c8880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6680097c0 tx=0x7fc66801ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4baef3ec6765dc18 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc671434640 1 -- 172.21.15.7:0/3999489600 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680a8930 con 0x7fc66c0c8410 2023-11-28T18:23:18.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc671434640 1 -- 172.21.15.7:0/3999489600 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6680a8a90 con 0x7fc66c0c8410 2023-11-28T18:23:18.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc671434640 1 -- 172.21.15.7:0/3999489600 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680a8e20 con 0x7fc66c0c8410 2023-11-28T18:23:18.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f0c9ffff640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c880039a0 0x7f0c88003de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009880 tx=0x7f0c9001c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c73fb5c132863ac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f7b92a5a640 1 -- 172.21.15.7:0/1348137259 shutdown_connections 2023-11-28T18:23:18.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f7b92a5a640 1 -- 172.21.15.7:0/1348137259 wait complete. 2023-11-28T18:23:18.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc67469a640 1 -- 172.21.15.7:0/3999489600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 msgr2=0x7fc66c0c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc67469a640 1 --2- 172.21.15.7:0/3999489600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 0x7fc66c0c8880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6680097c0 tx=0x7fc66801ec50 comp rx=0 tx=0).stop 2023-11-28T18:23:18.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c900a9660 con 0x7f0c880039a0 2023-11-28T18:23:18.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fe16a7fc640 1 -- 172.21.15.7:0/2043559105 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1540a8a30 con 0x7fe16c154140 2023-11-28T18:23:18.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fe16a7fc640 1 -- 172.21.15.7:0/2043559105 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1540a8b90 con 0x7fe16c154140 2023-11-28T18:23:18.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fe16a7fc640 1 -- 172.21.15.7:0/2043559105 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1540b0440 con 0x7fe16c154140 2023-11-28T18:23:18.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc67469a640 1 -- 172.21.15.7:0/3999489600 shutdown_connections 2023-11-28T18:23:18.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc67469a640 1 --2- 172.21.15.7:0/3999489600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 0x7fc66c0c8880 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc67469a640 1 -- 172.21.15.7:0/3999489600 >> 172.21.15.7:0/3999489600 conn(0x7fc66c0662e0 msgr2=0x7fc66c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c900a97c0 con 0x7f0c880039a0 2023-11-28T18:23:18.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f0c9effd640 1 -- 172.21.15.7:0/536999235 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c900a9b70 con 0x7f0c880039a0 2023-11-28T18:23:18.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fbae25b5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbadc157230 0x7fbadc1576c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fbae25b5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc0c7360 con 0x7fbadc06d780 2023-11-28T18:23:18.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fbae25b5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc157c00 con 0x7fbadc06d3a0 2023-11-28T18:23:18.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fbae25b5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbadc0c5d10 con 0x7fbadc157230 2023-11-28T18:23:18.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fb7dd7fa640 1 -- 172.21.15.7:0/1735660830 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb7d8051660 con 0x7fb7e006d3a0 2023-11-28T18:23:18.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fbae0db2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbadc06d780 0x7fbadc06e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55300/0 (socket says 172.21.15.7:55300) 2023-11-28T18:23:18.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fbae0db2640 1 -- 172.21.15.7:0/3916148662 learned_addr learned my addr 172.21.15.7:0/3916148662 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:18.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fbae1db4640 1 --2- 172.21.15.7:0/3916148662 >> v2:172.21.15.193:3300/0 conn(0x7fbadc157230 0x7fbadc1576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc67469a640 1 -- 172.21.15.7:0/3999489600 shutdown_connections 2023-11-28T18:23:18.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f0ca6565640 1 -- 172.21.15.7:0/536999235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c880039a0 msgr2=0x7f0c88003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f0ca6565640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c880039a0 0x7f0c88003de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009880 tx=0x7f0c9001c570 comp rx=0 tx=0).stop 2023-11-28T18:23:18.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7f076445f640 1 Processor -- start 2023-11-28T18:23:18.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fe171679640 1 -- 172.21.15.7:0/2043559105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c154140 msgr2=0x7fe16c159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fe171679640 1 --2- 172.21.15.7:0/2043559105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c154140 0x7fe16c159ec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe154009870 tx=0x7fe15401ed30 comp rx=0 tx=0).stop 2023-11-28T18:23:18.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350073119 0 0) 0x7fbadc0c7360 con 0x7fbadc06d780 2023-11-28T18:23:18.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac8003530 con 0x7fbadc06d780 2023-11-28T18:23:18.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173812358 0 0) 0x7fbadc157c00 con 0x7fbadc06d3a0 2023-11-28T18:23:18.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbadc0c7360 con 0x7fbadc06d3a0 2023-11-28T18:23:18.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.712+0000 7fc67469a640 1 -- 172.21.15.7:0/3999489600 wait complete. 2023-11-28T18:23:18.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 361255230 0 0) 0x7fbadc0c7360 con 0x7fbadc06d3a0 2023-11-28T18:23:18.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 >> v2:172.21.15.193:3300/0 conn(0x7fbadc157230 msgr2=0x7fbadc1576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f7b92a5a640 1 Processor -- start 2023-11-28T18:23:18.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 --2- 172.21.15.7:0/3916148662 >> v2:172.21.15.193:3300/0 conn(0x7fbadc157230 0x7fbadc1576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 >> v1:172.21.15.7:6789/0 conn(0x7fbadc06d780 legacy=0x7fbadc06e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc0c5ff0 con 0x7fbadc06d3a0 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fbacc001680 con 0x7fbadc06d3a0 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 >> v1:172.21.15.193:6789/0 conn(0x7fbadc06d3a0 legacy=0x7fbadc156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004ba0 0x7fbac8004f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8003770 0x7fbac8003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbac8004130 0x7fbac800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8004be0 con 0x7fbac8003770 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc0c5d10 con 0x7fbac8004ba0 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac800c1b0 con 0x7fbac8004130 2023-11-28T18:23:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbae1db4640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbac8004130 0x7fbac800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fe171679640 1 -- 172.21.15.7:0/2043559105 shutdown_connections 2023-11-28T18:23:18.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fe171679640 1 --2- 172.21.15.7:0/2043559105 >> v2:172.21.15.7:3300/0 conn(0x7fe16c0cb420 0x7fe16c072620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fe171679640 1 --2- 172.21.15.7:0/2043559105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c154140 0x7fe16c159ec0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fe171679640 1 -- 172.21.15.7:0/2043559105 >> 172.21.15.7:0/2043559105 conn(0x7fe16c0cfaa0 msgr2=0x7fe16c0cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f076445f640 1 -- start start 2023-11-28T18:23:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fc67469a640 1 Processor -- start 2023-11-28T18:23:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f0ca6565640 1 -- 172.21.15.7:0/536999235 shutdown_connections 2023-11-28T18:23:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f0ca6565640 1 --2- 172.21.15.7:0/536999235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c880039a0 0x7f0c88003de0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f0ca6565640 1 -- 172.21.15.7:0/536999235 >> 172.21.15.7:0/536999235 conn(0x7f0ca00662e0 msgr2=0x7f0ca00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbae0db2640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8003770 0x7fbac8003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fc67469a640 1 -- start start 2023-11-28T18:23:18.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f0ca6565640 1 -- 172.21.15.7:0/536999235 shutdown_connections 2023-11-28T18:23:18.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f7b92a5a640 1 -- start start 2023-11-28T18:23:18.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fe171679640 1 -- 172.21.15.7:0/2043559105 shutdown_connections 2023-11-28T18:23:18.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f076445f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075c0c5d70 0x7f075c159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f076445f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c7be0 0x7f075c160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f076445f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f075c158480 0x7f075c1433d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f076445f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c13d100 con 0x7f075c0c5d70 2023-11-28T18:23:18.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f076445f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0c9fc0 con 0x7f075c158480 2023-11-28T18:23:18.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f076445f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c142bf0 con 0x7f075c0c7be0 2023-11-28T18:23:18.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbae1db4640 1 -- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004ba0 msgr2=0x7fbac8004f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbae1db4640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004ba0 0x7fbac8004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbae1db4640 1 -- 172.21.15.7:0/3916148662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8003770 msgr2=0x7fbac8003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:18.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbae1db4640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8003770 0x7fbac8003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fbae1db4640 1 -- 172.21.15.7:0/3916148662 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc0c7360 con 0x7fbac8004130 2023-11-28T18:23:18.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7fe171679640 1 -- 172.21.15.7:0/2043559105 wait complete. 2023-11-28T18:23:18.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f075c158480 0x7f075c1433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc67469a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 0x7fc66c1f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc67469a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc66c158450 0x7fc66c151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc67469a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc66c15e160 0x7fc66c14f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f075c158480 0x7f075c1433d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50644/0 (socket says 172.21.15.7:50644) 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 -- 172.21.15.7:0/1962427910 learned_addr learned my addr 172.21.15.7:0/1962427910 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07619fa640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c7be0 0x7f075c160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 -- 172.21.15.7:0/1962427910 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c7be0 msgr2=0x7f075c160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c7be0 0x7f075c160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 -- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075c0c5d70 msgr2=0x7f075c159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075c0c5d70 0x7f075c159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 -- 172.21.15.7:0/1962427910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754009b30 con 0x7f075c158480 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f07629fc640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f075c158480 0x7f075c1433d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f07540af040 tx=0x7f07540196c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc67469a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0ca740 con 0x7fc66c158450 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc67469a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c14fe00 con 0x7fc66c0c8410 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc67469a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c1537c0 con 0x7fc66c15e160 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc672436640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 0x7fc66c1f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc671c35640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc66c158450 0x7fc66c151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc672c37640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc66c15e160 0x7fc66c14f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc671c35640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc66c158450 0x7fc66c151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59710/0 (socket says 172.21.15.7:59710) 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc672c37640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc66c15e160 0x7fc66c14f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38888/0 (socket says 172.21.15.7:38888) 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc671c35640 1 -- 172.21.15.7:0/3868183430 learned_addr learned my addr 172.21.15.7:0/3868183430 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.716+0000 7f0ca6565640 1 -- 172.21.15.7:0/536999235 wait complete. 2023-11-28T18:23:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f0ca6565640 1 Processor -- start 2023-11-28T18:23:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fe171679640 1 Processor -- start 2023-11-28T18:23:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc672c37640 1 -- 172.21.15.7:0/3868183430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 msgr2=0x7fc66c1f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc672c37640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 0x7fc66c1f7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc672c37640 1 -- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc66c158450 msgr2=0x7fc66c151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc672c37640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc66c158450 0x7fc66c151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fc672c37640 1 -- 172.21.15.7:0/3868183430 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668009420 con 0x7fc66c15e160 2023-11-28T18:23:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fbae1db4640 1 -- 172.21.15.7:0/3916148662 reap_dead start 2023-11-28T18:23:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7fbae1db4640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbac8004130 0x7fbac800b990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbad400ace0 tx=0x7fbad4020f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0cc8a59a5ee7c80 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fe171679640 1 -- start start 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f7b92a5a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b8c0cb420 0x7f7b8c0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f7b92a5a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 0x7f7b8c149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f7b92a5a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c154140 0x7f7b8c148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f7b92a5a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c159dd0 con 0x7f7b8c0cb890 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f7b92a5a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1576f0 con 0x7f7b8c154140 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f7b92a5a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c157ce0 con 0x7f7b8c0cb420 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b90ff7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c154140 0x7f7b8c148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b90ff7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c154140 0x7f7b8c148e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50666/0 (socket says 172.21.15.7:50666) 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b90ff7640 1 -- 172.21.15.7:0/2004119028 learned_addr learned my addr 172.21.15.7:0/2004119028 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b8bfff640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b8c0cb420 0x7f7b8c0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b8b7fe640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 0x7f7b8c149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b90ff7640 1 -- 172.21.15.7:0/2004119028 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b8c0cb420 msgr2=0x7f7b8c0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b90ff7640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b8c0cb420 0x7f7b8c0ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b90ff7640 1 -- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 msgr2=0x7f7b8c149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b90ff7640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 0x7f7b8c149f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f7b90ff7640 1 -- 172.21.15.7:0/2004119028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b84009b30 con 0x7f7b8c154140 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f074effd640 1 -- 172.21.15.7:0/1962427910 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540b8670 con 0x7f075c158480 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c161f90 con 0x7f075c158480 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c158d00 con 0x7f075c158480 2023-11-28T18:23:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f074effd640 1 -- 172.21.15.7:0/1962427910 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07540b87d0 con 0x7f075c158480 2023-11-28T18:23:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f074effd640 1 -- 172.21.15.7:0/1962427910 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540b8670 con 0x7f075c158480 2023-11-28T18:23:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0744005260 con 0x7f075c158480 2023-11-28T18:23:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fc672c37640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc66c15e160 0x7fc66c14f8c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc6680af000 tx=0x7fc66801b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fc662ffd640 1 -- 172.21.15.7:0/3868183430 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680b1ce0 con 0x7fc66c15e160 2023-11-28T18:23:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fc662ffd640 1 -- 172.21.15.7:0/3868183430 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6680b1e40 con 0x7fc66c15e160 2023-11-28T18:23:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.720+0000 7f0ca6565640 1 -- start start 2023-11-28T18:23:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca6565640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c880039a0 0x7f0ca0142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca6565640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ca01416e0 0x7f0ca01ee770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca6565640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0ca01eecb0 0x7f0ca014b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca6565640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca013f420 con 0x7f0ca01416e0 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca6565640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00c6590 con 0x7f0c880039a0 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca6565640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca01422e0 con 0x7f0ca01eecb0 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0ca01eecb0 0x7f0ca014b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0ca01eecb0 0x7f0ca014b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38910/0 (socket says 172.21.15.7:38910) 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 -- 172.21.15.7:0/911839055 learned_addr learned my addr 172.21.15.7:0/911839055 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 -- 172.21.15.7:0/911839055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c880039a0 msgr2=0x7f0ca0142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c880039a0 0x7f0ca0142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 -- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ca01416e0 msgr2=0x7f0ca01ee770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ca01416e0 0x7f0ca01ee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 -- 172.21.15.7:0/911839055 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0143630 con 0x7f0ca01eecb0 2023-11-28T18:23:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad40a9e40 con 0x7fbac8004130 2023-11-28T18:23:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad40039a0 con 0x7fbac8004130 2023-11-28T18:23:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fbad3fff640 1 -- 172.21.15.7:0/3916148662 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad4003d30 con 0x7fbac8004130 2023-11-28T18:23:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66c14ee70 con 0x7fc66c15e160 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe171679640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe16c0cb420 0x7fe16c0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe171679640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c0cb860 0x7fe16c1f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe171679640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe16c154140 0x7fe16c1f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe171679640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0c8c70 con 0x7fe16c0cb420 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe171679640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0c70e0 con 0x7fe16c154140 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe171679640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0c91c0 con 0x7fe16c0cb860 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c0cb860 0x7fe16c1f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c0cb860 0x7fe16c1f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38920/0 (socket says 172.21.15.7:38920) 2023-11-28T18:23:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 -- 172.21.15.7:0/1484561401 learned_addr learned my addr 172.21.15.7:0/1484561401 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7f7b90ff7640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c154140 0x7f7b8c148e20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b8401b030 tx=0x7f7b84019720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66c154340 con 0x7fc66c15e160 2023-11-28T18:23:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fc662ffd640 1 -- 172.21.15.7:0/3868183430 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680b1ce0 con 0x7fc66c15e160 2023-11-28T18:23:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0ca4b02640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0ca01eecb0 0x7f0ca014b6f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00c95f0 tx=0x7f0c9800e970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0c9cff9640 1 -- 172.21.15.7:0/911839055 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98006e30 con 0x7f0ca01eecb0 2023-11-28T18:23:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0c9cff9640 1 -- 172.21.15.7:0/911839055 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c98003980 con 0x7f0ca01eecb0 2023-11-28T18:23:18.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7f0c9cff9640 1 -- 172.21.15.7:0/911839055 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98003d30 con 0x7f0ca01eecb0 2023-11-28T18:23:18.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca01f26f0 con 0x7f0ca01eecb0 2023-11-28T18:23:18.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca0141c90 con 0x7f0ca01eecb0 2023-11-28T18:23:18.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fbae25b5640 1 -- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbac8004130 msgr2=0x7fbac800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.724+0000 7fbae25b5640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbac8004130 0x7fbac800b990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbad400ace0 tx=0x7fbad4020f30 comp rx=0 tx=0).stop 2023-11-28T18:23:18.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66c04e890 con 0x7fc66c15e160 2023-11-28T18:23:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 -- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe16c154140 msgr2=0x7fe16c1f7aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T18:23:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe16b7fe640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe16c154140 0x7fe16c1f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe16c154140 0x7fe16c1f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 -- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe16c0cb420 msgr2=0x7fe16c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe16c0cb420 0x7fe16c0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 -- 172.21.15.7:0/1484561401 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1540094d0 con 0x7fe16c0cb860 2023-11-28T18:23:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe1627fc640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c0cb860 0x7fe16c1f7560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00b550 tx=0x7fe15c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe16b7fe640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe16c154140 0x7fe16c1f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe163fff640 1 -- 172.21.15.7:0/1484561401 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c010870 con 0x7fe16c0cb860 2023-11-28T18:23:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe163fff640 1 -- 172.21.15.7:0/1484561401 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c0109d0 con 0x7fe16c0cb860 2023-11-28T18:23:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe163fff640 1 -- 172.21.15.7:0/1484561401 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c010d40 con 0x7fe16c0cb860 2023-11-28T18:23:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7f7b88ff9640 1 -- 172.21.15.7:0/2004119028 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b840b8670 con 0x7f7b8c154140 2023-11-28T18:23:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7f7b88ff9640 1 -- 172.21.15.7:0/2004119028 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b840b87d0 con 0x7f7b8c154140 2023-11-28T18:23:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7f7b88ff9640 1 -- 172.21.15.7:0/2004119028 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b840b8b60 con 0x7f7b8c154140 2023-11-28T18:23:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f074effd640 1 -- 172.21.15.7:0/1962427910 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f07540b8940 con 0x7f075c158480 2023-11-28T18:23:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fc662ffd640 1 -- 172.21.15.7:0/3868183430 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc6680b7040 con 0x7fc66c15e160 2023-11-28T18:23:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c1f80d0 con 0x7fe16c0cb860 2023-11-28T18:23:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c154be0 con 0x7fe16c0cb860 2023-11-28T18:23:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fc662ffd640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc650066930 0x7fc650068df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f074effd640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0740066930 0x7f0740068df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f074effd640 1 -- 172.21.15.7:0/1962427910 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f075411f690 con 0x7f075c158480 2023-11-28T18:23:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fc662ffd640 1 -- 172.21.15.7:0/3868183430 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc668121830 con 0x7fc66c15e160 2023-11-28T18:23:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fc672436640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc650066930 0x7fc650068df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7f0c9cff9640 1 -- 172.21.15.7:0/911839055 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0c98005070 con 0x7f0ca01eecb0 2023-11-28T18:23:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.728+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca00c98a0 con 0x7f0ca01eecb0 2023-11-28T18:23:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f0c9cff9640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0c8c0669d0 0x7f0c8c068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f0c9cff9640 1 -- 172.21.15.7:0/911839055 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c98083bb0 con 0x7f0ca01eecb0 2023-11-28T18:23:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f0c9ffff640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0c8c0669d0 0x7f0c8c068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f7b92a5a640 1 -- 172.21.15.7:0/2004119028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c158d40 con 0x7f7b8c154140 2023-11-28T18:23:18.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c154dd0 con 0x7fe16c0cb860 2023-11-28T18:23:18.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f7b92a5a640 1 -- 172.21.15.7:0/2004119028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c158260 con 0x7f7b8c154140 2023-11-28T18:23:18.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fc672436640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc650066930 0x7fc650068df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc65c005890 tx=0x7fc65c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fbae25b5640 1 -- 172.21.15.7:0/3916148662 shutdown_connections 2023-11-28T18:23:18.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fbae25b5640 1 --2- 172.21.15.7:0/3916148662 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbac8004130 0x7fbac800b990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7fbae25b5640 1 -- 172.21.15.7:0/3916148662 >> 172.21.15.7:0/3916148662 conn(0x7fbadc0664b0 msgr2=0x7fbadc0668c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f0c9ffff640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0c8c0669d0 0x7f0c8c068e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c900ae000 tx=0x7f0c90004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f07621fb640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0740066930 0x7f0740068df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.732+0000 7f7b727fc640 1 -- 172.21.15.7:0/2004119028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b64005260 con 0x7f7b8c154140 2023-11-28T18:23:18.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.736+0000 7f07621fb640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0740066930 0x7f0740068df0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0750009650 tx=0x7f0750006bb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.736+0000 7fbae25b5640 1 -- 172.21.15.7:0/3916148662 shutdown_connections 2023-11-28T18:23:18.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.736+0000 7fbae25b5640 1 -- 172.21.15.7:0/3916148662 wait complete. 2023-11-28T18:23:18.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.736+0000 7fbae25b5640 1 Processor -- start 2023-11-28T18:23:18.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.736+0000 7fbae25b5640 1 -- start start 2023-11-28T18:23:18.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae25b5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004130 0x7fbadc071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae25b5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbadc1f3060 0x7fbadc1f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae25b5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbadc147e30 0x7fbadc1e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae25b5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc0723e0 con 0x7fbadc147e30 2023-11-28T18:23:18.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae25b5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc1f6a70 con 0x7fbac8004130 2023-11-28T18:23:18.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae25b5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc0c2d50 con 0x7fbadc1f3060 2023-11-28T18:23:18.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fc662ffd640 1 -- 172.21.15.7:0/3868183430 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc6680eeb00 con 0x7fc66c15e160 2023-11-28T18:23:18.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae15b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004130 0x7fbadc071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae15b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004130 0x7fbadc071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50688/0 (socket says 172.21.15.7:50688) 2023-11-28T18:23:18.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae15b3640 1 -- 172.21.15.7:0/434864877 learned_addr learned my addr 172.21.15.7:0/434864877 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:18.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae0db2640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbadc1f3060 0x7fbadc1f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae1db4640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbadc147e30 0x7fbadc1e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7f074effd640 1 -- 172.21.15.7:0/1962427910 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f07540ec8c0 con 0x7f075c158480 2023-11-28T18:23:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae1db4640 1 -- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbadc1f3060 msgr2=0x7fbadc1f6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae1db4640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbadc1f3060 0x7fbadc1f6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae1db4640 1 -- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004130 msgr2=0x7fbadc071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae1db4640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004130 0x7fbadc071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae1db4640 1 -- 172.21.15.7:0/434864877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc0c24b0 con 0x7fbadc147e30 2023-11-28T18:23:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae0db2640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbadc1f3060 0x7fbadc1f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae15b3640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004130 0x7fbadc071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae1db4640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbadc147e30 0x7fbadc1e8d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbadc06fff0 tx=0x7fbac8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbad1ffb640 1 -- 172.21.15.7:0/434864877 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad40b2dc0 con 0x7fbadc147e30 2023-11-28T18:23:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbad1ffb640 1 -- 172.21.15.7:0/434864877 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad4015020 con 0x7fbadc147e30 2023-11-28T18:23:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbad1ffb640 1 -- 172.21.15.7:0/434864877 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad40153d0 con 0x7fbadc147e30 2023-11-28T18:23:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbadc0c2790 con 0x7fbadc147e30 2023-11-28T18:23:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.740+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbadc1f6700 con 0x7fbadc147e30 2023-11-28T18:23:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.744+0000 7f7b88ff9640 1 -- 172.21.15.7:0/2004119028 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7b840b8cc0 con 0x7f7b8c154140 2023-11-28T18:23:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.744+0000 7f7b88ff9640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b780669a0 0x7f7b78068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.744+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbadc04e890 con 0x7fbadc147e30 2023-11-28T18:23:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.744+0000 7f7b88ff9640 1 -- 172.21.15.7:0/2004119028 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b84120470 con 0x7f7b8c154140 2023-11-28T18:23:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.744+0000 7fbad1ffb640 1 -- 172.21.15.7:0/434864877 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbad40c0450 con 0x7fbadc147e30 2023-11-28T18:23:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.744+0000 7f7b8bfff640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b780669a0 0x7f7b78068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.744+0000 7fbad1ffb640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbac00668d0 0x7fbac0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.748+0000 7fbad1ffb640 1 -- 172.21.15.7:0/434864877 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbad411dda0 con 0x7fbadc147e30 2023-11-28T18:23:18.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.748+0000 7f7b8bfff640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b780669a0 0x7f7b78068e60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00b930 tx=0x7f7b7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.748+0000 7fbae15b3640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbac00668d0 0x7fbac0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.748+0000 7fbae15b3640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbac00668d0 0x7fbac0068d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00ab60 tx=0x7fbacc00aad0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.748+0000 7f0c9cff9640 1 -- 172.21.15.7:0/911839055 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0c98050e60 con 0x7f0ca01eecb0 2023-11-28T18:23:18.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.748+0000 7f7b88ff9640 1 -- 172.21.15.7:0/2004119028 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7b840ed7b0 con 0x7f7b8c154140 2023-11-28T18:23:18.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.752+0000 7fe163fff640 1 -- 172.21.15.7:0/1484561401 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe15c023480 con 0x7fe16c0cb860 2023-11-28T18:23:18.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.752+0000 7fe163fff640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1500668d0 0x7fe150068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:18.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.752+0000 7fe163fff640 1 -- 172.21.15.7:0/1484561401 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe15c0848d0 con 0x7fe16c0cb860 2023-11-28T18:23:18.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.752+0000 7fbad1ffb640 1 -- 172.21.15.7:0/434864877 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbad40eaff0 con 0x7fbadc147e30 2023-11-28T18:23:18.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.752+0000 7fe163fff640 1 -- 172.21.15.7:0/1484561401 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe15c051b80 con 0x7fe16c0cb860 2023-11-28T18:23:18.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.756+0000 7fe16affd640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1500668d0 0x7fe150068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:18.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.756+0000 7fe16affd640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1500668d0 0x7fe150068d90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe1540b2000 tx=0x7fe154004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.984+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f2198067cd0 con 0x7f21980c3090 2023-11-28T18:23:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.984+0000 7f217ffff640 1 -- 172.21.15.7:0/1005780872 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2188054d60 con 0x7f21980c3090 2023-11-28T18:23:18.991 INFO:teuthology.orchestra.run.smithi007.stdout:55834574856 2023-11-28T18:23:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.988+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21840668d0 msgr2=0x7f2184068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.988+0000 7f219e1c4640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21840668d0 0x7f2184068d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f21900af000 tx=0x7f21900a8070 comp rx=0 tx=0).stop 2023-11-28T18:23:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.988+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21980c3090 msgr2=0x7f2198150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.988+0000 7f219e1c4640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21980c3090 0x7f2198150870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f218800b3e0 tx=0x7f218800b8a0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.992+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 shutdown_connections 2023-11-28T18:23:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.992+0000 7f219e1c4640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21840668d0 0x7f2184068d90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.992+0000 7f219e1c4640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f219814fef0 0x7f2198145fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.992+0000 7f219e1c4640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21980c6170 0x7f2198145a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.992+0000 7f219e1c4640 1 --2- 172.21.15.7:0/1005780872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21980c3090 0x7f2198150870 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.992+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 >> 172.21.15.7:0/1005780872 conn(0x7f219806c2c0 msgr2=0x7f2198154390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.992+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 shutdown_connections 2023-11-28T18:23:18.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:18.992+0000 7f219e1c4640 1 -- 172.21.15.7:0/1005780872 wait complete. 2023-11-28T18:23:19.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2023-11-28T18:23:19.028 DEBUG:teuthology.parallel:result is None 2023-11-28T18:23:19.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.044+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0744005510 con 0x7f075c158480 2023-11-28T18:23:19.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f074effd640 1 -- 172.21.15.7:0/1962427910 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07540efee0 con 0x7f075c158480 2023-11-28T18:23:19.051 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:19.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0740066930 msgr2=0x7f0740068df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0740066930 0x7f0740068df0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0750009650 tx=0x7f0750006bb0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f075c158480 msgr2=0x7f075c1433d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f075c158480 0x7f075c1433d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f07540af040 tx=0x7f07540196c0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 shutdown_connections 2023-11-28T18:23:19.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0740066930 0x7f0740068df0 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0750009650 tx=0x7f0750006bb0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f075c158480 0x7f075c1433d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f075c0c7be0 0x7f075c160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 --2- 172.21.15.7:0/1962427910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075c0c5d70 0x7f075c159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 >> 172.21.15.7:0/1962427910 conn(0x7f075c06c2c0 msgr2=0x7f075c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 shutdown_connections 2023-11-28T18:23:19.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.048+0000 7f076445f640 1 -- 172.21.15.7:0/1962427910 wait complete. 2023-11-28T18:23:19.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.060+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb7e0155060 con 0x7fb7e006d3a0 2023-11-28T18:23:19.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.064+0000 7fb7dd7fa640 1 -- 172.21.15.7:0/1735660830 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb7d8054c80 con 0x7fb7e006d3a0 2023-11-28T18:23:19.070 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7cc0669a0 msgr2=0x7fb7cc068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7cc0669a0 0x7fb7cc068e60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00c4f70 tx=0x7fb7d4007e40 comp rx=0 tx=0).stop 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 msgr2=0x7fb7e02025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e02025e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800ba80 tx=0x7fb7d800bf40 comp rx=0 tx=0).stop 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 shutdown_connections 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7cc0669a0 0x7fb7cc068e60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7e0157230 0x7fb7e00725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e006d780 0x7fb7e01f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 --2- 172.21.15.7:0/1735660830 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7e006d3a0 0x7fb7e02025e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 >> 172.21.15.7:0/1735660830 conn(0x7fb7e00664b0 msgr2=0x7fb7e0146fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 shutdown_connections 2023-11-28T18:23:19.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.068+0000 7fb7e5b56640 1 -- 172.21.15.7:0/1735660830 wait complete. 2023-11-28T18:23:19.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.072+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc66c04e890 con 0x7fc66c15e160 2023-11-28T18:23:19.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.072+0000 7fc662ffd640 1 -- 172.21.15.7:0/3868183430 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc6680b5090 con 0x7fc66c15e160 2023-11-28T18:23:19.077 INFO:teuthology.orchestra.run.smithi007.stdout:55834574856 2023-11-28T18:23:19.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2023-11-28T18:23:19.077 DEBUG:teuthology.parallel:result is None 2023-11-28T18:23:19.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.072+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc650066930 msgr2=0x7fc650068df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.072+0000 7fc67469a640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc650066930 0x7fc650068df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc65c005890 tx=0x7fc65c009040 comp rx=0 tx=0).stop 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.072+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc66c15e160 msgr2=0x7fc66c14f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.072+0000 7fc67469a640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc66c15e160 0x7fc66c14f8c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc6680af000 tx=0x7fc66801b9f0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.072+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 shutdown_connections 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.072+0000 7fc67469a640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc650066930 0x7fc650068df0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.076+0000 7fc67469a640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc66c15e160 0x7fc66c14f8c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.076+0000 7fc67469a640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc66c158450 0x7fc66c151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.076+0000 7fc67469a640 1 --2- 172.21.15.7:0/3868183430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc66c0c8410 0x7fc66c1f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.076+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 >> 172.21.15.7:0/3868183430 conn(0x7fc66c0662e0 msgr2=0x7fc66c0c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.076+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 shutdown_connections 2023-11-28T18:23:19.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.076+0000 7fc67469a640 1 -- 172.21.15.7:0/3868183430 wait complete. 2023-11-28T18:23:19.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.084+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0ca004e890 con 0x7f0ca01eecb0 2023-11-28T18:23:19.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.084+0000 7f0c9cff9640 1 -- 172.21.15.7:0/911839055 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0c98019090 con 0x7f0ca01eecb0 2023-11-28T18:23:19.090 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:19.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.084+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbadc1f6bb0 con 0x7fbadc147e30 2023-11-28T18:23:19.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.084+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe16c04e890 con 0x7fe16c0cb860 2023-11-28T18:23:19.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0c8c0669d0 msgr2=0x7f0c8c068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe163fff640 1 -- 172.21.15.7:0/1484561401 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe15c0551a0 con 0x7fe16c0cb860 2023-11-28T18:23:19.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0c8c0669d0 0x7f0c8c068e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c900ae000 tx=0x7f0c90004740 comp rx=0 tx=0).stop 2023-11-28T18:23:19.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbad1ffb640 1 -- 172.21.15.7:0/434864877 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbad40af020 con 0x7fbadc147e30 2023-11-28T18:23:19.093 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:19.093 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-11-28T18:23:19.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0ca01eecb0 msgr2=0x7f0ca014b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0ca01eecb0 0x7f0ca014b6f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00c95f0 tx=0x7f0c9800e970 comp rx=0 tx=0).stop 2023-11-28T18:23:19.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 shutdown_connections 2023-11-28T18:23:19.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0c8c0669d0 0x7f0c8c068e90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0ca01eecb0 0x7f0ca014b6f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ca01416e0 0x7f0ca01ee770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 --2- 172.21.15.7:0/911839055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c880039a0 0x7f0ca0142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 >> 172.21.15.7:0/911839055 conn(0x7f0ca00662e0 msgr2=0x7f0ca00cac60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 shutdown_connections 2023-11-28T18:23:19.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f0ca6565640 1 -- 172.21.15.7:0/911839055 wait complete. 2023-11-28T18:23:19.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f7b727fc640 1 -- 172.21.15.7:0/2004119028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7b640057e0 con 0x7f7b8c154140 2023-11-28T18:23:19.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7f7b88ff9640 1 -- 172.21.15.7:0/2004119028 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7b840f0dd0 con 0x7f7b8c154140 2023-11-28T18:23:19.096 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2023-11-28T18:23:19.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbac00668d0 msgr2=0x7fbac0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbac00668d0 0x7fbac0068d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00ab60 tx=0x7fbacc00aad0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbadc147e30 msgr2=0x7fbadc1e8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbadc147e30 0x7fbadc1e8d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbadc06fff0 tx=0x7fbac8005800 comp rx=0 tx=0).stop 2023-11-28T18:23:19.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1500668d0 msgr2=0x7fe150068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1500668d0 0x7fe150068d90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe1540b2000 tx=0x7fe154004740 comp rx=0 tx=0).stop 2023-11-28T18:23:19.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c0cb860 msgr2=0x7fe16c1f7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c0cb860 0x7fe16c1f7560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00b550 tx=0x7fe15c00ba10 comp rx=0 tx=0).stop 2023-11-28T18:23:19.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 shutdown_connections 2023-11-28T18:23:19.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbac00668d0 0x7fbac0068d90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbadc147e30 0x7fbadc1e8d40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbadc1f3060 0x7fbadc1f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 --2- 172.21.15.7:0/434864877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbac8004130 0x7fbadc071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 >> 172.21.15.7:0/434864877 conn(0x7fbadc0664b0 msgr2=0x7fbadc066890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 shutdown_connections 2023-11-28T18:23:19.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1500668d0 0x7fe150068d90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe16c154140 0x7fe16c1f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe16c0cb860 0x7fe16c1f7560 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 --2- 172.21.15.7:0/1484561401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe16c0cb420 0x7fe16c0c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.088+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 >> 172.21.15.7:0/1484561401 conn(0x7fe16c0cfaa0 msgr2=0x7fe16c151920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 shutdown_connections 2023-11-28T18:23:19.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 shutdown_connections 2023-11-28T18:23:19.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7fe171679640 1 -- 172.21.15.7:0/1484561401 wait complete. 2023-11-28T18:23:19.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7fbae25b5640 1 -- 172.21.15.7:0/434864877 wait complete. 2023-11-28T18:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 -- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b780669a0 msgr2=0x7f7b78068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b780669a0 0x7f7b78068e60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00b930 tx=0x7f7b7c008040 comp rx=0 tx=0).stop 2023-11-28T18:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 -- 172.21.15.7:0/2004119028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c154140 msgr2=0x7f7b8c148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c154140 0x7f7b8c148e20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b8401b030 tx=0x7f7b84019720 comp rx=0 tx=0).stop 2023-11-28T18:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 -- 172.21.15.7:0/2004119028 shutdown_connections 2023-11-28T18:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b780669a0 0x7f7b78068e60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c154140 0x7f7b8c148e20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b8c0cb890 0x7f7b8c149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 --2- 172.21.15.7:0/2004119028 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b8c0cb420 0x7f7b8c0ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 -- 172.21.15.7:0/2004119028 >> 172.21.15.7:0/2004119028 conn(0x7f7b8c0cfaa0 msgr2=0x7f7b8c0b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 -- 172.21.15.7:0/2004119028 shutdown_connections 2023-11-28T18:23:19.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.092+0000 7f7b727fc640 1 -- 172.21.15.7:0/2004119028 wait complete. 2023-11-28T18:23:19.103 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-11-28T18:23:19.103 DEBUG:teuthology.parallel:result is None 2023-11-28T18:23:19.114 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2023-11-28T18:23:19.115 DEBUG:teuthology.parallel:result is None 2023-11-28T18:23:19.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2023-11-28T18:23:19.128 DEBUG:teuthology.parallel:result is None 2023-11-28T18:23:19.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2023-11-28T18:23:19.130 DEBUG:teuthology.parallel:result is None 2023-11-28T18:23:19.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2023-11-28T18:23:19.136 DEBUG:teuthology.parallel:result is None 2023-11-28T18:23:19.136 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2023-11-28T18:23:19.136 DEBUG:teuthology.parallel:result is None 2023-11-28T18:23:19.136 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T18:23:19.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T18:23:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8f0109c640 1 Processor -- start 2023-11-28T18:23:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8f0109c640 1 -- start start 2023-11-28T18:23:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8f0109c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8efc0cae40 0x7f8efc0ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8f0109c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 0x7f8efc070f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8f0109c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8efc0cf9f0 0x7f8efc0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8f0109c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc15d4e0 con 0x7f8efc0cae40 2023-11-28T18:23:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8f0109c640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc0d03c0 con 0x7f8efc0cf9f0 2023-11-28T18:23:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8f0109c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0717f0 con 0x7f8efc0cb220 2023-11-28T18:23:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8efc0cae40 0x7f8efc0ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8efc0cae40 0x7f8efc0ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59754/0 (socket says 172.21.15.7:59754) 2023-11-28T18:23:19.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efad9d640 1 -- 172.21.15.7:0/1585348261 learned_addr learned my addr 172.21.15.7:0/1585348261 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:19.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8ef3fff640 1 --2- 172.21.15.7:0/1585348261 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 0x7f8efc070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efb59e640 1 --2- 172.21.15.7:0/1585348261 >> v2:172.21.15.193:3300/0 conn(0x7f8efc0cf9f0 0x7f8efc0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efad9d640 1 -- 172.21.15.7:0/1585348261 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 msgr2=0x7f8efc070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efad9d640 1 --2- 172.21.15.7:0/1585348261 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 0x7f8efc070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efad9d640 1 -- 172.21.15.7:0/1585348261 >> v2:172.21.15.193:3300/0 conn(0x7f8efc0cf9f0 msgr2=0x7f8efc0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efad9d640 1 --2- 172.21.15.7:0/1585348261 >> v2:172.21.15.193:3300/0 conn(0x7f8efc0cf9f0 0x7f8efc0cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efad9d640 1 -- 172.21.15.7:0/1585348261 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc15d730 con 0x7f8efc0cae40 2023-11-28T18:23:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.316+0000 7f8efb59e640 1 --2- 172.21.15.7:0/1585348261 >> v2:172.21.15.193:3300/0 conn(0x7f8efc0cf9f0 0x7f8efc0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:19.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8efad9d640 1 --2- 172.21.15.7:0/1585348261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc0ca4a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8efc071c20 tx=0x7f8eec01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1999863c76121ee6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8efa59c640 1 -- 172.21.15.7:0/1585348261 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0a89b0 con 0x7f8efc0cae40 2023-11-28T18:23:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8efa59c640 1 -- 172.21.15.7:0/1585348261 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec0a8b10 con 0x7f8efc0cae40 2023-11-28T18:23:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8efa59c640 1 -- 172.21.15.7:0/1585348261 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0a8ea0 con 0x7f8efc0cae40 2023-11-28T18:23:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 -- 172.21.15.7:0/1585348261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 msgr2=0x7f8efc0ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 --2- 172.21.15.7:0/1585348261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc0ca4a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8efc071c20 tx=0x7f8eec01ecf0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 -- 172.21.15.7:0/1585348261 shutdown_connections 2023-11-28T18:23:19.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 --2- 172.21.15.7:0/1585348261 >> v2:172.21.15.193:3300/0 conn(0x7f8efc0cf9f0 0x7f8efc0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 --2- 172.21.15.7:0/1585348261 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 0x7f8efc070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 --2- 172.21.15.7:0/1585348261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc0ca4a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 -- 172.21.15.7:0/1585348261 >> 172.21.15.7:0/1585348261 conn(0x7f8efc072310 msgr2=0x7f8efc072720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 -- 172.21.15.7:0/1585348261 shutdown_connections 2023-11-28T18:23:19.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.320+0000 7f8f0109c640 1 -- 172.21.15.7:0/1585348261 wait complete. 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 Processor -- start 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 -- start start 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 0x7f8efc1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc0cf9f0 0x7f8efc1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc1f0150 con 0x7f8efc0cae40 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc1f0640 con 0x7f8efc0cf9f0 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc154710 con 0x7f8efc0cb220 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59766/0 (socket says 172.21.15.7:59766) 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 -- 172.21.15.7:0/499466231 learned_addr learned my addr 172.21.15.7:0/499466231 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efb59e640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc0cf9f0 0x7f8efc1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8ef3fff640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 0x7f8efc1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 -- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 msgr2=0x7f8efc1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 0x7f8efc1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 -- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc0cf9f0 msgr2=0x7f8efc1556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc0cf9f0 0x7f8efc1556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 -- 172.21.15.7:0/499466231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec0094d0 con 0x7f8efc0cae40 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efb59e640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc0cf9f0 0x7f8efc1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8efad9d640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc156650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8eec01d3a0 tx=0x7f8eec01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8ef37fe640 1 -- 172.21.15.7:0/499466231 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0ba600 con 0x7f8efc0cae40 2023-11-28T18:23:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8ef37fe640 1 -- 172.21.15.7:0/499466231 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec0ba760 con 0x7f8efc0cae40 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc1f0360 con 0x7f8efc0cae40 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8ef37fe640 1 -- 172.21.15.7:0/499466231 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0bab10 con 0x7f8efc0cae40 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.324+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc154c90 con 0x7f8efc0cae40 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.328+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc04e890 con 0x7f8efc0cae40 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.328+0000 7f8ef37fe640 1 -- 172.21.15.7:0/499466231 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8eec0bac70 con 0x7f8efc0cae40 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.328+0000 7f8ef37fe640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8edc0668d0 0x7f8edc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.328+0000 7f8ef37fe640 1 -- 172.21.15.7:0/499466231 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8eec1229f0 con 0x7f8efc0cae40 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.328+0000 7f8ef3fff640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8edc0668d0 0x7f8edc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.328+0000 7f8ef3fff640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8edc0668d0 0x7f8edc068d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8efc15d410 tx=0x7f8ee4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:19.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.332+0000 7f8ef37fe640 1 -- 172.21.15.7:0/499466231 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8eec0efb90 con 0x7f8efc0cae40 2023-11-28T18:23:19.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.492+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 --> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8efc154240 con 0x7f8edc0668d0 2023-11-28T18:23:19.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.500+0000 7f8ef37fe640 1 -- 172.21.15.7:0/499466231 <== mgr.4102 v2:172.21.15.7:6832/264069339 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264558 (secure 0 0 0) 0x7f8efc154240 con 0x7f8edc0668d0 2023-11-28T18:23:19.507 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:19.508 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-28T18:23:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8edc0668d0 msgr2=0x7f8edc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8edc0668d0 0x7f8edc068d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8efc15d410 tx=0x7f8ee4008040 comp rx=0 tx=0).stop 2023-11-28T18:23:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 msgr2=0x7f8efc156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc156650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8eec01d3a0 tx=0x7f8eec01ba90 comp rx=0 tx=0).stop 2023-11-28T18:23:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 shutdown_connections 2023-11-28T18:23:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8edc0668d0 0x7f8edc068d90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc0cf9f0 0x7f8efc1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8efc0cb220 0x7f8efc1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 --2- 172.21.15.7:0/499466231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8efc0cae40 0x7f8efc156650 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 >> 172.21.15.7:0/499466231 conn(0x7f8efc072310 msgr2=0x7f8efc0c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 shutdown_connections 2023-11-28T18:23:19.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.504+0000 7f8f0109c640 1 -- 172.21.15.7:0/499466231 wait complete. 2023-11-28T18:23:19.527 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-11-28T18:23:19.463797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235592,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739128,"statfs":{"total":773094113280,"available":772852867072,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.360594"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001273+0000","last_change":"2023-11-28T18:23:08.134633+0000","last_active":"2023-11-28T18:23:11.001273+0000","last_peered":"2023-11-28T18:23:11.001273+0000","last_clean":"2023-11-28T18:23:11.001273+0000","last_became_active":"2023-11-28T18:23:08.134320+0000","last_became_peered":"2023-11-28T18:23:08.134320+0000","last_unstale":"2023-11-28T18:23:11.001273+0000","last_undegraded":"2023-11-28T18:23:11.001273+0000","last_fullsized":"2023-11-28T18:23:11.001273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:54.995718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105857+0000","last_change":"2023-11-28T18:23:08.122460+0000","last_active":"2023-11-28T18:23:09.105857+0000","last_peered":"2023-11-28T18:23:09.105857+0000","last_clean":"2023-11-28T18:23:09.105857+0000","last_became_active":"2023-11-28T18:23:08.122153+0000","last_became_peered":"2023-11-28T18:23:08.122153+0000","last_unstale":"2023-11-28T18:23:09.105857+0000","last_undegraded":"2023-11-28T18:23:09.105857+0000","last_fullsized":"2023-11-28T18:23:09.105857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:17:31.074978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914207+0000","last_change":"2023-11-28T18:23:08.110187+0000","last_active":"2023-11-28T18:23:10.914207+0000","last_peered":"2023-11-28T18:23:10.914207+0000","last_clean":"2023-11-28T18:23:10.914207+0000","last_became_active":"2023-11-28T18:23:08.109915+0000","last_became_peered":"2023-11-28T18:23:08.109915+0000","last_unstale":"2023-11-28T18:23:10.914207+0000","last_undegraded":"2023-11-28T18:23:10.914207+0000","last_fullsized":"2023-11-28T18:23:10.914207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:32.098408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113335+0000","last_change":"2023-11-28T18:23:08.122000+0000","last_active":"2023-11-28T18:23:09.113335+0000","last_peered":"2023-11-28T18:23:09.113335+0000","last_clean":"2023-11-28T18:23:09.113335+0000","last_became_active":"2023-11-28T18:23:08.121652+0000","last_became_peered":"2023-11-28T18:23:08.121652+0000","last_unstale":"2023-11-28T18:23:09.113335+0000","last_undegraded":"2023-11-28T18:23:09.113335+0000","last_fullsized":"2023-11-28T18:23:09.113335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:15.540606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000810+0000","last_change":"2023-11-28T18:23:08.126587+0000","last_active":"2023-11-28T18:23:11.000810+0000","last_peered":"2023-11-28T18:23:11.000810+0000","last_clean":"2023-11-28T18:23:11.000810+0000","last_became_active":"2023-11-28T18:23:08.126313+0000","last_became_peered":"2023-11-28T18:23:08.126313+0000","last_unstale":"2023-11-28T18:23:11.000810+0000","last_undegraded":"2023-11-28T18:23:11.000810+0000","last_fullsized":"2023-11-28T18:23:11.000810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.956197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113040+0000","last_change":"2023-11-28T18:23:08.122878+0000","last_active":"2023-11-28T18:23:09.113040+0000","last_peered":"2023-11-28T18:23:09.113040+0000","last_clean":"2023-11-28T18:23:09.113040+0000","last_became_active":"2023-11-28T18:23:08.122609+0000","last_became_peered":"2023-11-28T18:23:08.122609+0000","last_unstale":"2023-11-28T18:23:09.113040+0000","last_undegraded":"2023-11-28T18:23:09.113040+0000","last_fullsized":"2023-11-28T18:23:09.113040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:00.610074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000604+0000","last_change":"2023-11-28T18:23:08.133485+0000","last_active":"2023-11-28T18:23:11.000604+0000","last_peered":"2023-11-28T18:23:11.000604+0000","last_clean":"2023-11-28T18:23:11.000604+0000","last_became_active":"2023-11-28T18:23:08.133211+0000","last_became_peered":"2023-11-28T18:23:08.133211+0000","last_unstale":"2023-11-28T18:23:11.000604+0000","last_undegraded":"2023-11-28T18:23:11.000604+0000","last_fullsized":"2023-11-28T18:23:11.000604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:04:33.507685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565355+0000","last_change":"2023-11-28T18:23:08.117903+0000","last_active":"2023-11-28T18:23:09.565355+0000","last_peered":"2023-11-28T18:23:09.565355+0000","last_clean":"2023-11-28T18:23:09.565355+0000","last_became_active":"2023-11-28T18:23:08.117482+0000","last_became_peered":"2023-11-28T18:23:08.117482+0000","last_unstale":"2023-11-28T18:23:09.565355+0000","last_undegraded":"2023-11-28T18:23:09.565355+0000","last_fullsized":"2023-11-28T18:23:09.565355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:45.293402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001351+0000","last_change":"2023-11-28T18:23:08.135606+0000","last_active":"2023-11-28T18:23:11.001351+0000","last_peered":"2023-11-28T18:23:11.001351+0000","last_clean":"2023-11-28T18:23:11.001351+0000","last_became_active":"2023-11-28T18:23:08.135328+0000","last_became_peered":"2023-11-28T18:23:08.135328+0000","last_unstale":"2023-11-28T18:23:11.001351+0000","last_undegraded":"2023-11-28T18:23:11.001351+0000","last_fullsized":"2023-11-28T18:23:11.001351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:25.011869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110046+0000","last_change":"2023-11-28T18:23:08.123993+0000","last_active":"2023-11-28T18:23:09.110046+0000","last_peered":"2023-11-28T18:23:09.110046+0000","last_clean":"2023-11-28T18:23:09.110046+0000","last_became_active":"2023-11-28T18:23:08.123583+0000","last_became_peered":"2023-11-28T18:23:08.123583+0000","last_unstale":"2023-11-28T18:23:09.110046+0000","last_undegraded":"2023-11-28T18:23:09.110046+0000","last_fullsized":"2023-11-28T18:23:09.110046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:16:33.089307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:08.120983+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:08.120486+0000","last_became_peered":"2023-11-28T18:23:08.120486+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:46:12.881629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637634+0000","last_change":"2023-11-28T18:23:08.130169+0000","last_active":"2023-11-28T18:23:09.637634+0000","last_peered":"2023-11-28T18:23:09.637634+0000","last_clean":"2023-11-28T18:23:09.637634+0000","last_became_active":"2023-11-28T18:23:08.129771+0000","last_became_peered":"2023-11-28T18:23:08.129771+0000","last_unstale":"2023-11-28T18:23:09.637634+0000","last_undegraded":"2023-11-28T18:23:09.637634+0000","last_fullsized":"2023-11-28T18:23:09.637634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:45.753322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565505+0000","last_change":"2023-11-28T18:23:08.123268+0000","last_active":"2023-11-28T18:23:09.565505+0000","last_peered":"2023-11-28T18:23:09.565505+0000","last_clean":"2023-11-28T18:23:09.565505+0000","last_became_active":"2023-11-28T18:23:08.122978+0000","last_became_peered":"2023-11-28T18:23:08.122978+0000","last_unstale":"2023-11-28T18:23:09.565505+0000","last_undegraded":"2023-11-28T18:23:09.565505+0000","last_fullsized":"2023-11-28T18:23:09.565505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:15:01.320918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913943+0000","last_change":"2023-11-28T18:23:08.118361+0000","last_active":"2023-11-28T18:23:10.913943+0000","last_peered":"2023-11-28T18:23:10.913943+0000","last_clean":"2023-11-28T18:23:10.913943+0000","last_became_active":"2023-11-28T18:23:08.117973+0000","last_became_peered":"2023-11-28T18:23:08.117973+0000","last_unstale":"2023-11-28T18:23:10.913943+0000","last_undegraded":"2023-11-28T18:23:10.913943+0000","last_fullsized":"2023-11-28T18:23:10.913943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:57.485937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105530+0000","last_change":"2023-11-28T18:23:08.139744+0000","last_active":"2023-11-28T18:23:09.105530+0000","last_peered":"2023-11-28T18:23:09.105530+0000","last_clean":"2023-11-28T18:23:09.105530+0000","last_became_active":"2023-11-28T18:23:08.139464+0000","last_became_peered":"2023-11-28T18:23:08.139464+0000","last_unstale":"2023-11-28T18:23:09.105530+0000","last_undegraded":"2023-11-28T18:23:09.105530+0000","last_fullsized":"2023-11-28T18:23:09.105530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:19:34.780979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113036+0000","last_change":"2023-11-28T18:23:08.119103+0000","last_active":"2023-11-28T18:23:09.113036+0000","last_peered":"2023-11-28T18:23:09.113036+0000","last_clean":"2023-11-28T18:23:09.113036+0000","last_became_active":"2023-11-28T18:23:08.116080+0000","last_became_peered":"2023-11-28T18:23:08.116080+0000","last_unstale":"2023-11-28T18:23:09.113036+0000","last_undegraded":"2023-11-28T18:23:09.113036+0000","last_fullsized":"2023-11-28T18:23:09.113036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:44.279413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113688+0000","last_change":"2023-11-28T18:23:08.126817+0000","last_active":"2023-11-28T18:23:09.113688+0000","last_peered":"2023-11-28T18:23:09.113688+0000","last_clean":"2023-11-28T18:23:09.113688+0000","last_became_active":"2023-11-28T18:23:08.126540+0000","last_became_peered":"2023-11-28T18:23:08.126540+0000","last_unstale":"2023-11-28T18:23:09.113688+0000","last_undegraded":"2023-11-28T18:23:09.113688+0000","last_fullsized":"2023-11-28T18:23:09.113688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:08.807847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271940+0000","last_change":"2023-11-28T18:23:08.121775+0000","last_active":"2023-11-28T18:23:10.271940+0000","last_peered":"2023-11-28T18:23:10.271940+0000","last_clean":"2023-11-28T18:23:10.271940+0000","last_became_active":"2023-11-28T18:23:08.121466+0000","last_became_peered":"2023-11-28T18:23:08.121466+0000","last_unstale":"2023-11-28T18:23:10.271940+0000","last_undegraded":"2023-11-28T18:23:10.271940+0000","last_fullsized":"2023-11-28T18:23:10.271940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:32:51.612113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001115+0000","last_change":"2023-11-28T18:23:08.138282+0000","last_active":"2023-11-28T18:23:11.001115+0000","last_peered":"2023-11-28T18:23:11.001115+0000","last_clean":"2023-11-28T18:23:11.001115+0000","last_became_active":"2023-11-28T18:23:08.137171+0000","last_became_peered":"2023-11-28T18:23:08.137171+0000","last_unstale":"2023-11-28T18:23:11.001115+0000","last_undegraded":"2023-11-28T18:23:11.001115+0000","last_fullsized":"2023-11-28T18:23:11.001115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:39:07.113697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110124+0000","last_change":"2023-11-28T18:23:08.133521+0000","last_active":"2023-11-28T18:23:09.110124+0000","last_peered":"2023-11-28T18:23:09.110124+0000","last_clean":"2023-11-28T18:23:09.110124+0000","last_became_active":"2023-11-28T18:23:08.133248+0000","last_became_peered":"2023-11-28T18:23:08.133248+0000","last_unstale":"2023-11-28T18:23:09.110124+0000","last_undegraded":"2023-11-28T18:23:09.110124+0000","last_fullsized":"2023-11-28T18:23:09.110124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:17:34.585808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914195+0000","last_change":"2023-11-28T18:23:08.129150+0000","last_active":"2023-11-28T18:23:10.914195+0000","last_peered":"2023-11-28T18:23:10.914195+0000","last_clean":"2023-11-28T18:23:10.914195+0000","last_became_active":"2023-11-28T18:23:08.128622+0000","last_became_peered":"2023-11-28T18:23:08.128622+0000","last_unstale":"2023-11-28T18:23:10.914195+0000","last_undegraded":"2023-11-28T18:23:10.914195+0000","last_fullsized":"2023-11-28T18:23:10.914195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:33.197161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113274+0000","last_change":"2023-11-28T18:23:08.131333+0000","last_active":"2023-11-28T18:23:09.113274+0000","last_peered":"2023-11-28T18:23:09.113274+0000","last_clean":"2023-11-28T18:23:09.113274+0000","last_became_active":"2023-11-28T18:23:08.130917+0000","last_became_peered":"2023-11-28T18:23:08.130917+0000","last_unstale":"2023-11-28T18:23:09.113274+0000","last_undegraded":"2023-11-28T18:23:09.113274+0000","last_fullsized":"2023-11-28T18:23:09.113274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:04:48.114417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565507+0000","last_change":"2023-11-28T18:23:08.132355+0000","last_active":"2023-11-28T18:23:09.565507+0000","last_peered":"2023-11-28T18:23:09.565507+0000","last_clean":"2023-11-28T18:23:09.565507+0000","last_became_active":"2023-11-28T18:23:08.132016+0000","last_became_peered":"2023-11-28T18:23:08.132016+0000","last_unstale":"2023-11-28T18:23:09.565507+0000","last_undegraded":"2023-11-28T18:23:09.565507+0000","last_fullsized":"2023-11-28T18:23:09.565507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:11.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113066+0000","last_change":"2023-11-28T18:23:08.119935+0000","last_active":"2023-11-28T18:23:09.113066+0000","last_peered":"2023-11-28T18:23:09.113066+0000","last_clean":"2023-11-28T18:23:09.113066+0000","last_became_active":"2023-11-28T18:23:08.118372+0000","last_became_peered":"2023-11-28T18:23:08.118372+0000","last_unstale":"2023-11-28T18:23:09.113066+0000","last_undegraded":"2023-11-28T18:23:09.113066+0000","last_fullsized":"2023-11-28T18:23:09.113066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:52.339024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.132272+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.131985+0000","last_became_peered":"2023-11-28T18:23:08.131985+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:44.594691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110084+0000","last_change":"2023-11-28T18:23:08.127816+0000","last_active":"2023-11-28T18:23:09.110084+0000","last_peered":"2023-11-28T18:23:09.110084+0000","last_clean":"2023-11-28T18:23:09.110084+0000","last_became_active":"2023-11-28T18:23:08.127380+0000","last_became_peered":"2023-11-28T18:23:08.127380+0000","last_unstale":"2023-11-28T18:23:09.110084+0000","last_undegraded":"2023-11-28T18:23:09.110084+0000","last_fullsized":"2023-11-28T18:23:09.110084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:31.242361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105706+0000","last_change":"2023-11-28T18:23:08.128188+0000","last_active":"2023-11-28T18:23:09.105706+0000","last_peered":"2023-11-28T18:23:09.105706+0000","last_clean":"2023-11-28T18:23:09.105706+0000","last_became_active":"2023-11-28T18:23:08.127860+0000","last_became_peered":"2023-11-28T18:23:08.127860+0000","last_unstale":"2023-11-28T18:23:09.105706+0000","last_undegraded":"2023-11-28T18:23:09.105706+0000","last_fullsized":"2023-11-28T18:23:09.105706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:12.095768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565712+0000","last_change":"2023-11-28T18:23:08.132381+0000","last_active":"2023-11-28T18:23:09.565712+0000","last_peered":"2023-11-28T18:23:09.565712+0000","last_clean":"2023-11-28T18:23:09.565712+0000","last_became_active":"2023-11-28T18:23:08.132072+0000","last_became_peered":"2023-11-28T18:23:08.132072+0000","last_unstale":"2023-11-28T18:23:09.565712+0000","last_undegraded":"2023-11-28T18:23:09.565712+0000","last_fullsized":"2023-11-28T18:23:09.565712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:16:38.666702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914263+0000","last_change":"2023-11-28T18:23:08.115379+0000","last_active":"2023-11-28T18:23:10.914263+0000","last_peered":"2023-11-28T18:23:10.914263+0000","last_clean":"2023-11-28T18:23:10.914263+0000","last_became_active":"2023-11-28T18:23:08.115236+0000","last_became_peered":"2023-11-28T18:23:08.115236+0000","last_unstale":"2023-11-28T18:23:10.914263+0000","last_undegraded":"2023-11-28T18:23:10.914263+0000","last_fullsized":"2023-11-28T18:23:10.914263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:26:52.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113293+0000","last_change":"2023-11-28T18:23:08.121402+0000","last_active":"2023-11-28T18:23:09.113293+0000","last_peered":"2023-11-28T18:23:09.113293+0000","last_clean":"2023-11-28T18:23:09.113293+0000","last_became_active":"2023-11-28T18:23:08.121129+0000","last_became_peered":"2023-11-28T18:23:08.121129+0000","last_unstale":"2023-11-28T18:23:09.113293+0000","last_undegraded":"2023-11-28T18:23:09.113293+0000","last_fullsized":"2023-11-28T18:23:09.113293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:02.927891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271829+0000","last_change":"2023-11-28T18:23:08.121594+0000","last_active":"2023-11-28T18:23:10.271829+0000","last_peered":"2023-11-28T18:23:10.271829+0000","last_clean":"2023-11-28T18:23:10.271829+0000","last_became_active":"2023-11-28T18:23:08.119009+0000","last_became_peered":"2023-11-28T18:23:08.119009+0000","last_unstale":"2023-11-28T18:23:10.271829+0000","last_undegraded":"2023-11-28T18:23:10.271829+0000","last_fullsized":"2023-11-28T18:23:10.271829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:38.801632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109676+0000","last_change":"2023-11-28T18:23:08.115403+0000","last_active":"2023-11-28T18:23:09.109676+0000","last_peered":"2023-11-28T18:23:09.109676+0000","last_clean":"2023-11-28T18:23:09.109676+0000","last_became_active":"2023-11-28T18:23:08.114955+0000","last_became_peered":"2023-11-28T18:23:08.114955+0000","last_unstale":"2023-11-28T18:23:09.109676+0000","last_undegraded":"2023-11-28T18:23:09.109676+0000","last_fullsized":"2023-11-28T18:23:09.109676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:41.242830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637404+0000","last_change":"2023-11-28T18:23:08.129745+0000","last_active":"2023-11-28T18:23:09.637404+0000","last_peered":"2023-11-28T18:23:09.637404+0000","last_clean":"2023-11-28T18:23:09.637404+0000","last_became_active":"2023-11-28T18:23:08.129388+0000","last_became_peered":"2023-11-28T18:23:08.129388+0000","last_unstale":"2023-11-28T18:23:09.637404+0000","last_undegraded":"2023-11-28T18:23:09.637404+0000","last_fullsized":"2023-11-28T18:23:09.637404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:50.570821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565681+0000","last_change":"2023-11-28T18:23:07.664773+0000","last_active":"2023-11-28T18:23:09.565681+0000","last_peered":"2023-11-28T18:23:09.565681+0000","last_clean":"2023-11-28T18:23:09.565681+0000","last_became_active":"2023-11-28T18:23:07.664478+0000","last_became_peered":"2023-11-28T18:23:07.664478+0000","last_unstale":"2023-11-28T18:23:09.565681+0000","last_undegraded":"2023-11-28T18:23:09.565681+0000","last_fullsized":"2023-11-28T18:23:09.565681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:28.322454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914337+0000","last_change":"2023-11-28T18:23:08.129441+0000","last_active":"2023-11-28T18:23:10.914337+0000","last_peered":"2023-11-28T18:23:10.914337+0000","last_clean":"2023-11-28T18:23:10.914337+0000","last_became_active":"2023-11-28T18:23:08.129143+0000","last_became_peered":"2023-11-28T18:23:08.129143+0000","last_unstale":"2023-11-28T18:23:10.914337+0000","last_undegraded":"2023-11-28T18:23:10.914337+0000","last_fullsized":"2023-11-28T18:23:10.914337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:36.540819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.479325+0000","last_change":"2023-11-28T18:23:07.668192+0000","last_active":"2023-11-28T18:23:09.479325+0000","last_peered":"2023-11-28T18:23:09.479325+0000","last_clean":"2023-11-28T18:23:09.479325+0000","last_became_active":"2023-11-28T18:23:07.667780+0000","last_became_peered":"2023-11-28T18:23:07.667780+0000","last_unstale":"2023-11-28T18:23:09.479325+0000","last_undegraded":"2023-11-28T18:23:09.479325+0000","last_fullsized":"2023-11-28T18:23:09.479325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.491973+0000","last_change":"2023-11-28T18:23:07.665608+0000","last_active":"2023-11-28T18:23:09.491973+0000","last_peered":"2023-11-28T18:23:09.491973+0000","last_clean":"2023-11-28T18:23:09.491973+0000","last_became_active":"2023-11-28T18:23:07.665361+0000","last_became_peered":"2023-11-28T18:23:07.665361+0000","last_unstale":"2023-11-28T18:23:09.491973+0000","last_undegraded":"2023-11-28T18:23:09.491973+0000","last_fullsized":"2023-11-28T18:23:09.491973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:17.279597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.118091+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.117524+0000","last_became_peered":"2023-11-28T18:23:08.117524+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:04.054226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113169+0000","last_change":"2023-11-28T18:23:07.667847+0000","last_active":"2023-11-28T18:23:09.113169+0000","last_peered":"2023-11-28T18:23:09.113169+0000","last_clean":"2023-11-28T18:23:09.113169+0000","last_became_active":"2023-11-28T18:23:07.667466+0000","last_became_peered":"2023-11-28T18:23:07.667466+0000","last_unstale":"2023-11-28T18:23:09.113169+0000","last_undegraded":"2023-11-28T18:23:09.113169+0000","last_fullsized":"2023-11-28T18:23:09.113169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:06.807146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.127497+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.127216+0000","last_became_peered":"2023-11-28T18:23:08.127216+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:03:14.999484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109883+0000","last_change":"2023-11-28T18:23:07.681698+0000","last_active":"2023-11-28T18:23:09.109883+0000","last_peered":"2023-11-28T18:23:09.109883+0000","last_clean":"2023-11-28T18:23:09.109883+0000","last_became_active":"2023-11-28T18:23:07.681365+0000","last_became_peered":"2023-11-28T18:23:07.681365+0000","last_unstale":"2023-11-28T18:23:09.109883+0000","last_undegraded":"2023-11-28T18:23:09.109883+0000","last_fullsized":"2023-11-28T18:23:09.109883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:21.985029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113361+0000","last_change":"2023-11-28T18:23:08.132797+0000","last_active":"2023-11-28T18:23:09.113361+0000","last_peered":"2023-11-28T18:23:09.113361+0000","last_clean":"2023-11-28T18:23:09.113361+0000","last_became_active":"2023-11-28T18:23:08.132525+0000","last_became_peered":"2023-11-28T18:23:08.132525+0000","last_unstale":"2023-11-28T18:23:09.113361+0000","last_undegraded":"2023-11-28T18:23:09.113361+0000","last_fullsized":"2023-11-28T18:23:09.113361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:12:10.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113468+0000","last_change":"2023-11-28T18:23:07.672337+0000","last_active":"2023-11-28T18:23:09.113468+0000","last_peered":"2023-11-28T18:23:09.113468+0000","last_clean":"2023-11-28T18:23:09.113468+0000","last_became_active":"2023-11-28T18:23:07.671922+0000","last_became_peered":"2023-11-28T18:23:07.671922+0000","last_unstale":"2023-11-28T18:23:09.113468+0000","last_undegraded":"2023-11-28T18:23:09.113468+0000","last_fullsized":"2023-11-28T18:23:09.113468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:22.936770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914223+0000","last_change":"2023-11-28T18:23:08.127919+0000","last_active":"2023-11-28T18:23:10.914223+0000","last_peered":"2023-11-28T18:23:10.914223+0000","last_clean":"2023-11-28T18:23:10.914223+0000","last_became_active":"2023-11-28T18:23:08.127602+0000","last_became_peered":"2023-11-28T18:23:08.127602+0000","last_unstale":"2023-11-28T18:23:10.914223+0000","last_undegraded":"2023-11-28T18:23:10.914223+0000","last_fullsized":"2023-11-28T18:23:10.914223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:32:45.967018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.669406+0000","last_change":"2023-11-28T18:23:07.679219+0000","last_active":"2023-11-28T18:23:13.669406+0000","last_peered":"2023-11-28T18:23:13.669406+0000","last_clean":"2023-11-28T18:23:13.669406+0000","last_became_active":"2023-11-28T18:23:07.678920+0000","last_became_peered":"2023-11-28T18:23:07.678920+0000","last_unstale":"2023-11-28T18:23:13.669406+0000","last_undegraded":"2023-11-28T18:23:13.669406+0000","last_fullsized":"2023-11-28T18:23:13.669406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:02.332690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637820+0000","last_change":"2023-11-28T18:23:08.132429+0000","last_active":"2023-11-28T18:23:09.637820+0000","last_peered":"2023-11-28T18:23:09.637820+0000","last_clean":"2023-11-28T18:23:09.637820+0000","last_became_active":"2023-11-28T18:23:08.132106+0000","last_became_peered":"2023-11-28T18:23:08.132106+0000","last_unstale":"2023-11-28T18:23:09.637820+0000","last_undegraded":"2023-11-28T18:23:09.637820+0000","last_fullsized":"2023-11-28T18:23:09.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:52.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.674719+0000","last_change":"2023-11-28T18:23:07.667479+0000","last_active":"2023-11-28T18:23:13.674719+0000","last_peered":"2023-11-28T18:23:13.674719+0000","last_clean":"2023-11-28T18:23:13.674719+0000","last_became_active":"2023-11-28T18:23:07.665310+0000","last_became_peered":"2023-11-28T18:23:07.665310+0000","last_unstale":"2023-11-28T18:23:13.674719+0000","last_undegraded":"2023-11-28T18:23:13.674719+0000","last_fullsized":"2023-11-28T18:23:13.674719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:03.174425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000928+0000","last_change":"2023-11-28T18:23:08.136812+0000","last_active":"2023-11-28T18:23:11.000928+0000","last_peered":"2023-11-28T18:23:11.000928+0000","last_clean":"2023-11-28T18:23:11.000928+0000","last_became_active":"2023-11-28T18:23:08.136525+0000","last_became_peered":"2023-11-28T18:23:08.136525+0000","last_unstale":"2023-11-28T18:23:11.000928+0000","last_undegraded":"2023-11-28T18:23:11.000928+0000","last_fullsized":"2023-11-28T18:23:11.000928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:07.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.672020+0000","last_change":"2023-11-28T18:23:07.660519+0000","last_active":"2023-11-28T18:23:13.672020+0000","last_peered":"2023-11-28T18:23:13.672020+0000","last_clean":"2023-11-28T18:23:13.672020+0000","last_became_active":"2023-11-28T18:23:07.660205+0000","last_became_peered":"2023-11-28T18:23:07.660205+0000","last_unstale":"2023-11-28T18:23:13.672020+0000","last_undegraded":"2023-11-28T18:23:13.672020+0000","last_fullsized":"2023-11-28T18:23:13.672020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:04:54.125737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914264+0000","last_change":"2023-11-28T18:23:08.127921+0000","last_active":"2023-11-28T18:23:10.914264+0000","last_peered":"2023-11-28T18:23:10.914264+0000","last_clean":"2023-11-28T18:23:10.914264+0000","last_became_active":"2023-11-28T18:23:08.127678+0000","last_became_peered":"2023-11-28T18:23:08.127678+0000","last_unstale":"2023-11-28T18:23:10.914264+0000","last_undegraded":"2023-11-28T18:23:10.914264+0000","last_fullsized":"2023-11-28T18:23:10.914264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:38:47.175911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113417+0000","last_change":"2023-11-28T18:23:07.671589+0000","last_active":"2023-11-28T18:23:09.113417+0000","last_peered":"2023-11-28T18:23:09.113417+0000","last_clean":"2023-11-28T18:23:09.113417+0000","last_became_active":"2023-11-28T18:23:07.671291+0000","last_became_peered":"2023-11-28T18:23:07.671291+0000","last_unstale":"2023-11-28T18:23:09.113417+0000","last_undegraded":"2023-11-28T18:23:09.113417+0000","last_fullsized":"2023-11-28T18:23:09.113417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:23.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000866+0000","last_change":"2023-11-28T18:23:08.141316+0000","last_active":"2023-11-28T18:23:11.000866+0000","last_peered":"2023-11-28T18:23:11.000866+0000","last_clean":"2023-11-28T18:23:11.000866+0000","last_became_active":"2023-11-28T18:23:08.141011+0000","last_became_peered":"2023-11-28T18:23:08.141011+0000","last_unstale":"2023-11-28T18:23:11.000866+0000","last_undegraded":"2023-11-28T18:23:11.000866+0000","last_fullsized":"2023-11-28T18:23:11.000866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:13:25.849340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109447+0000","last_change":"2023-11-28T18:23:07.671192+0000","last_active":"2023-11-28T18:23:09.109447+0000","last_peered":"2023-11-28T18:23:09.109447+0000","last_clean":"2023-11-28T18:23:09.109447+0000","last_became_active":"2023-11-28T18:23:07.670812+0000","last_became_peered":"2023-11-28T18:23:07.670812+0000","last_unstale":"2023-11-28T18:23:09.109447+0000","last_undegraded":"2023-11-28T18:23:09.109447+0000","last_fullsized":"2023-11-28T18:23:09.109447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:28.806061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109506+0000","last_change":"2023-11-28T18:23:08.115229+0000","last_active":"2023-11-28T18:23:09.109506+0000","last_peered":"2023-11-28T18:23:09.109506+0000","last_clean":"2023-11-28T18:23:09.109506+0000","last_became_active":"2023-11-28T18:23:08.114541+0000","last_became_peered":"2023-11-28T18:23:08.114541+0000","last_unstale":"2023-11-28T18:23:09.109506+0000","last_undegraded":"2023-11-28T18:23:09.109506+0000","last_fullsized":"2023-11-28T18:23:09.109506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:38.816731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.673263+0000","last_change":"2023-11-28T18:23:07.673554+0000","last_active":"2023-11-28T18:23:13.673263+0000","last_peered":"2023-11-28T18:23:13.673263+0000","last_clean":"2023-11-28T18:23:13.673263+0000","last_became_active":"2023-11-28T18:23:07.672923+0000","last_became_peered":"2023-11-28T18:23:07.672923+0000","last_unstale":"2023-11-28T18:23:13.673263+0000","last_undegraded":"2023-11-28T18:23:13.673263+0000","last_fullsized":"2023-11-28T18:23:13.673263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:49.805572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000699+0000","last_change":"2023-11-28T18:23:08.141169+0000","last_active":"2023-11-28T18:23:11.000699+0000","last_peered":"2023-11-28T18:23:11.000699+0000","last_clean":"2023-11-28T18:23:11.000699+0000","last_became_active":"2023-11-28T18:23:08.140801+0000","last_became_peered":"2023-11-28T18:23:08.140801+0000","last_unstale":"2023-11-28T18:23:11.000699+0000","last_undegraded":"2023-11-28T18:23:11.000699+0000","last_fullsized":"2023-11-28T18:23:11.000699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:16.669717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637810+0000","last_change":"2023-11-28T18:23:07.680094+0000","last_active":"2023-11-28T18:23:09.637810+0000","last_peered":"2023-11-28T18:23:09.637810+0000","last_clean":"2023-11-28T18:23:09.637810+0000","last_became_active":"2023-11-28T18:23:07.679801+0000","last_became_peered":"2023-11-28T18:23:07.679801+0000","last_unstale":"2023-11-28T18:23:09.637810+0000","last_undegraded":"2023-11-28T18:23:09.637810+0000","last_fullsized":"2023-11-28T18:23:09.637810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:02.916155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001208+0000","last_change":"2023-11-28T18:23:08.137825+0000","last_active":"2023-11-28T18:23:11.001208+0000","last_peered":"2023-11-28T18:23:11.001208+0000","last_clean":"2023-11-28T18:23:11.001208+0000","last_became_active":"2023-11-28T18:23:08.137499+0000","last_became_peered":"2023-11-28T18:23:08.137499+0000","last_unstale":"2023-11-28T18:23:11.001208+0000","last_undegraded":"2023-11-28T18:23:11.001208+0000","last_fullsized":"2023-11-28T18:23:11.001208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:42.807830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.670328+0000","last_change":"2023-11-28T18:23:07.670359+0000","last_active":"2023-11-28T18:23:13.670328+0000","last_peered":"2023-11-28T18:23:13.670328+0000","last_clean":"2023-11-28T18:23:13.670328+0000","last_became_active":"2023-11-28T18:23:07.669687+0000","last_became_peered":"2023-11-28T18:23:07.669687+0000","last_unstale":"2023-11-28T18:23:13.670328+0000","last_undegraded":"2023-11-28T18:23:13.670328+0000","last_fullsized":"2023-11-28T18:23:13.670328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:45.450623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001353+0000","last_change":"2023-11-28T18:23:08.141300+0000","last_active":"2023-11-28T18:23:11.001353+0000","last_peered":"2023-11-28T18:23:11.001353+0000","last_clean":"2023-11-28T18:23:11.001353+0000","last_became_active":"2023-11-28T18:23:08.141037+0000","last_became_peered":"2023-11-28T18:23:08.141037+0000","last_unstale":"2023-11-28T18:23:11.001353+0000","last_undegraded":"2023-11-28T18:23:11.001353+0000","last_fullsized":"2023-11-28T18:23:11.001353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:11.093903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.493244+0000","last_change":"2023-11-28T18:23:07.673086+0000","last_active":"2023-11-28T18:23:09.493244+0000","last_peered":"2023-11-28T18:23:09.493244+0000","last_clean":"2023-11-28T18:23:09.493244+0000","last_became_active":"2023-11-28T18:23:07.672773+0000","last_became_peered":"2023-11-28T18:23:07.672773+0000","last_unstale":"2023-11-28T18:23:09.493244+0000","last_undegraded":"2023-11-28T18:23:09.493244+0000","last_fullsized":"2023-11-28T18:23:09.493244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:34.962428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113586+0000","last_change":"2023-11-28T18:23:08.121987+0000","last_active":"2023-11-28T18:23:09.113586+0000","last_peered":"2023-11-28T18:23:09.113586+0000","last_clean":"2023-11-28T18:23:09.113586+0000","last_became_active":"2023-11-28T18:23:08.121279+0000","last_became_peered":"2023-11-28T18:23:08.121279+0000","last_unstale":"2023-11-28T18:23:09.113586+0000","last_undegraded":"2023-11-28T18:23:09.113586+0000","last_fullsized":"2023-11-28T18:23:09.113586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:47.973973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.480344+0000","last_change":"2023-11-28T18:23:07.681312+0000","last_active":"2023-11-28T18:23:09.480344+0000","last_peered":"2023-11-28T18:23:09.480344+0000","last_clean":"2023-11-28T18:23:09.480344+0000","last_became_active":"2023-11-28T18:23:07.680806+0000","last_became_peered":"2023-11-28T18:23:07.680806+0000","last_unstale":"2023-11-28T18:23:09.480344+0000","last_undegraded":"2023-11-28T18:23:09.480344+0000","last_fullsized":"2023-11-28T18:23:09.480344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:58:41.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001014+0000","last_change":"2023-11-28T18:23:08.137874+0000","last_active":"2023-11-28T18:23:11.001014+0000","last_peered":"2023-11-28T18:23:11.001014+0000","last_clean":"2023-11-28T18:23:11.001014+0000","last_became_active":"2023-11-28T18:23:08.137104+0000","last_became_peered":"2023-11-28T18:23:08.137104+0000","last_unstale":"2023-11-28T18:23:11.001014+0000","last_undegraded":"2023-11-28T18:23:11.001014+0000","last_fullsized":"2023-11-28T18:23:11.001014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:39:11.857836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117053+0000","last_change":"2023-11-28T18:22:56.774057+0000","last_active":"2023-11-28T18:23:08.117053+0000","last_peered":"2023-11-28T18:23:08.117053+0000","last_clean":"2023-11-28T18:23:08.117053+0000","last_became_active":"2023-11-28T18:22:56.772763+0000","last_became_peered":"2023-11-28T18:22:56.772763+0000","last_unstale":"2023-11-28T18:23:08.117053+0000","last_undegraded":"2023-11-28T18:23:08.117053+0000","last_fullsized":"2023-11-28T18:23:08.117053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_clean_scrub_stamp":"2023-11-28T18:22:55.591288+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:05.470408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113059+0000","last_change":"2023-11-28T18:23:07.674072+0000","last_active":"2023-11-28T18:23:09.113059+0000","last_peered":"2023-11-28T18:23:09.113059+0000","last_clean":"2023-11-28T18:23:09.113059+0000","last_became_active":"2023-11-28T18:23:07.673220+0000","last_became_peered":"2023-11-28T18:23:07.673220+0000","last_unstale":"2023-11-28T18:23:09.113059+0000","last_undegraded":"2023-11-28T18:23:09.113059+0000","last_fullsized":"2023-11-28T18:23:09.113059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:58.524112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565598+0000","last_change":"2023-11-28T18:23:08.130989+0000","last_active":"2023-11-28T18:23:09.565598+0000","last_peered":"2023-11-28T18:23:09.565598+0000","last_clean":"2023-11-28T18:23:09.565598+0000","last_became_active":"2023-11-28T18:23:08.130646+0000","last_became_peered":"2023-11-28T18:23:08.130646+0000","last_unstale":"2023-11-28T18:23:09.565598+0000","last_undegraded":"2023-11-28T18:23:09.565598+0000","last_fullsized":"2023-11-28T18:23:09.565598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:50.371045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.482690+0000","last_change":"2023-11-28T18:23:07.686046+0000","last_active":"2023-11-28T18:23:09.482690+0000","last_peered":"2023-11-28T18:23:09.482690+0000","last_clean":"2023-11-28T18:23:09.482690+0000","last_became_active":"2023-11-28T18:23:07.685739+0000","last_became_peered":"2023-11-28T18:23:07.685739+0000","last_unstale":"2023-11-28T18:23:09.482690+0000","last_undegraded":"2023-11-28T18:23:09.482690+0000","last_fullsized":"2023-11-28T18:23:09.482690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:06.485631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913951+0000","last_change":"2023-11-28T18:23:08.130355+0000","last_active":"2023-11-28T18:23:10.913951+0000","last_peered":"2023-11-28T18:23:10.913951+0000","last_clean":"2023-11-28T18:23:10.913951+0000","last_became_active":"2023-11-28T18:23:08.129946+0000","last_became_peered":"2023-11-28T18:23:08.129946+0000","last_unstale":"2023-11-28T18:23:10.913951+0000","last_undegraded":"2023-11-28T18:23:10.913951+0000","last_fullsized":"2023-11-28T18:23:10.913951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:58.140191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.480326+0000","last_change":"2023-11-28T18:23:07.664827+0000","last_active":"2023-11-28T18:23:09.480326+0000","last_peered":"2023-11-28T18:23:09.480326+0000","last_clean":"2023-11-28T18:23:09.480326+0000","last_became_active":"2023-11-28T18:23:07.664503+0000","last_became_peered":"2023-11-28T18:23:07.664503+0000","last_unstale":"2023-11-28T18:23:09.480326+0000","last_undegraded":"2023-11-28T18:23:09.480326+0000","last_fullsized":"2023-11-28T18:23:09.480326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.238694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.123498+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.123197+0000","last_became_peered":"2023-11-28T18:23:08.123197+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:08:42.817848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113555+0000","last_change":"2023-11-28T18:23:07.673345+0000","last_active":"2023-11-28T18:23:09.113555+0000","last_peered":"2023-11-28T18:23:09.113555+0000","last_clean":"2023-11-28T18:23:09.113555+0000","last_became_active":"2023-11-28T18:23:07.672982+0000","last_became_peered":"2023-11-28T18:23:07.672982+0000","last_unstale":"2023-11-28T18:23:09.113555+0000","last_undegraded":"2023-11-28T18:23:09.113555+0000","last_fullsized":"2023-11-28T18:23:09.113555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:26.594232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637566+0000","last_change":"2023-11-28T18:23:08.129091+0000","last_active":"2023-11-28T18:23:09.637566+0000","last_peered":"2023-11-28T18:23:09.637566+0000","last_clean":"2023-11-28T18:23:09.637566+0000","last_became_active":"2023-11-28T18:23:08.128589+0000","last_became_peered":"2023-11-28T18:23:08.128589+0000","last_unstale":"2023-11-28T18:23:09.637566+0000","last_undegraded":"2023-11-28T18:23:09.637566+0000","last_fullsized":"2023-11-28T18:23:09.637566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:28.975412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105665+0000","last_change":"2023-11-28T18:23:07.671874+0000","last_active":"2023-11-28T18:23:09.105665+0000","last_peered":"2023-11-28T18:23:09.105665+0000","last_clean":"2023-11-28T18:23:09.105665+0000","last_became_active":"2023-11-28T18:23:07.671500+0000","last_became_peered":"2023-11-28T18:23:07.671500+0000","last_unstale":"2023-11-28T18:23:09.105665+0000","last_undegraded":"2023-11-28T18:23:09.105665+0000","last_fullsized":"2023-11-28T18:23:09.105665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:42.922770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113612+0000","last_change":"2023-11-28T18:23:08.127419+0000","last_active":"2023-11-28T18:23:09.113612+0000","last_peered":"2023-11-28T18:23:09.113612+0000","last_clean":"2023-11-28T18:23:09.113612+0000","last_became_active":"2023-11-28T18:23:08.127157+0000","last_became_peered":"2023-11-28T18:23:08.127157+0000","last_unstale":"2023-11-28T18:23:09.113612+0000","last_undegraded":"2023-11-28T18:23:09.113612+0000","last_fullsized":"2023-11-28T18:23:09.113612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:08.473882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.482879+0000","last_change":"2023-11-28T18:23:07.684725+0000","last_active":"2023-11-28T18:23:09.482879+0000","last_peered":"2023-11-28T18:23:09.482879+0000","last_clean":"2023-11-28T18:23:09.482879+0000","last_became_active":"2023-11-28T18:23:07.684363+0000","last_became_peered":"2023-11-28T18:23:07.684363+0000","last_unstale":"2023-11-28T18:23:09.482879+0000","last_undegraded":"2023-11-28T18:23:09.482879+0000","last_fullsized":"2023-11-28T18:23:09.482879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:16.380789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105690+0000","last_change":"2023-11-28T18:23:08.120741+0000","last_active":"2023-11-28T18:23:09.105690+0000","last_peered":"2023-11-28T18:23:09.105690+0000","last_clean":"2023-11-28T18:23:09.105690+0000","last_became_active":"2023-11-28T18:23:08.120421+0000","last_became_peered":"2023-11-28T18:23:08.120421+0000","last_unstale":"2023-11-28T18:23:09.105690+0000","last_undegraded":"2023-11-28T18:23:09.105690+0000","last_fullsized":"2023-11-28T18:23:09.105690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:53:27.131631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105889+0000","last_change":"2023-11-28T18:23:07.672505+0000","last_active":"2023-11-28T18:23:09.105889+0000","last_peered":"2023-11-28T18:23:09.105889+0000","last_clean":"2023-11-28T18:23:09.105889+0000","last_became_active":"2023-11-28T18:23:07.672204+0000","last_became_peered":"2023-11-28T18:23:07.672204+0000","last_unstale":"2023-11-28T18:23:09.105889+0000","last_undegraded":"2023-11-28T18:23:09.105889+0000","last_fullsized":"2023-11-28T18:23:09.105889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:01.341643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565869+0000","last_change":"2023-11-28T18:23:08.126198+0000","last_active":"2023-11-28T18:23:09.565869+0000","last_peered":"2023-11-28T18:23:09.565869+0000","last_clean":"2023-11-28T18:23:09.565869+0000","last_became_active":"2023-11-28T18:23:08.125935+0000","last_became_peered":"2023-11-28T18:23:08.125935+0000","last_unstale":"2023-11-28T18:23:09.565869+0000","last_undegraded":"2023-11-28T18:23:09.565869+0000","last_fullsized":"2023-11-28T18:23:09.565869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.473922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.674971+0000","last_change":"2023-11-28T18:23:07.672170+0000","last_active":"2023-11-28T18:23:13.674971+0000","last_peered":"2023-11-28T18:23:13.674971+0000","last_clean":"2023-11-28T18:23:13.674971+0000","last_became_active":"2023-11-28T18:23:07.671683+0000","last_became_peered":"2023-11-28T18:23:07.671683+0000","last_unstale":"2023-11-28T18:23:13.674971+0000","last_undegraded":"2023-11-28T18:23:13.674971+0000","last_fullsized":"2023-11-28T18:23:13.674971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:41.860341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914424+0000","last_change":"2023-11-28T18:23:08.119844+0000","last_active":"2023-11-28T18:23:10.914424+0000","last_peered":"2023-11-28T18:23:10.914424+0000","last_clean":"2023-11-28T18:23:10.914424+0000","last_became_active":"2023-11-28T18:23:08.119525+0000","last_became_peered":"2023-11-28T18:23:08.119525+0000","last_unstale":"2023-11-28T18:23:10.914424+0000","last_undegraded":"2023-11-28T18:23:10.914424+0000","last_fullsized":"2023-11-28T18:23:10.914424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:34:00.230528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.482666+0000","last_change":"2023-11-28T18:23:07.685651+0000","last_active":"2023-11-28T18:23:09.482666+0000","last_peered":"2023-11-28T18:23:09.482666+0000","last_clean":"2023-11-28T18:23:09.482666+0000","last_became_active":"2023-11-28T18:23:07.683869+0000","last_became_peered":"2023-11-28T18:23:07.683869+0000","last_unstale":"2023-11-28T18:23:09.482666+0000","last_undegraded":"2023-11-28T18:23:09.482666+0000","last_fullsized":"2023-11-28T18:23:09.482666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:35.383852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637574+0000","last_change":"2023-11-28T18:23:08.130016+0000","last_active":"2023-11-28T18:23:09.637574+0000","last_peered":"2023-11-28T18:23:09.637574+0000","last_clean":"2023-11-28T18:23:09.637574+0000","last_became_active":"2023-11-28T18:23:08.129586+0000","last_became_peered":"2023-11-28T18:23:08.129586+0000","last_unstale":"2023-11-28T18:23:09.637574+0000","last_undegraded":"2023-11-28T18:23:09.637574+0000","last_fullsized":"2023-11-28T18:23:09.637574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:00.449941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105620+0000","last_change":"2023-11-28T18:23:07.671773+0000","last_active":"2023-11-28T18:23:09.105620+0000","last_peered":"2023-11-28T18:23:09.105620+0000","last_clean":"2023-11-28T18:23:09.105620+0000","last_became_active":"2023-11-28T18:23:07.671465+0000","last_became_peered":"2023-11-28T18:23:07.671465+0000","last_unstale":"2023-11-28T18:23:09.105620+0000","last_undegraded":"2023-11-28T18:23:09.105620+0000","last_fullsized":"2023-11-28T18:23:09.105620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:44.862780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565865+0000","last_change":"2023-11-28T18:23:08.118146+0000","last_active":"2023-11-28T18:23:09.565865+0000","last_peered":"2023-11-28T18:23:09.565865+0000","last_clean":"2023-11-28T18:23:09.565865+0000","last_became_active":"2023-11-28T18:23:08.117848+0000","last_became_peered":"2023-11-28T18:23:08.117848+0000","last_unstale":"2023-11-28T18:23:09.565865+0000","last_undegraded":"2023-11-28T18:23:09.565865+0000","last_fullsized":"2023-11-28T18:23:09.565865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:33:20.745184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105627+0000","last_change":"2023-11-28T18:23:07.676709+0000","last_active":"2023-11-28T18:23:09.105627+0000","last_peered":"2023-11-28T18:23:09.105627+0000","last_clean":"2023-11-28T18:23:09.105627+0000","last_became_active":"2023-11-28T18:23:07.676414+0000","last_became_peered":"2023-11-28T18:23:07.676414+0000","last_unstale":"2023-11-28T18:23:09.105627+0000","last_undegraded":"2023-11-28T18:23:09.105627+0000","last_fullsized":"2023-11-28T18:23:09.105627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.282005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565753+0000","last_change":"2023-11-28T18:23:08.112410+0000","last_active":"2023-11-28T18:23:09.565753+0000","last_peered":"2023-11-28T18:23:09.565753+0000","last_clean":"2023-11-28T18:23:09.565753+0000","last_became_active":"2023-11-28T18:23:08.112097+0000","last_became_peered":"2023-11-28T18:23:08.112097+0000","last_unstale":"2023-11-28T18:23:09.565753+0000","last_undegraded":"2023-11-28T18:23:09.565753+0000","last_fullsized":"2023-11-28T18:23:09.565753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:57.891789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.668542+0000","last_change":"2023-11-28T18:23:07.684818+0000","last_active":"2023-11-28T18:23:13.668542+0000","last_peered":"2023-11-28T18:23:13.668542+0000","last_clean":"2023-11-28T18:23:13.668542+0000","last_became_active":"2023-11-28T18:23:07.684502+0000","last_became_peered":"2023-11-28T18:23:07.684502+0000","last_unstale":"2023-11-28T18:23:13.668542+0000","last_undegraded":"2023-11-28T18:23:13.668542+0000","last_fullsized":"2023-11-28T18:23:13.668542+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:16:21.223266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109527+0000","last_change":"2023-11-28T18:23:08.133972+0000","last_active":"2023-11-28T18:23:09.109527+0000","last_peered":"2023-11-28T18:23:09.109527+0000","last_clean":"2023-11-28T18:23:09.109527+0000","last_became_active":"2023-11-28T18:23:08.133747+0000","last_became_peered":"2023-11-28T18:23:08.133747+0000","last_unstale":"2023-11-28T18:23:09.109527+0000","last_undegraded":"2023-11-28T18:23:09.109527+0000","last_fullsized":"2023-11-28T18:23:09.109527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:24.968241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113538+0000","last_change":"2023-11-28T18:23:08.134117+0000","last_active":"2023-11-28T18:23:09.113538+0000","last_peered":"2023-11-28T18:23:09.113538+0000","last_clean":"2023-11-28T18:23:09.113538+0000","last_became_active":"2023-11-28T18:23:08.133801+0000","last_became_peered":"2023-11-28T18:23:08.133801+0000","last_unstale":"2023-11-28T18:23:09.113538+0000","last_undegraded":"2023-11-28T18:23:09.113538+0000","last_fullsized":"2023-11-28T18:23:09.113538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:02.370862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.669428+0000","last_change":"2023-11-28T18:23:07.686399+0000","last_active":"2023-11-28T18:23:13.669428+0000","last_peered":"2023-11-28T18:23:13.669428+0000","last_clean":"2023-11-28T18:23:13.669428+0000","last_became_active":"2023-11-28T18:23:07.686051+0000","last_became_peered":"2023-11-28T18:23:07.686051+0000","last_unstale":"2023-11-28T18:23:13.669428+0000","last_undegraded":"2023-11-28T18:23:13.669428+0000","last_fullsized":"2023-11-28T18:23:13.669428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:11.797516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113317+0000","last_change":"2023-11-28T18:23:08.134870+0000","last_active":"2023-11-28T18:23:09.113317+0000","last_peered":"2023-11-28T18:23:09.113317+0000","last_clean":"2023-11-28T18:23:09.113317+0000","last_became_active":"2023-11-28T18:23:08.133732+0000","last_became_peered":"2023-11-28T18:23:08.133732+0000","last_unstale":"2023-11-28T18:23:09.113317+0000","last_undegraded":"2023-11-28T18:23:09.113317+0000","last_fullsized":"2023-11-28T18:23:09.113317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:01.364242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.668135+0000","last_change":"2023-11-28T18:23:07.682230+0000","last_active":"2023-11-28T18:23:13.668135+0000","last_peered":"2023-11-28T18:23:13.668135+0000","last_clean":"2023-11-28T18:23:13.668135+0000","last_became_active":"2023-11-28T18:23:07.681715+0000","last_became_peered":"2023-11-28T18:23:07.681715+0000","last_unstale":"2023-11-28T18:23:13.668135+0000","last_undegraded":"2023-11-28T18:23:13.668135+0000","last_fullsized":"2023-11-28T18:23:13.668135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:02.343996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914216+0000","last_change":"2023-11-28T18:23:08.128927+0000","last_active":"2023-11-28T18:23:10.914216+0000","last_peered":"2023-11-28T18:23:10.914216+0000","last_clean":"2023-11-28T18:23:10.914216+0000","last_became_active":"2023-11-28T18:23:08.127775+0000","last_became_peered":"2023-11-28T18:23:08.127775+0000","last_unstale":"2023-11-28T18:23:10.914216+0000","last_undegraded":"2023-11-28T18:23:10.914216+0000","last_fullsized":"2023-11-28T18:23:10.914216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:50:05.531589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565730+0000","last_change":"2023-11-28T18:23:07.667929+0000","last_active":"2023-11-28T18:23:09.565730+0000","last_peered":"2023-11-28T18:23:09.565730+0000","last_clean":"2023-11-28T18:23:09.565730+0000","last_became_active":"2023-11-28T18:23:07.667666+0000","last_became_peered":"2023-11-28T18:23:07.667666+0000","last_unstale":"2023-11-28T18:23:09.565730+0000","last_undegraded":"2023-11-28T18:23:09.565730+0000","last_fullsized":"2023-11-28T18:23:09.565730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:32.132092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565609+0000","last_change":"2023-11-28T18:23:08.126404+0000","last_active":"2023-11-28T18:23:09.565609+0000","last_peered":"2023-11-28T18:23:09.565609+0000","last_clean":"2023-11-28T18:23:09.565609+0000","last_became_active":"2023-11-28T18:23:08.125315+0000","last_became_peered":"2023-11-28T18:23:08.125315+0000","last_unstale":"2023-11-28T18:23:09.565609+0000","last_undegraded":"2023-11-28T18:23:09.565609+0000","last_fullsized":"2023-11-28T18:23:09.565609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:24:59.527642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.674794+0000","last_change":"2023-11-28T18:23:07.677758+0000","last_active":"2023-11-28T18:23:13.674794+0000","last_peered":"2023-11-28T18:23:13.674794+0000","last_clean":"2023-11-28T18:23:13.674794+0000","last_became_active":"2023-11-28T18:23:07.676902+0000","last_became_peered":"2023-11-28T18:23:07.676902+0000","last_unstale":"2023-11-28T18:23:13.674794+0000","last_undegraded":"2023-11-28T18:23:13.674794+0000","last_fullsized":"2023-11-28T18:23:13.674794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:50.953647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T18:23:19.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T18:23:19.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b80fe640 1 Processor -- start 2023-11-28T18:23:19.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b80fe640 1 -- start start 2023-11-28T18:23:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b80fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c8090 0x7f01b01575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b80fe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b00c6140 con 0x7f01b00c24f0 2023-11-28T18:23:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b80fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0157b00 con 0x7f01b0150b30 2023-11-28T18:23:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b80fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b00c72b0 con 0x7f01b00c8090 2023-11-28T18:23:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b5699640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f01b00c24f0 0x7f01b00c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55308/0 (socket says 172.21.15.7:55308) 2023-11-28T18:23:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b5699640 1 -- 172.21.15.7:0/4031337195 learned_addr learned my addr 172.21.15.7:0/4031337195 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b669b640 1 --2- 172.21.15.7:0/4031337195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c8090 0x7f01b01575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.708+0000 7f01b4e98640 1 -- 172.21.15.7:0/4031337195 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95065613 0 0) 0x7f01b00c6140 con 0x7f01b00c24f0 2023-11-28T18:23:19.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b4e98640 1 -- 172.21.15.7:0/4031337195 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f019c003530 con 0x7f01b00c24f0 2023-11-28T18:23:19.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b4e98640 1 -- 172.21.15.7:0/4031337195 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320032756 0 0) 0x7f01b0157b00 con 0x7f01b0150b30 2023-11-28T18:23:19.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b4e98640 1 -- 172.21.15.7:0/4031337195 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b00c6140 con 0x7f01b0150b30 2023-11-28T18:23:19.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b4e98640 1 -- 172.21.15.7:0/4031337195 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725563700 0 0) 0x7f019c003530 con 0x7f01b00c24f0 2023-11-28T18:23:19.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b669b640 1 -- 172.21.15.7:0/4031337195 >> v1:172.21.15.193:6789/0 conn(0x7f01b0150b30 legacy=0x7f01b00c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b669b640 1 -- 172.21.15.7:0/4031337195 >> v1:172.21.15.7:6789/0 conn(0x7f01b00c24f0 legacy=0x7f01b00c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b669b640 1 -- 172.21.15.7:0/4031337195 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b00c2960 con 0x7f01b00c8090 2023-11-28T18:23:19.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b669b640 1 --2- 172.21.15.7:0/4031337195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c8090 0x7f01b01575c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009dc0 tx=0x7f01a801f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54231c06497e88b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:19.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b4e98640 1 -- 172.21.15.7:0/4031337195 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0001680 con 0x7f01b00c8090 2023-11-28T18:23:19.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b4e98640 1 -- 172.21.15.7:0/4031337195 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a80a8c10 con 0x7f01b00c8090 2023-11-28T18:23:19.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.712+0000 7f01b4e98640 1 -- 172.21.15.7:0/4031337195 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a80b0440 con 0x7f01b00c8090 2023-11-28T18:23:19.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- 172.21.15.7:0/4031337195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c8090 msgr2=0x7f01b01575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 --2- 172.21.15.7:0/4031337195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c8090 0x7f01b01575c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009dc0 tx=0x7f01a801f080 comp rx=0 tx=0).stop 2023-11-28T18:23:19.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- 172.21.15.7:0/4031337195 shutdown_connections 2023-11-28T18:23:19.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 --2- 172.21.15.7:0/4031337195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c8090 0x7f01b01575c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- 172.21.15.7:0/4031337195 >> 172.21.15.7:0/4031337195 conn(0x7f01b00662e0 msgr2=0x7f01b00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- 172.21.15.7:0/4031337195 shutdown_connections 2023-11-28T18:23:19.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- 172.21.15.7:0/4031337195 wait complete. 2023-11-28T18:23:19.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 Processor -- start 2023-11-28T18:23:19.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- start start 2023-11-28T18:23:19.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c24f0 0x7f01b00714e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00c8090 0x7f01b0155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01b0150b30 0x7f01b013c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0140f20 con 0x7f01b00c8090 2023-11-28T18:23:19.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b01454c0 con 0x7f01b0150b30 2023-11-28T18:23:19.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b80fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0155810 con 0x7f01b00c24f0 2023-11-28T18:23:19.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00c8090 0x7f01b0155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00c8090 0x7f01b0155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59776/0 (socket says 172.21.15.7:59776) 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 -- 172.21.15.7:0/128750059 learned_addr learned my addr 172.21.15.7:0/128750059 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b669b640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01b0150b30 0x7f01b013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 -- 172.21.15.7:0/128750059 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c24f0 msgr2=0x7f01b00714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c24f0 0x7f01b00714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 -- 172.21.15.7:0/128750059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01b0150b30 msgr2=0x7f01b013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01b0150b30 0x7f01b013c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 -- 172.21.15.7:0/128750059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8009a70 con 0x7f01b00c8090 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b669b640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01b0150b30 0x7f01b013c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.716+0000 7f01b5699640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00c8090 0x7f01b0155d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f01a000c750 tx=0x7f01a000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01a67fc640 1 -- 172.21.15.7:0/128750059 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0005ae0 con 0x7f01b00c8090 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01a67fc640 1 -- 172.21.15.7:0/128750059 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a0005c40 con 0x7f01b00c8090 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b0072e90 con 0x7f01b00c8090 2023-11-28T18:23:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01a67fc640 1 -- 172.21.15.7:0/128750059 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0007070 con 0x7f01b00c8090 2023-11-28T18:23:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b0140b40 con 0x7f01b00c8090 2023-11-28T18:23:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b004e890 con 0x7f01b00c8090 2023-11-28T18:23:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01a67fc640 1 -- 172.21.15.7:0/128750059 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f01a00071d0 con 0x7f01b00c8090 2023-11-28T18:23:19.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01a67fc640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01940668d0 0x7f0194068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:19.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01a67fc640 1 -- 172.21.15.7:0/128750059 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01a0084290 con 0x7f01b00c8090 2023-11-28T18:23:19.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01b5e9a640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01940668d0 0x7f0194068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:19.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.720+0000 7f01b5e9a640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01940668d0 0x7f0194068d90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01ac007fd0 tx=0x7f01ac007960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:19.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.724+0000 7f01a67fc640 1 -- 172.21.15.7:0/128750059 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01a0051430 con 0x7f01b00c8090 2023-11-28T18:23:19.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.880+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 --> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01b0074330 con 0x7f01940668d0 2023-11-28T18:23:19.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.884+0000 7f01a67fc640 1 -- 172.21.15.7:0/128750059 <== mgr.4102 v2:172.21.15.7:6832/264069339 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264558 (secure 0 0 0) 0x7f01b0074330 con 0x7f01940668d0 2023-11-28T18:23:19.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:19.894 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-28T18:23:19.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01940668d0 msgr2=0x7f0194068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01940668d0 0x7f0194068d90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01ac007fd0 tx=0x7f01ac007960 comp rx=0 tx=0).stop 2023-11-28T18:23:19.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00c8090 msgr2=0x7f01b0155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:19.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00c8090 0x7f01b0155d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f01a000c750 tx=0x7f01a000cc10 comp rx=0 tx=0).stop 2023-11-28T18:23:19.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 shutdown_connections 2023-11-28T18:23:19.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01940668d0 0x7f0194068d90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01b0150b30 0x7f01b013c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00c8090 0x7f01b0155d90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 --2- 172.21.15.7:0/128750059 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01b00c24f0 0x7f01b00714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:19.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 >> 172.21.15.7:0/128750059 conn(0x7f01b00662e0 msgr2=0x7f01b00c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:19.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 shutdown_connections 2023-11-28T18:23:19.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:19.888+0000 7f01b80fe640 1 -- 172.21.15.7:0/128750059 wait complete. 2023-11-28T18:23:19.913 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-11-28T18:23:19.463797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235592,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739128,"statfs":{"total":773094113280,"available":772852867072,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.360594"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001273+0000","last_change":"2023-11-28T18:23:08.134633+0000","last_active":"2023-11-28T18:23:11.001273+0000","last_peered":"2023-11-28T18:23:11.001273+0000","last_clean":"2023-11-28T18:23:11.001273+0000","last_became_active":"2023-11-28T18:23:08.134320+0000","last_became_peered":"2023-11-28T18:23:08.134320+0000","last_unstale":"2023-11-28T18:23:11.001273+0000","last_undegraded":"2023-11-28T18:23:11.001273+0000","last_fullsized":"2023-11-28T18:23:11.001273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:54.995718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105857+0000","last_change":"2023-11-28T18:23:08.122460+0000","last_active":"2023-11-28T18:23:09.105857+0000","last_peered":"2023-11-28T18:23:09.105857+0000","last_clean":"2023-11-28T18:23:09.105857+0000","last_became_active":"2023-11-28T18:23:08.122153+0000","last_became_peered":"2023-11-28T18:23:08.122153+0000","last_unstale":"2023-11-28T18:23:09.105857+0000","last_undegraded":"2023-11-28T18:23:09.105857+0000","last_fullsized":"2023-11-28T18:23:09.105857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:17:31.074978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914207+0000","last_change":"2023-11-28T18:23:08.110187+0000","last_active":"2023-11-28T18:23:10.914207+0000","last_peered":"2023-11-28T18:23:10.914207+0000","last_clean":"2023-11-28T18:23:10.914207+0000","last_became_active":"2023-11-28T18:23:08.109915+0000","last_became_peered":"2023-11-28T18:23:08.109915+0000","last_unstale":"2023-11-28T18:23:10.914207+0000","last_undegraded":"2023-11-28T18:23:10.914207+0000","last_fullsized":"2023-11-28T18:23:10.914207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:32.098408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113335+0000","last_change":"2023-11-28T18:23:08.122000+0000","last_active":"2023-11-28T18:23:09.113335+0000","last_peered":"2023-11-28T18:23:09.113335+0000","last_clean":"2023-11-28T18:23:09.113335+0000","last_became_active":"2023-11-28T18:23:08.121652+0000","last_became_peered":"2023-11-28T18:23:08.121652+0000","last_unstale":"2023-11-28T18:23:09.113335+0000","last_undegraded":"2023-11-28T18:23:09.113335+0000","last_fullsized":"2023-11-28T18:23:09.113335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:15.540606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000810+0000","last_change":"2023-11-28T18:23:08.126587+0000","last_active":"2023-11-28T18:23:11.000810+0000","last_peered":"2023-11-28T18:23:11.000810+0000","last_clean":"2023-11-28T18:23:11.000810+0000","last_became_active":"2023-11-28T18:23:08.126313+0000","last_became_peered":"2023-11-28T18:23:08.126313+0000","last_unstale":"2023-11-28T18:23:11.000810+0000","last_undegraded":"2023-11-28T18:23:11.000810+0000","last_fullsized":"2023-11-28T18:23:11.000810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.956197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113040+0000","last_change":"2023-11-28T18:23:08.122878+0000","last_active":"2023-11-28T18:23:09.113040+0000","last_peered":"2023-11-28T18:23:09.113040+0000","last_clean":"2023-11-28T18:23:09.113040+0000","last_became_active":"2023-11-28T18:23:08.122609+0000","last_became_peered":"2023-11-28T18:23:08.122609+0000","last_unstale":"2023-11-28T18:23:09.113040+0000","last_undegraded":"2023-11-28T18:23:09.113040+0000","last_fullsized":"2023-11-28T18:23:09.113040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:00.610074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000604+0000","last_change":"2023-11-28T18:23:08.133485+0000","last_active":"2023-11-28T18:23:11.000604+0000","last_peered":"2023-11-28T18:23:11.000604+0000","last_clean":"2023-11-28T18:23:11.000604+0000","last_became_active":"2023-11-28T18:23:08.133211+0000","last_became_peered":"2023-11-28T18:23:08.133211+0000","last_unstale":"2023-11-28T18:23:11.000604+0000","last_undegraded":"2023-11-28T18:23:11.000604+0000","last_fullsized":"2023-11-28T18:23:11.000604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:04:33.507685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565355+0000","last_change":"2023-11-28T18:23:08.117903+0000","last_active":"2023-11-28T18:23:09.565355+0000","last_peered":"2023-11-28T18:23:09.565355+0000","last_clean":"2023-11-28T18:23:09.565355+0000","last_became_active":"2023-11-28T18:23:08.117482+0000","last_became_peered":"2023-11-28T18:23:08.117482+0000","last_unstale":"2023-11-28T18:23:09.565355+0000","last_undegraded":"2023-11-28T18:23:09.565355+0000","last_fullsized":"2023-11-28T18:23:09.565355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:45.293402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001351+0000","last_change":"2023-11-28T18:23:08.135606+0000","last_active":"2023-11-28T18:23:11.001351+0000","last_peered":"2023-11-28T18:23:11.001351+0000","last_clean":"2023-11-28T18:23:11.001351+0000","last_became_active":"2023-11-28T18:23:08.135328+0000","last_became_peered":"2023-11-28T18:23:08.135328+0000","last_unstale":"2023-11-28T18:23:11.001351+0000","last_undegraded":"2023-11-28T18:23:11.001351+0000","last_fullsized":"2023-11-28T18:23:11.001351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:25.011869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110046+0000","last_change":"2023-11-28T18:23:08.123993+0000","last_active":"2023-11-28T18:23:09.110046+0000","last_peered":"2023-11-28T18:23:09.110046+0000","last_clean":"2023-11-28T18:23:09.110046+0000","last_became_active":"2023-11-28T18:23:08.123583+0000","last_became_peered":"2023-11-28T18:23:08.123583+0000","last_unstale":"2023-11-28T18:23:09.110046+0000","last_undegraded":"2023-11-28T18:23:09.110046+0000","last_fullsized":"2023-11-28T18:23:09.110046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:16:33.089307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:08.120983+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:08.120486+0000","last_became_peered":"2023-11-28T18:23:08.120486+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:46:12.881629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637634+0000","last_change":"2023-11-28T18:23:08.130169+0000","last_active":"2023-11-28T18:23:09.637634+0000","last_peered":"2023-11-28T18:23:09.637634+0000","last_clean":"2023-11-28T18:23:09.637634+0000","last_became_active":"2023-11-28T18:23:08.129771+0000","last_became_peered":"2023-11-28T18:23:08.129771+0000","last_unstale":"2023-11-28T18:23:09.637634+0000","last_undegraded":"2023-11-28T18:23:09.637634+0000","last_fullsized":"2023-11-28T18:23:09.637634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:45.753322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565505+0000","last_change":"2023-11-28T18:23:08.123268+0000","last_active":"2023-11-28T18:23:09.565505+0000","last_peered":"2023-11-28T18:23:09.565505+0000","last_clean":"2023-11-28T18:23:09.565505+0000","last_became_active":"2023-11-28T18:23:08.122978+0000","last_became_peered":"2023-11-28T18:23:08.122978+0000","last_unstale":"2023-11-28T18:23:09.565505+0000","last_undegraded":"2023-11-28T18:23:09.565505+0000","last_fullsized":"2023-11-28T18:23:09.565505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:15:01.320918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913943+0000","last_change":"2023-11-28T18:23:08.118361+0000","last_active":"2023-11-28T18:23:10.913943+0000","last_peered":"2023-11-28T18:23:10.913943+0000","last_clean":"2023-11-28T18:23:10.913943+0000","last_became_active":"2023-11-28T18:23:08.117973+0000","last_became_peered":"2023-11-28T18:23:08.117973+0000","last_unstale":"2023-11-28T18:23:10.913943+0000","last_undegraded":"2023-11-28T18:23:10.913943+0000","last_fullsized":"2023-11-28T18:23:10.913943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:57.485937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105530+0000","last_change":"2023-11-28T18:23:08.139744+0000","last_active":"2023-11-28T18:23:09.105530+0000","last_peered":"2023-11-28T18:23:09.105530+0000","last_clean":"2023-11-28T18:23:09.105530+0000","last_became_active":"2023-11-28T18:23:08.139464+0000","last_became_peered":"2023-11-28T18:23:08.139464+0000","last_unstale":"2023-11-28T18:23:09.105530+0000","last_undegraded":"2023-11-28T18:23:09.105530+0000","last_fullsized":"2023-11-28T18:23:09.105530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:19:34.780979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113036+0000","last_change":"2023-11-28T18:23:08.119103+0000","last_active":"2023-11-28T18:23:09.113036+0000","last_peered":"2023-11-28T18:23:09.113036+0000","last_clean":"2023-11-28T18:23:09.113036+0000","last_became_active":"2023-11-28T18:23:08.116080+0000","last_became_peered":"2023-11-28T18:23:08.116080+0000","last_unstale":"2023-11-28T18:23:09.113036+0000","last_undegraded":"2023-11-28T18:23:09.113036+0000","last_fullsized":"2023-11-28T18:23:09.113036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:44.279413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113688+0000","last_change":"2023-11-28T18:23:08.126817+0000","last_active":"2023-11-28T18:23:09.113688+0000","last_peered":"2023-11-28T18:23:09.113688+0000","last_clean":"2023-11-28T18:23:09.113688+0000","last_became_active":"2023-11-28T18:23:08.126540+0000","last_became_peered":"2023-11-28T18:23:08.126540+0000","last_unstale":"2023-11-28T18:23:09.113688+0000","last_undegraded":"2023-11-28T18:23:09.113688+0000","last_fullsized":"2023-11-28T18:23:09.113688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:08.807847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271940+0000","last_change":"2023-11-28T18:23:08.121775+0000","last_active":"2023-11-28T18:23:10.271940+0000","last_peered":"2023-11-28T18:23:10.271940+0000","last_clean":"2023-11-28T18:23:10.271940+0000","last_became_active":"2023-11-28T18:23:08.121466+0000","last_became_peered":"2023-11-28T18:23:08.121466+0000","last_unstale":"2023-11-28T18:23:10.271940+0000","last_undegraded":"2023-11-28T18:23:10.271940+0000","last_fullsized":"2023-11-28T18:23:10.271940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:32:51.612113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001115+0000","last_change":"2023-11-28T18:23:08.138282+0000","last_active":"2023-11-28T18:23:11.001115+0000","last_peered":"2023-11-28T18:23:11.001115+0000","last_clean":"2023-11-28T18:23:11.001115+0000","last_became_active":"2023-11-28T18:23:08.137171+0000","last_became_peered":"2023-11-28T18:23:08.137171+0000","last_unstale":"2023-11-28T18:23:11.001115+0000","last_undegraded":"2023-11-28T18:23:11.001115+0000","last_fullsized":"2023-11-28T18:23:11.001115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:39:07.113697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110124+0000","last_change":"2023-11-28T18:23:08.133521+0000","last_active":"2023-11-28T18:23:09.110124+0000","last_peered":"2023-11-28T18:23:09.110124+0000","last_clean":"2023-11-28T18:23:09.110124+0000","last_became_active":"2023-11-28T18:23:08.133248+0000","last_became_peered":"2023-11-28T18:23:08.133248+0000","last_unstale":"2023-11-28T18:23:09.110124+0000","last_undegraded":"2023-11-28T18:23:09.110124+0000","last_fullsized":"2023-11-28T18:23:09.110124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:17:34.585808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914195+0000","last_change":"2023-11-28T18:23:08.129150+0000","last_active":"2023-11-28T18:23:10.914195+0000","last_peered":"2023-11-28T18:23:10.914195+0000","last_clean":"2023-11-28T18:23:10.914195+0000","last_became_active":"2023-11-28T18:23:08.128622+0000","last_became_peered":"2023-11-28T18:23:08.128622+0000","last_unstale":"2023-11-28T18:23:10.914195+0000","last_undegraded":"2023-11-28T18:23:10.914195+0000","last_fullsized":"2023-11-28T18:23:10.914195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:33.197161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113274+0000","last_change":"2023-11-28T18:23:08.131333+0000","last_active":"2023-11-28T18:23:09.113274+0000","last_peered":"2023-11-28T18:23:09.113274+0000","last_clean":"2023-11-28T18:23:09.113274+0000","last_became_active":"2023-11-28T18:23:08.130917+0000","last_became_peered":"2023-11-28T18:23:08.130917+0000","last_unstale":"2023-11-28T18:23:09.113274+0000","last_undegraded":"2023-11-28T18:23:09.113274+0000","last_fullsized":"2023-11-28T18:23:09.113274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:04:48.114417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565507+0000","last_change":"2023-11-28T18:23:08.132355+0000","last_active":"2023-11-28T18:23:09.565507+0000","last_peered":"2023-11-28T18:23:09.565507+0000","last_clean":"2023-11-28T18:23:09.565507+0000","last_became_active":"2023-11-28T18:23:08.132016+0000","last_became_peered":"2023-11-28T18:23:08.132016+0000","last_unstale":"2023-11-28T18:23:09.565507+0000","last_undegraded":"2023-11-28T18:23:09.565507+0000","last_fullsized":"2023-11-28T18:23:09.565507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:11.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113066+0000","last_change":"2023-11-28T18:23:08.119935+0000","last_active":"2023-11-28T18:23:09.113066+0000","last_peered":"2023-11-28T18:23:09.113066+0000","last_clean":"2023-11-28T18:23:09.113066+0000","last_became_active":"2023-11-28T18:23:08.118372+0000","last_became_peered":"2023-11-28T18:23:08.118372+0000","last_unstale":"2023-11-28T18:23:09.113066+0000","last_undegraded":"2023-11-28T18:23:09.113066+0000","last_fullsized":"2023-11-28T18:23:09.113066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:52.339024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.132272+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.131985+0000","last_became_peered":"2023-11-28T18:23:08.131985+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:44.594691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110084+0000","last_change":"2023-11-28T18:23:08.127816+0000","last_active":"2023-11-28T18:23:09.110084+0000","last_peered":"2023-11-28T18:23:09.110084+0000","last_clean":"2023-11-28T18:23:09.110084+0000","last_became_active":"2023-11-28T18:23:08.127380+0000","last_became_peered":"2023-11-28T18:23:08.127380+0000","last_unstale":"2023-11-28T18:23:09.110084+0000","last_undegraded":"2023-11-28T18:23:09.110084+0000","last_fullsized":"2023-11-28T18:23:09.110084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:31.242361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105706+0000","last_change":"2023-11-28T18:23:08.128188+0000","last_active":"2023-11-28T18:23:09.105706+0000","last_peered":"2023-11-28T18:23:09.105706+0000","last_clean":"2023-11-28T18:23:09.105706+0000","last_became_active":"2023-11-28T18:23:08.127860+0000","last_became_peered":"2023-11-28T18:23:08.127860+0000","last_unstale":"2023-11-28T18:23:09.105706+0000","last_undegraded":"2023-11-28T18:23:09.105706+0000","last_fullsized":"2023-11-28T18:23:09.105706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:12.095768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565712+0000","last_change":"2023-11-28T18:23:08.132381+0000","last_active":"2023-11-28T18:23:09.565712+0000","last_peered":"2023-11-28T18:23:09.565712+0000","last_clean":"2023-11-28T18:23:09.565712+0000","last_became_active":"2023-11-28T18:23:08.132072+0000","last_became_peered":"2023-11-28T18:23:08.132072+0000","last_unstale":"2023-11-28T18:23:09.565712+0000","last_undegraded":"2023-11-28T18:23:09.565712+0000","last_fullsized":"2023-11-28T18:23:09.565712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:16:38.666702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914263+0000","last_change":"2023-11-28T18:23:08.115379+0000","last_active":"2023-11-28T18:23:10.914263+0000","last_peered":"2023-11-28T18:23:10.914263+0000","last_clean":"2023-11-28T18:23:10.914263+0000","last_became_active":"2023-11-28T18:23:08.115236+0000","last_became_peered":"2023-11-28T18:23:08.115236+0000","last_unstale":"2023-11-28T18:23:10.914263+0000","last_undegraded":"2023-11-28T18:23:10.914263+0000","last_fullsized":"2023-11-28T18:23:10.914263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:26:52.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113293+0000","last_change":"2023-11-28T18:23:08.121402+0000","last_active":"2023-11-28T18:23:09.113293+0000","last_peered":"2023-11-28T18:23:09.113293+0000","last_clean":"2023-11-28T18:23:09.113293+0000","last_became_active":"2023-11-28T18:23:08.121129+0000","last_became_peered":"2023-11-28T18:23:08.121129+0000","last_unstale":"2023-11-28T18:23:09.113293+0000","last_undegraded":"2023-11-28T18:23:09.113293+0000","last_fullsized":"2023-11-28T18:23:09.113293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:02.927891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271829+0000","last_change":"2023-11-28T18:23:08.121594+0000","last_active":"2023-11-28T18:23:10.271829+0000","last_peered":"2023-11-28T18:23:10.271829+0000","last_clean":"2023-11-28T18:23:10.271829+0000","last_became_active":"2023-11-28T18:23:08.119009+0000","last_became_peered":"2023-11-28T18:23:08.119009+0000","last_unstale":"2023-11-28T18:23:10.271829+0000","last_undegraded":"2023-11-28T18:23:10.271829+0000","last_fullsized":"2023-11-28T18:23:10.271829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:38.801632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109676+0000","last_change":"2023-11-28T18:23:08.115403+0000","last_active":"2023-11-28T18:23:09.109676+0000","last_peered":"2023-11-28T18:23:09.109676+0000","last_clean":"2023-11-28T18:23:09.109676+0000","last_became_active":"2023-11-28T18:23:08.114955+0000","last_became_peered":"2023-11-28T18:23:08.114955+0000","last_unstale":"2023-11-28T18:23:09.109676+0000","last_undegraded":"2023-11-28T18:23:09.109676+0000","last_fullsized":"2023-11-28T18:23:09.109676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:41.242830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637404+0000","last_change":"2023-11-28T18:23:08.129745+0000","last_active":"2023-11-28T18:23:09.637404+0000","last_peered":"2023-11-28T18:23:09.637404+0000","last_clean":"2023-11-28T18:23:09.637404+0000","last_became_active":"2023-11-28T18:23:08.129388+0000","last_became_peered":"2023-11-28T18:23:08.129388+0000","last_unstale":"2023-11-28T18:23:09.637404+0000","last_undegraded":"2023-11-28T18:23:09.637404+0000","last_fullsized":"2023-11-28T18:23:09.637404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:50.570821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565681+0000","last_change":"2023-11-28T18:23:07.664773+0000","last_active":"2023-11-28T18:23:09.565681+0000","last_peered":"2023-11-28T18:23:09.565681+0000","last_clean":"2023-11-28T18:23:09.565681+0000","last_became_active":"2023-11-28T18:23:07.664478+0000","last_became_peered":"2023-11-28T18:23:07.664478+0000","last_unstale":"2023-11-28T18:23:09.565681+0000","last_undegraded":"2023-11-28T18:23:09.565681+0000","last_fullsized":"2023-11-28T18:23:09.565681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:28.322454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914337+0000","last_change":"2023-11-28T18:23:08.129441+0000","last_active":"2023-11-28T18:23:10.914337+0000","last_peered":"2023-11-28T18:23:10.914337+0000","last_clean":"2023-11-28T18:23:10.914337+0000","last_became_active":"2023-11-28T18:23:08.129143+0000","last_became_peered":"2023-11-28T18:23:08.129143+0000","last_unstale":"2023-11-28T18:23:10.914337+0000","last_undegraded":"2023-11-28T18:23:10.914337+0000","last_fullsized":"2023-11-28T18:23:10.914337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:36.540819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.479325+0000","last_change":"2023-11-28T18:23:07.668192+0000","last_active":"2023-11-28T18:23:09.479325+0000","last_peered":"2023-11-28T18:23:09.479325+0000","last_clean":"2023-11-28T18:23:09.479325+0000","last_became_active":"2023-11-28T18:23:07.667780+0000","last_became_peered":"2023-11-28T18:23:07.667780+0000","last_unstale":"2023-11-28T18:23:09.479325+0000","last_undegraded":"2023-11-28T18:23:09.479325+0000","last_fullsized":"2023-11-28T18:23:09.479325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.491973+0000","last_change":"2023-11-28T18:23:07.665608+0000","last_active":"2023-11-28T18:23:09.491973+0000","last_peered":"2023-11-28T18:23:09.491973+0000","last_clean":"2023-11-28T18:23:09.491973+0000","last_became_active":"2023-11-28T18:23:07.665361+0000","last_became_peered":"2023-11-28T18:23:07.665361+0000","last_unstale":"2023-11-28T18:23:09.491973+0000","last_undegraded":"2023-11-28T18:23:09.491973+0000","last_fullsized":"2023-11-28T18:23:09.491973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:17.279597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.118091+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.117524+0000","last_became_peered":"2023-11-28T18:23:08.117524+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:04.054226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113169+0000","last_change":"2023-11-28T18:23:07.667847+0000","last_active":"2023-11-28T18:23:09.113169+0000","last_peered":"2023-11-28T18:23:09.113169+0000","last_clean":"2023-11-28T18:23:09.113169+0000","last_became_active":"2023-11-28T18:23:07.667466+0000","last_became_peered":"2023-11-28T18:23:07.667466+0000","last_unstale":"2023-11-28T18:23:09.113169+0000","last_undegraded":"2023-11-28T18:23:09.113169+0000","last_fullsized":"2023-11-28T18:23:09.113169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:06.807146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.127497+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.127216+0000","last_became_peered":"2023-11-28T18:23:08.127216+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:03:14.999484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109883+0000","last_change":"2023-11-28T18:23:07.681698+0000","last_active":"2023-11-28T18:23:09.109883+0000","last_peered":"2023-11-28T18:23:09.109883+0000","last_clean":"2023-11-28T18:23:09.109883+0000","last_became_active":"2023-11-28T18:23:07.681365+0000","last_became_peered":"2023-11-28T18:23:07.681365+0000","last_unstale":"2023-11-28T18:23:09.109883+0000","last_undegraded":"2023-11-28T18:23:09.109883+0000","last_fullsized":"2023-11-28T18:23:09.109883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:21.985029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113361+0000","last_change":"2023-11-28T18:23:08.132797+0000","last_active":"2023-11-28T18:23:09.113361+0000","last_peered":"2023-11-28T18:23:09.113361+0000","last_clean":"2023-11-28T18:23:09.113361+0000","last_became_active":"2023-11-28T18:23:08.132525+0000","last_became_peered":"2023-11-28T18:23:08.132525+0000","last_unstale":"2023-11-28T18:23:09.113361+0000","last_undegraded":"2023-11-28T18:23:09.113361+0000","last_fullsized":"2023-11-28T18:23:09.113361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:12:10.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113468+0000","last_change":"2023-11-28T18:23:07.672337+0000","last_active":"2023-11-28T18:23:09.113468+0000","last_peered":"2023-11-28T18:23:09.113468+0000","last_clean":"2023-11-28T18:23:09.113468+0000","last_became_active":"2023-11-28T18:23:07.671922+0000","last_became_peered":"2023-11-28T18:23:07.671922+0000","last_unstale":"2023-11-28T18:23:09.113468+0000","last_undegraded":"2023-11-28T18:23:09.113468+0000","last_fullsized":"2023-11-28T18:23:09.113468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:22.936770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914223+0000","last_change":"2023-11-28T18:23:08.127919+0000","last_active":"2023-11-28T18:23:10.914223+0000","last_peered":"2023-11-28T18:23:10.914223+0000","last_clean":"2023-11-28T18:23:10.914223+0000","last_became_active":"2023-11-28T18:23:08.127602+0000","last_became_peered":"2023-11-28T18:23:08.127602+0000","last_unstale":"2023-11-28T18:23:10.914223+0000","last_undegraded":"2023-11-28T18:23:10.914223+0000","last_fullsized":"2023-11-28T18:23:10.914223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:32:45.967018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.669406+0000","last_change":"2023-11-28T18:23:07.679219+0000","last_active":"2023-11-28T18:23:13.669406+0000","last_peered":"2023-11-28T18:23:13.669406+0000","last_clean":"2023-11-28T18:23:13.669406+0000","last_became_active":"2023-11-28T18:23:07.678920+0000","last_became_peered":"2023-11-28T18:23:07.678920+0000","last_unstale":"2023-11-28T18:23:13.669406+0000","last_undegraded":"2023-11-28T18:23:13.669406+0000","last_fullsized":"2023-11-28T18:23:13.669406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:02.332690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637820+0000","last_change":"2023-11-28T18:23:08.132429+0000","last_active":"2023-11-28T18:23:09.637820+0000","last_peered":"2023-11-28T18:23:09.637820+0000","last_clean":"2023-11-28T18:23:09.637820+0000","last_became_active":"2023-11-28T18:23:08.132106+0000","last_became_peered":"2023-11-28T18:23:08.132106+0000","last_unstale":"2023-11-28T18:23:09.637820+0000","last_undegraded":"2023-11-28T18:23:09.637820+0000","last_fullsized":"2023-11-28T18:23:09.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:52.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.674719+0000","last_change":"2023-11-28T18:23:07.667479+0000","last_active":"2023-11-28T18:23:13.674719+0000","last_peered":"2023-11-28T18:23:13.674719+0000","last_clean":"2023-11-28T18:23:13.674719+0000","last_became_active":"2023-11-28T18:23:07.665310+0000","last_became_peered":"2023-11-28T18:23:07.665310+0000","last_unstale":"2023-11-28T18:23:13.674719+0000","last_undegraded":"2023-11-28T18:23:13.674719+0000","last_fullsized":"2023-11-28T18:23:13.674719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:03.174425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000928+0000","last_change":"2023-11-28T18:23:08.136812+0000","last_active":"2023-11-28T18:23:11.000928+0000","last_peered":"2023-11-28T18:23:11.000928+0000","last_clean":"2023-11-28T18:23:11.000928+0000","last_became_active":"2023-11-28T18:23:08.136525+0000","last_became_peered":"2023-11-28T18:23:08.136525+0000","last_unstale":"2023-11-28T18:23:11.000928+0000","last_undegraded":"2023-11-28T18:23:11.000928+0000","last_fullsized":"2023-11-28T18:23:11.000928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:07.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.672020+0000","last_change":"2023-11-28T18:23:07.660519+0000","last_active":"2023-11-28T18:23:13.672020+0000","last_peered":"2023-11-28T18:23:13.672020+0000","last_clean":"2023-11-28T18:23:13.672020+0000","last_became_active":"2023-11-28T18:23:07.660205+0000","last_became_peered":"2023-11-28T18:23:07.660205+0000","last_unstale":"2023-11-28T18:23:13.672020+0000","last_undegraded":"2023-11-28T18:23:13.672020+0000","last_fullsized":"2023-11-28T18:23:13.672020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:04:54.125737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914264+0000","last_change":"2023-11-28T18:23:08.127921+0000","last_active":"2023-11-28T18:23:10.914264+0000","last_peered":"2023-11-28T18:23:10.914264+0000","last_clean":"2023-11-28T18:23:10.914264+0000","last_became_active":"2023-11-28T18:23:08.127678+0000","last_became_peered":"2023-11-28T18:23:08.127678+0000","last_unstale":"2023-11-28T18:23:10.914264+0000","last_undegraded":"2023-11-28T18:23:10.914264+0000","last_fullsized":"2023-11-28T18:23:10.914264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:38:47.175911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113417+0000","last_change":"2023-11-28T18:23:07.671589+0000","last_active":"2023-11-28T18:23:09.113417+0000","last_peered":"2023-11-28T18:23:09.113417+0000","last_clean":"2023-11-28T18:23:09.113417+0000","last_became_active":"2023-11-28T18:23:07.671291+0000","last_became_peered":"2023-11-28T18:23:07.671291+0000","last_unstale":"2023-11-28T18:23:09.113417+0000","last_undegraded":"2023-11-28T18:23:09.113417+0000","last_fullsized":"2023-11-28T18:23:09.113417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:23.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000866+0000","last_change":"2023-11-28T18:23:08.141316+0000","last_active":"2023-11-28T18:23:11.000866+0000","last_peered":"2023-11-28T18:23:11.000866+0000","last_clean":"2023-11-28T18:23:11.000866+0000","last_became_active":"2023-11-28T18:23:08.141011+0000","last_became_peered":"2023-11-28T18:23:08.141011+0000","last_unstale":"2023-11-28T18:23:11.000866+0000","last_undegraded":"2023-11-28T18:23:11.000866+0000","last_fullsized":"2023-11-28T18:23:11.000866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:13:25.849340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109447+0000","last_change":"2023-11-28T18:23:07.671192+0000","last_active":"2023-11-28T18:23:09.109447+0000","last_peered":"2023-11-28T18:23:09.109447+0000","last_clean":"2023-11-28T18:23:09.109447+0000","last_became_active":"2023-11-28T18:23:07.670812+0000","last_became_peered":"2023-11-28T18:23:07.670812+0000","last_unstale":"2023-11-28T18:23:09.109447+0000","last_undegraded":"2023-11-28T18:23:09.109447+0000","last_fullsized":"2023-11-28T18:23:09.109447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:28.806061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109506+0000","last_change":"2023-11-28T18:23:08.115229+0000","last_active":"2023-11-28T18:23:09.109506+0000","last_peered":"2023-11-28T18:23:09.109506+0000","last_clean":"2023-11-28T18:23:09.109506+0000","last_became_active":"2023-11-28T18:23:08.114541+0000","last_became_peered":"2023-11-28T18:23:08.114541+0000","last_unstale":"2023-11-28T18:23:09.109506+0000","last_undegraded":"2023-11-28T18:23:09.109506+0000","last_fullsized":"2023-11-28T18:23:09.109506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:38.816731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.673263+0000","last_change":"2023-11-28T18:23:07.673554+0000","last_active":"2023-11-28T18:23:13.673263+0000","last_peered":"2023-11-28T18:23:13.673263+0000","last_clean":"2023-11-28T18:23:13.673263+0000","last_became_active":"2023-11-28T18:23:07.672923+0000","last_became_peered":"2023-11-28T18:23:07.672923+0000","last_unstale":"2023-11-28T18:23:13.673263+0000","last_undegraded":"2023-11-28T18:23:13.673263+0000","last_fullsized":"2023-11-28T18:23:13.673263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:49.805572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000699+0000","last_change":"2023-11-28T18:23:08.141169+0000","last_active":"2023-11-28T18:23:11.000699+0000","last_peered":"2023-11-28T18:23:11.000699+0000","last_clean":"2023-11-28T18:23:11.000699+0000","last_became_active":"2023-11-28T18:23:08.140801+0000","last_became_peered":"2023-11-28T18:23:08.140801+0000","last_unstale":"2023-11-28T18:23:11.000699+0000","last_undegraded":"2023-11-28T18:23:11.000699+0000","last_fullsized":"2023-11-28T18:23:11.000699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:16.669717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637810+0000","last_change":"2023-11-28T18:23:07.680094+0000","last_active":"2023-11-28T18:23:09.637810+0000","last_peered":"2023-11-28T18:23:09.637810+0000","last_clean":"2023-11-28T18:23:09.637810+0000","last_became_active":"2023-11-28T18:23:07.679801+0000","last_became_peered":"2023-11-28T18:23:07.679801+0000","last_unstale":"2023-11-28T18:23:09.637810+0000","last_undegraded":"2023-11-28T18:23:09.637810+0000","last_fullsized":"2023-11-28T18:23:09.637810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:02.916155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001208+0000","last_change":"2023-11-28T18:23:08.137825+0000","last_active":"2023-11-28T18:23:11.001208+0000","last_peered":"2023-11-28T18:23:11.001208+0000","last_clean":"2023-11-28T18:23:11.001208+0000","last_became_active":"2023-11-28T18:23:08.137499+0000","last_became_peered":"2023-11-28T18:23:08.137499+0000","last_unstale":"2023-11-28T18:23:11.001208+0000","last_undegraded":"2023-11-28T18:23:11.001208+0000","last_fullsized":"2023-11-28T18:23:11.001208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:42.807830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.670328+0000","last_change":"2023-11-28T18:23:07.670359+0000","last_active":"2023-11-28T18:23:13.670328+0000","last_peered":"2023-11-28T18:23:13.670328+0000","last_clean":"2023-11-28T18:23:13.670328+0000","last_became_active":"2023-11-28T18:23:07.669687+0000","last_became_peered":"2023-11-28T18:23:07.669687+0000","last_unstale":"2023-11-28T18:23:13.670328+0000","last_undegraded":"2023-11-28T18:23:13.670328+0000","last_fullsized":"2023-11-28T18:23:13.670328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:45.450623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001353+0000","last_change":"2023-11-28T18:23:08.141300+0000","last_active":"2023-11-28T18:23:11.001353+0000","last_peered":"2023-11-28T18:23:11.001353+0000","last_clean":"2023-11-28T18:23:11.001353+0000","last_became_active":"2023-11-28T18:23:08.141037+0000","last_became_peered":"2023-11-28T18:23:08.141037+0000","last_unstale":"2023-11-28T18:23:11.001353+0000","last_undegraded":"2023-11-28T18:23:11.001353+0000","last_fullsized":"2023-11-28T18:23:11.001353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:11.093903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.493244+0000","last_change":"2023-11-28T18:23:07.673086+0000","last_active":"2023-11-28T18:23:09.493244+0000","last_peered":"2023-11-28T18:23:09.493244+0000","last_clean":"2023-11-28T18:23:09.493244+0000","last_became_active":"2023-11-28T18:23:07.672773+0000","last_became_peered":"2023-11-28T18:23:07.672773+0000","last_unstale":"2023-11-28T18:23:09.493244+0000","last_undegraded":"2023-11-28T18:23:09.493244+0000","last_fullsized":"2023-11-28T18:23:09.493244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:34.962428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113586+0000","last_change":"2023-11-28T18:23:08.121987+0000","last_active":"2023-11-28T18:23:09.113586+0000","last_peered":"2023-11-28T18:23:09.113586+0000","last_clean":"2023-11-28T18:23:09.113586+0000","last_became_active":"2023-11-28T18:23:08.121279+0000","last_became_peered":"2023-11-28T18:23:08.121279+0000","last_unstale":"2023-11-28T18:23:09.113586+0000","last_undegraded":"2023-11-28T18:23:09.113586+0000","last_fullsized":"2023-11-28T18:23:09.113586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:47.973973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.480344+0000","last_change":"2023-11-28T18:23:07.681312+0000","last_active":"2023-11-28T18:23:09.480344+0000","last_peered":"2023-11-28T18:23:09.480344+0000","last_clean":"2023-11-28T18:23:09.480344+0000","last_became_active":"2023-11-28T18:23:07.680806+0000","last_became_peered":"2023-11-28T18:23:07.680806+0000","last_unstale":"2023-11-28T18:23:09.480344+0000","last_undegraded":"2023-11-28T18:23:09.480344+0000","last_fullsized":"2023-11-28T18:23:09.480344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:58:41.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001014+0000","last_change":"2023-11-28T18:23:08.137874+0000","last_active":"2023-11-28T18:23:11.001014+0000","last_peered":"2023-11-28T18:23:11.001014+0000","last_clean":"2023-11-28T18:23:11.001014+0000","last_became_active":"2023-11-28T18:23:08.137104+0000","last_became_peered":"2023-11-28T18:23:08.137104+0000","last_unstale":"2023-11-28T18:23:11.001014+0000","last_undegraded":"2023-11-28T18:23:11.001014+0000","last_fullsized":"2023-11-28T18:23:11.001014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:39:11.857836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117053+0000","last_change":"2023-11-28T18:22:56.774057+0000","last_active":"2023-11-28T18:23:08.117053+0000","last_peered":"2023-11-28T18:23:08.117053+0000","last_clean":"2023-11-28T18:23:08.117053+0000","last_became_active":"2023-11-28T18:22:56.772763+0000","last_became_peered":"2023-11-28T18:22:56.772763+0000","last_unstale":"2023-11-28T18:23:08.117053+0000","last_undegraded":"2023-11-28T18:23:08.117053+0000","last_fullsized":"2023-11-28T18:23:08.117053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_clean_scrub_stamp":"2023-11-28T18:22:55.591288+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:05.470408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113059+0000","last_change":"2023-11-28T18:23:07.674072+0000","last_active":"2023-11-28T18:23:09.113059+0000","last_peered":"2023-11-28T18:23:09.113059+0000","last_clean":"2023-11-28T18:23:09.113059+0000","last_became_active":"2023-11-28T18:23:07.673220+0000","last_became_peered":"2023-11-28T18:23:07.673220+0000","last_unstale":"2023-11-28T18:23:09.113059+0000","last_undegraded":"2023-11-28T18:23:09.113059+0000","last_fullsized":"2023-11-28T18:23:09.113059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:58.524112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565598+0000","last_change":"2023-11-28T18:23:08.130989+0000","last_active":"2023-11-28T18:23:09.565598+0000","last_peered":"2023-11-28T18:23:09.565598+0000","last_clean":"2023-11-28T18:23:09.565598+0000","last_became_active":"2023-11-28T18:23:08.130646+0000","last_became_peered":"2023-11-28T18:23:08.130646+0000","last_unstale":"2023-11-28T18:23:09.565598+0000","last_undegraded":"2023-11-28T18:23:09.565598+0000","last_fullsized":"2023-11-28T18:23:09.565598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:50.371045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.482690+0000","last_change":"2023-11-28T18:23:07.686046+0000","last_active":"2023-11-28T18:23:09.482690+0000","last_peered":"2023-11-28T18:23:09.482690+0000","last_clean":"2023-11-28T18:23:09.482690+0000","last_became_active":"2023-11-28T18:23:07.685739+0000","last_became_peered":"2023-11-28T18:23:07.685739+0000","last_unstale":"2023-11-28T18:23:09.482690+0000","last_undegraded":"2023-11-28T18:23:09.482690+0000","last_fullsized":"2023-11-28T18:23:09.482690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:06.485631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913951+0000","last_change":"2023-11-28T18:23:08.130355+0000","last_active":"2023-11-28T18:23:10.913951+0000","last_peered":"2023-11-28T18:23:10.913951+0000","last_clean":"2023-11-28T18:23:10.913951+0000","last_became_active":"2023-11-28T18:23:08.129946+0000","last_became_peered":"2023-11-28T18:23:08.129946+0000","last_unstale":"2023-11-28T18:23:10.913951+0000","last_undegraded":"2023-11-28T18:23:10.913951+0000","last_fullsized":"2023-11-28T18:23:10.913951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:58.140191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.480326+0000","last_change":"2023-11-28T18:23:07.664827+0000","last_active":"2023-11-28T18:23:09.480326+0000","last_peered":"2023-11-28T18:23:09.480326+0000","last_clean":"2023-11-28T18:23:09.480326+0000","last_became_active":"2023-11-28T18:23:07.664503+0000","last_became_peered":"2023-11-28T18:23:07.664503+0000","last_unstale":"2023-11-28T18:23:09.480326+0000","last_undegraded":"2023-11-28T18:23:09.480326+0000","last_fullsized":"2023-11-28T18:23:09.480326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.238694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.123498+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.123197+0000","last_became_peered":"2023-11-28T18:23:08.123197+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:08:42.817848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113555+0000","last_change":"2023-11-28T18:23:07.673345+0000","last_active":"2023-11-28T18:23:09.113555+0000","last_peered":"2023-11-28T18:23:09.113555+0000","last_clean":"2023-11-28T18:23:09.113555+0000","last_became_active":"2023-11-28T18:23:07.672982+0000","last_became_peered":"2023-11-28T18:23:07.672982+0000","last_unstale":"2023-11-28T18:23:09.113555+0000","last_undegraded":"2023-11-28T18:23:09.113555+0000","last_fullsized":"2023-11-28T18:23:09.113555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:26.594232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637566+0000","last_change":"2023-11-28T18:23:08.129091+0000","last_active":"2023-11-28T18:23:09.637566+0000","last_peered":"2023-11-28T18:23:09.637566+0000","last_clean":"2023-11-28T18:23:09.637566+0000","last_became_active":"2023-11-28T18:23:08.128589+0000","last_became_peered":"2023-11-28T18:23:08.128589+0000","last_unstale":"2023-11-28T18:23:09.637566+0000","last_undegraded":"2023-11-28T18:23:09.637566+0000","last_fullsized":"2023-11-28T18:23:09.637566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:28.975412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105665+0000","last_change":"2023-11-28T18:23:07.671874+0000","last_active":"2023-11-28T18:23:09.105665+0000","last_peered":"2023-11-28T18:23:09.105665+0000","last_clean":"2023-11-28T18:23:09.105665+0000","last_became_active":"2023-11-28T18:23:07.671500+0000","last_became_peered":"2023-11-28T18:23:07.671500+0000","last_unstale":"2023-11-28T18:23:09.105665+0000","last_undegraded":"2023-11-28T18:23:09.105665+0000","last_fullsized":"2023-11-28T18:23:09.105665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:42.922770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113612+0000","last_change":"2023-11-28T18:23:08.127419+0000","last_active":"2023-11-28T18:23:09.113612+0000","last_peered":"2023-11-28T18:23:09.113612+0000","last_clean":"2023-11-28T18:23:09.113612+0000","last_became_active":"2023-11-28T18:23:08.127157+0000","last_became_peered":"2023-11-28T18:23:08.127157+0000","last_unstale":"2023-11-28T18:23:09.113612+0000","last_undegraded":"2023-11-28T18:23:09.113612+0000","last_fullsized":"2023-11-28T18:23:09.113612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:08.473882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.482879+0000","last_change":"2023-11-28T18:23:07.684725+0000","last_active":"2023-11-28T18:23:09.482879+0000","last_peered":"2023-11-28T18:23:09.482879+0000","last_clean":"2023-11-28T18:23:09.482879+0000","last_became_active":"2023-11-28T18:23:07.684363+0000","last_became_peered":"2023-11-28T18:23:07.684363+0000","last_unstale":"2023-11-28T18:23:09.482879+0000","last_undegraded":"2023-11-28T18:23:09.482879+0000","last_fullsized":"2023-11-28T18:23:09.482879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:16.380789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105690+0000","last_change":"2023-11-28T18:23:08.120741+0000","last_active":"2023-11-28T18:23:09.105690+0000","last_peered":"2023-11-28T18:23:09.105690+0000","last_clean":"2023-11-28T18:23:09.105690+0000","last_became_active":"2023-11-28T18:23:08.120421+0000","last_became_peered":"2023-11-28T18:23:08.120421+0000","last_unstale":"2023-11-28T18:23:09.105690+0000","last_undegraded":"2023-11-28T18:23:09.105690+0000","last_fullsized":"2023-11-28T18:23:09.105690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:53:27.131631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105889+0000","last_change":"2023-11-28T18:23:07.672505+0000","last_active":"2023-11-28T18:23:09.105889+0000","last_peered":"2023-11-28T18:23:09.105889+0000","last_clean":"2023-11-28T18:23:09.105889+0000","last_became_active":"2023-11-28T18:23:07.672204+0000","last_became_peered":"2023-11-28T18:23:07.672204+0000","last_unstale":"2023-11-28T18:23:09.105889+0000","last_undegraded":"2023-11-28T18:23:09.105889+0000","last_fullsized":"2023-11-28T18:23:09.105889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:01.341643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565869+0000","last_change":"2023-11-28T18:23:08.126198+0000","last_active":"2023-11-28T18:23:09.565869+0000","last_peered":"2023-11-28T18:23:09.565869+0000","last_clean":"2023-11-28T18:23:09.565869+0000","last_became_active":"2023-11-28T18:23:08.125935+0000","last_became_peered":"2023-11-28T18:23:08.125935+0000","last_unstale":"2023-11-28T18:23:09.565869+0000","last_undegraded":"2023-11-28T18:23:09.565869+0000","last_fullsized":"2023-11-28T18:23:09.565869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.473922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.674971+0000","last_change":"2023-11-28T18:23:07.672170+0000","last_active":"2023-11-28T18:23:13.674971+0000","last_peered":"2023-11-28T18:23:13.674971+0000","last_clean":"2023-11-28T18:23:13.674971+0000","last_became_active":"2023-11-28T18:23:07.671683+0000","last_became_peered":"2023-11-28T18:23:07.671683+0000","last_unstale":"2023-11-28T18:23:13.674971+0000","last_undegraded":"2023-11-28T18:23:13.674971+0000","last_fullsized":"2023-11-28T18:23:13.674971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:41.860341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914424+0000","last_change":"2023-11-28T18:23:08.119844+0000","last_active":"2023-11-28T18:23:10.914424+0000","last_peered":"2023-11-28T18:23:10.914424+0000","last_clean":"2023-11-28T18:23:10.914424+0000","last_became_active":"2023-11-28T18:23:08.119525+0000","last_became_peered":"2023-11-28T18:23:08.119525+0000","last_unstale":"2023-11-28T18:23:10.914424+0000","last_undegraded":"2023-11-28T18:23:10.914424+0000","last_fullsized":"2023-11-28T18:23:10.914424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:34:00.230528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.482666+0000","last_change":"2023-11-28T18:23:07.685651+0000","last_active":"2023-11-28T18:23:09.482666+0000","last_peered":"2023-11-28T18:23:09.482666+0000","last_clean":"2023-11-28T18:23:09.482666+0000","last_became_active":"2023-11-28T18:23:07.683869+0000","last_became_peered":"2023-11-28T18:23:07.683869+0000","last_unstale":"2023-11-28T18:23:09.482666+0000","last_undegraded":"2023-11-28T18:23:09.482666+0000","last_fullsized":"2023-11-28T18:23:09.482666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:35.383852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637574+0000","last_change":"2023-11-28T18:23:08.130016+0000","last_active":"2023-11-28T18:23:09.637574+0000","last_peered":"2023-11-28T18:23:09.637574+0000","last_clean":"2023-11-28T18:23:09.637574+0000","last_became_active":"2023-11-28T18:23:08.129586+0000","last_became_peered":"2023-11-28T18:23:08.129586+0000","last_unstale":"2023-11-28T18:23:09.637574+0000","last_undegraded":"2023-11-28T18:23:09.637574+0000","last_fullsized":"2023-11-28T18:23:09.637574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:00.449941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105620+0000","last_change":"2023-11-28T18:23:07.671773+0000","last_active":"2023-11-28T18:23:09.105620+0000","last_peered":"2023-11-28T18:23:09.105620+0000","last_clean":"2023-11-28T18:23:09.105620+0000","last_became_active":"2023-11-28T18:23:07.671465+0000","last_became_peered":"2023-11-28T18:23:07.671465+0000","last_unstale":"2023-11-28T18:23:09.105620+0000","last_undegraded":"2023-11-28T18:23:09.105620+0000","last_fullsized":"2023-11-28T18:23:09.105620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:44.862780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565865+0000","last_change":"2023-11-28T18:23:08.118146+0000","last_active":"2023-11-28T18:23:09.565865+0000","last_peered":"2023-11-28T18:23:09.565865+0000","last_clean":"2023-11-28T18:23:09.565865+0000","last_became_active":"2023-11-28T18:23:08.117848+0000","last_became_peered":"2023-11-28T18:23:08.117848+0000","last_unstale":"2023-11-28T18:23:09.565865+0000","last_undegraded":"2023-11-28T18:23:09.565865+0000","last_fullsized":"2023-11-28T18:23:09.565865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:33:20.745184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105627+0000","last_change":"2023-11-28T18:23:07.676709+0000","last_active":"2023-11-28T18:23:09.105627+0000","last_peered":"2023-11-28T18:23:09.105627+0000","last_clean":"2023-11-28T18:23:09.105627+0000","last_became_active":"2023-11-28T18:23:07.676414+0000","last_became_peered":"2023-11-28T18:23:07.676414+0000","last_unstale":"2023-11-28T18:23:09.105627+0000","last_undegraded":"2023-11-28T18:23:09.105627+0000","last_fullsized":"2023-11-28T18:23:09.105627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.282005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565753+0000","last_change":"2023-11-28T18:23:08.112410+0000","last_active":"2023-11-28T18:23:09.565753+0000","last_peered":"2023-11-28T18:23:09.565753+0000","last_clean":"2023-11-28T18:23:09.565753+0000","last_became_active":"2023-11-28T18:23:08.112097+0000","last_became_peered":"2023-11-28T18:23:08.112097+0000","last_unstale":"2023-11-28T18:23:09.565753+0000","last_undegraded":"2023-11-28T18:23:09.565753+0000","last_fullsized":"2023-11-28T18:23:09.565753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:57.891789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.668542+0000","last_change":"2023-11-28T18:23:07.684818+0000","last_active":"2023-11-28T18:23:13.668542+0000","last_peered":"2023-11-28T18:23:13.668542+0000","last_clean":"2023-11-28T18:23:13.668542+0000","last_became_active":"2023-11-28T18:23:07.684502+0000","last_became_peered":"2023-11-28T18:23:07.684502+0000","last_unstale":"2023-11-28T18:23:13.668542+0000","last_undegraded":"2023-11-28T18:23:13.668542+0000","last_fullsized":"2023-11-28T18:23:13.668542+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:16:21.223266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109527+0000","last_change":"2023-11-28T18:23:08.133972+0000","last_active":"2023-11-28T18:23:09.109527+0000","last_peered":"2023-11-28T18:23:09.109527+0000","last_clean":"2023-11-28T18:23:09.109527+0000","last_became_active":"2023-11-28T18:23:08.133747+0000","last_became_peered":"2023-11-28T18:23:08.133747+0000","last_unstale":"2023-11-28T18:23:09.109527+0000","last_undegraded":"2023-11-28T18:23:09.109527+0000","last_fullsized":"2023-11-28T18:23:09.109527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:24.968241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113538+0000","last_change":"2023-11-28T18:23:08.134117+0000","last_active":"2023-11-28T18:23:09.113538+0000","last_peered":"2023-11-28T18:23:09.113538+0000","last_clean":"2023-11-28T18:23:09.113538+0000","last_became_active":"2023-11-28T18:23:08.133801+0000","last_became_peered":"2023-11-28T18:23:08.133801+0000","last_unstale":"2023-11-28T18:23:09.113538+0000","last_undegraded":"2023-11-28T18:23:09.113538+0000","last_fullsized":"2023-11-28T18:23:09.113538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:02.370862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.669428+0000","last_change":"2023-11-28T18:23:07.686399+0000","last_active":"2023-11-28T18:23:13.669428+0000","last_peered":"2023-11-28T18:23:13.669428+0000","last_clean":"2023-11-28T18:23:13.669428+0000","last_became_active":"2023-11-28T18:23:07.686051+0000","last_became_peered":"2023-11-28T18:23:07.686051+0000","last_unstale":"2023-11-28T18:23:13.669428+0000","last_undegraded":"2023-11-28T18:23:13.669428+0000","last_fullsized":"2023-11-28T18:23:13.669428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:11.797516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113317+0000","last_change":"2023-11-28T18:23:08.134870+0000","last_active":"2023-11-28T18:23:09.113317+0000","last_peered":"2023-11-28T18:23:09.113317+0000","last_clean":"2023-11-28T18:23:09.113317+0000","last_became_active":"2023-11-28T18:23:08.133732+0000","last_became_peered":"2023-11-28T18:23:08.133732+0000","last_unstale":"2023-11-28T18:23:09.113317+0000","last_undegraded":"2023-11-28T18:23:09.113317+0000","last_fullsized":"2023-11-28T18:23:09.113317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:01.364242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.668135+0000","last_change":"2023-11-28T18:23:07.682230+0000","last_active":"2023-11-28T18:23:13.668135+0000","last_peered":"2023-11-28T18:23:13.668135+0000","last_clean":"2023-11-28T18:23:13.668135+0000","last_became_active":"2023-11-28T18:23:07.681715+0000","last_became_peered":"2023-11-28T18:23:07.681715+0000","last_unstale":"2023-11-28T18:23:13.668135+0000","last_undegraded":"2023-11-28T18:23:13.668135+0000","last_fullsized":"2023-11-28T18:23:13.668135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:02.343996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914216+0000","last_change":"2023-11-28T18:23:08.128927+0000","last_active":"2023-11-28T18:23:10.914216+0000","last_peered":"2023-11-28T18:23:10.914216+0000","last_clean":"2023-11-28T18:23:10.914216+0000","last_became_active":"2023-11-28T18:23:08.127775+0000","last_became_peered":"2023-11-28T18:23:08.127775+0000","last_unstale":"2023-11-28T18:23:10.914216+0000","last_undegraded":"2023-11-28T18:23:10.914216+0000","last_fullsized":"2023-11-28T18:23:10.914216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:50:05.531589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565730+0000","last_change":"2023-11-28T18:23:07.667929+0000","last_active":"2023-11-28T18:23:09.565730+0000","last_peered":"2023-11-28T18:23:09.565730+0000","last_clean":"2023-11-28T18:23:09.565730+0000","last_became_active":"2023-11-28T18:23:07.667666+0000","last_became_peered":"2023-11-28T18:23:07.667666+0000","last_unstale":"2023-11-28T18:23:09.565730+0000","last_undegraded":"2023-11-28T18:23:09.565730+0000","last_fullsized":"2023-11-28T18:23:09.565730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:32.132092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565609+0000","last_change":"2023-11-28T18:23:08.126404+0000","last_active":"2023-11-28T18:23:09.565609+0000","last_peered":"2023-11-28T18:23:09.565609+0000","last_clean":"2023-11-28T18:23:09.565609+0000","last_became_active":"2023-11-28T18:23:08.125315+0000","last_became_peered":"2023-11-28T18:23:08.125315+0000","last_unstale":"2023-11-28T18:23:09.565609+0000","last_undegraded":"2023-11-28T18:23:09.565609+0000","last_fullsized":"2023-11-28T18:23:09.565609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:24:59.527642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.674794+0000","last_change":"2023-11-28T18:23:07.677758+0000","last_active":"2023-11-28T18:23:13.674794+0000","last_peered":"2023-11-28T18:23:13.674794+0000","last_clean":"2023-11-28T18:23:13.674794+0000","last_became_active":"2023-11-28T18:23:07.676902+0000","last_became_peered":"2023-11-28T18:23:07.676902+0000","last_unstale":"2023-11-28T18:23:13.674794+0000","last_undegraded":"2023-11-28T18:23:13.674794+0000","last_fullsized":"2023-11-28T18:23:13.674794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:50.953647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T18:23:19.918 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T18:23:19.918 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-28T18:23:19.918 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-28T18:23:19.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T18:23:20.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4faa417640 1 Processor -- start 2023-11-28T18:23:20.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4faa417640 1 -- start start 2023-11-28T18:23:20.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4faa417640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4fa41542e0 0x7f4fa415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:20.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4faa417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cbb70 0x7f4fa40cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:20.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4faa417640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa40ca920 con 0x7f4fa40cc810 2023-11-28T18:23:20.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4faa417640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa4071d50 con 0x7f4fa41542e0 2023-11-28T18:23:20.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4faa417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4072030 con 0x7f4fa40cbb70 2023-11-28T18:23:20.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4fa41542e0 0x7f4fa415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:20.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4fa41542e0 0x7f4fa415e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59786/0 (socket says 172.21.15.7:59786) 2023-11-28T18:23:20.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa3fff640 1 -- 172.21.15.7:0/3964977221 learned_addr learned my addr 172.21.15.7:0/3964977221 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:20.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa3fff640 1 -- 172.21.15.7:0/3964977221 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cbb70 msgr2=0x7f4fa40cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:20.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa37fe640 1 --2- 172.21.15.7:0/3964977221 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cbb70 0x7f4fa40cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:20.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa3fff640 1 --2- 172.21.15.7:0/3964977221 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cbb70 0x7f4fa40cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa3fff640 1 -- 172.21.15.7:0/3964977221 >> v1:172.21.15.193:6789/0 conn(0x7f4fa40cc810 legacy=0x7f4fa4072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:20.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa3fff640 1 -- 172.21.15.7:0/3964977221 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa415e550 con 0x7f4fa41542e0 2023-11-28T18:23:20.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.096+0000 7f4fa37fe640 1 --2- 172.21.15.7:0/3964977221 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cbb70 0x7f4fa40cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4fa3fff640 1 --2- 172.21.15.7:0/3964977221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa41542e0 0x7f4fa415e010 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009870 tx=0x7f4f9401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f6967296456e3e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4fa2ffd640 1 -- 172.21.15.7:0/3964977221 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f940ac630 con 0x7f4fa41542e0 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4fa2ffd640 1 -- 172.21.15.7:0/3964977221 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f940ac790 con 0x7f4fa41542e0 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4fa2ffd640 1 -- 172.21.15.7:0/3964977221 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f940acb20 con 0x7f4fa41542e0 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4faa417640 1 -- 172.21.15.7:0/3964977221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa41542e0 msgr2=0x7f4fa415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4faa417640 1 --2- 172.21.15.7:0/3964977221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa41542e0 0x7f4fa415e010 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009870 tx=0x7f4f9401c500 comp rx=0 tx=0).stop 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4faa417640 1 -- 172.21.15.7:0/3964977221 shutdown_connections 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4faa417640 1 --2- 172.21.15.7:0/3964977221 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cbb70 0x7f4fa40cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4faa417640 1 --2- 172.21.15.7:0/3964977221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa41542e0 0x7f4fa415e010 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4faa417640 1 -- 172.21.15.7:0/3964977221 >> 172.21.15.7:0/3964977221 conn(0x7f4fa40717e0 msgr2=0x7f4fa40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:20.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4faa417640 1 -- 172.21.15.7:0/3964977221 shutdown_connections 2023-11-28T18:23:20.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.100+0000 7f4faa417640 1 -- 172.21.15.7:0/3964977221 wait complete. 2023-11-28T18:23:20.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 Processor -- start 2023-11-28T18:23:20.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 -- start start 2023-11-28T18:23:20.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa40cbb70 0x7f4fa41527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:20.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cc810 0x7f4fa4156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:20.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fa41542e0 0x7f4fa4157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:20.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa40c9b10 con 0x7f4fa40cbb70 2023-11-28T18:23:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4156980 con 0x7f4fa41542e0 2023-11-28T18:23:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4155b30 con 0x7f4fa40cc810 2023-11-28T18:23:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa40cbb70 0x7f4fa41527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa40cbb70 0x7f4fa41527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59798/0 (socket says 172.21.15.7:59798) 2023-11-28T18:23:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 -- 172.21.15.7:0/1175212961 learned_addr learned my addr 172.21.15.7:0/1175212961 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa37fe640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cc810 0x7f4fa4156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa89b4640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fa41542e0 0x7f4fa4157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 -- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cc810 msgr2=0x7f4fa4156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cc810 0x7f4fa4156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 -- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fa41542e0 msgr2=0x7f4fa4157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fa41542e0 0x7f4fa4157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 -- 172.21.15.7:0/1175212961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f940094d0 con 0x7f4fa40cbb70 2023-11-28T18:23:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa89b4640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fa41542e0 0x7f4fa4157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa37fe640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cc810 0x7f4fa4156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa3fff640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa40cbb70 0x7f4fa41527e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f9401cda0 tx=0x7f4f940192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:20.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa0ff9640 1 -- 172.21.15.7:0/1175212961 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f940ba070 con 0x7f4fa40cbb70 2023-11-28T18:23:20.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa0ff9640 1 -- 172.21.15.7:0/1175212961 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f940ac5b0 con 0x7f4fa40cbb70 2023-11-28T18:23:20.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4fa0ff9640 1 -- 172.21.15.7:0/1175212961 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f940b5870 con 0x7f4fa40cbb70 2023-11-28T18:23:20.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa4155570 con 0x7f4fa40cbb70 2023-11-28T18:23:20.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.104+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa4156330 con 0x7f4fa40cbb70 2023-11-28T18:23:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.108+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa406c670 con 0x7f4fa40cbb70 2023-11-28T18:23:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.108+0000 7f4fa0ff9640 1 -- 172.21.15.7:0/1175212961 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4f940b3070 con 0x7f4fa40cbb70 2023-11-28T18:23:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.108+0000 7f4fa0ff9640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f90066900 0x7f4f90068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.108+0000 7f4fa0ff9640 1 -- 172.21.15.7:0/1175212961 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f9411dbb0 con 0x7f4fa40cbb70 2023-11-28T18:23:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.108+0000 7f4fa37fe640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f90066900 0x7f4f90068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.108+0000 7f4fa37fe640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f90066900 0x7f4f90068dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4fa415b3b0 tx=0x7f4f98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:20.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.112+0000 7f4fa0ff9640 1 -- 172.21.15.7:0/1175212961 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4f940eae70 con 0x7f4fa40cbb70 2023-11-28T18:23:20.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4fa404e890 con 0x7f4fa40cbb70 2023-11-28T18:23:20.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4fa0ff9640 1 -- 172.21.15.7:0/1175212961 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f4f940a80e0 con 0x7f4fa40cbb70 2023-11-28T18:23:20.308 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:20.309 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-11-28T18:23:20.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f90066900 msgr2=0x7f4f90068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:20.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f90066900 0x7f4f90068dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4fa415b3b0 tx=0x7f4f98008040 comp rx=0 tx=0).stop 2023-11-28T18:23:20.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa40cbb70 msgr2=0x7f4fa41527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:20.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa40cbb70 0x7f4fa41527e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f9401cda0 tx=0x7f4f940192a0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 shutdown_connections 2023-11-28T18:23:20.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f90066900 0x7f4f90068dc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fa41542e0 0x7f4fa4157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4fa40cc810 0x7f4fa4156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 --2- 172.21.15.7:0/1175212961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fa40cbb70 0x7f4fa41527e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:20.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 >> 172.21.15.7:0/1175212961 conn(0x7f4fa40717e0 msgr2=0x7f4fa415dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:20.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 shutdown_connections 2023-11-28T18:23:20.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:20.304+0000 7f4faa417640 1 -- 172.21.15.7:0/1175212961 wait complete. 2023-11-28T18:23:21.972 INFO:tasks.ceph.mds.e.smithi007.stderr:2023-11-28T18:23:21.968+0000 7f7e1dce5640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-11-28T18:23:23.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T18:23:23.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17fd372640 1 Processor -- start 2023-11-28T18:23:23.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17fd372640 1 -- start start 2023-11-28T18:23:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17fd372640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f17f81542e0 0x7f17f815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17fd372640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f8072450 0x7f17f80728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17fd372640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f80ca920 con 0x7f17f80cbb70 2023-11-28T18:23:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17fd372640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f8071d50 con 0x7f17f81542e0 2023-11-28T18:23:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17fd372640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8072030 con 0x7f17f8072450 2023-11-28T18:23:23.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17f6ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f17f81542e0 0x7f17f815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:23.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17f77fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f8072450 0x7f17f80728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:23.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17f77fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f8072450 0x7f17f80728c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:44900/0 (socket says 172.21.15.7:44900) 2023-11-28T18:23:23.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17f77fe640 1 -- 172.21.15.7:0/1675976996 learned_addr learned my addr 172.21.15.7:0/1675976996 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:23.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.480+0000 7f17f6ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f17f81542e0 0x7f17f815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:47184/0 (socket says 172.21.15.7:47184) 2023-11-28T18:23:23.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17f6ffd640 1 -- 172.21.15.7:0/1675976996 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f8072450 msgr2=0x7f17f80728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:23.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17f6ffd640 1 --2- 172.21.15.7:0/1675976996 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f8072450 0x7f17f80728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17f6ffd640 1 -- 172.21.15.7:0/1675976996 >> v1:172.21.15.193:6789/0 conn(0x7f17f80cbb70 legacy=0x7f17f80cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:23.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17f6ffd640 1 -- 172.21.15.7:0/1675976996 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f815e550 con 0x7f17f81542e0 2023-11-28T18:23:23.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17f6ffd640 1 --2- 172.21.15.7:0/1675976996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f81542e0 0x7f17f815e010 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f17e80097f0 tx=0x7f17e801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1c694d30191c189 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:23.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17f67fc640 1 -- 172.21.15.7:0/1675976996 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e80ad640 con 0x7f17f81542e0 2023-11-28T18:23:23.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17f67fc640 1 -- 172.21.15.7:0/1675976996 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e80ad7a0 con 0x7f17f81542e0 2023-11-28T18:23:23.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17f67fc640 1 -- 172.21.15.7:0/1675976996 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e80adb30 con 0x7f17f81542e0 2023-11-28T18:23:23.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17fd372640 1 -- 172.21.15.7:0/1675976996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f81542e0 msgr2=0x7f17f815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:23.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.484+0000 7f17fd372640 1 --2- 172.21.15.7:0/1675976996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f81542e0 0x7f17f815e010 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f17e80097f0 tx=0x7f17e801c480 comp rx=0 tx=0).stop 2023-11-28T18:23:23.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 -- 172.21.15.7:0/1675976996 shutdown_connections 2023-11-28T18:23:23.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 --2- 172.21.15.7:0/1675976996 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f8072450 0x7f17f80728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 --2- 172.21.15.7:0/1675976996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f81542e0 0x7f17f815e010 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 -- 172.21.15.7:0/1675976996 >> 172.21.15.7:0/1675976996 conn(0x7f17f80717e0 msgr2=0x7f17f80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:23.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 -- 172.21.15.7:0/1675976996 shutdown_connections 2023-11-28T18:23:23.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 -- 172.21.15.7:0/1675976996 wait complete. 2023-11-28T18:23:23.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 Processor -- start 2023-11-28T18:23:23.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 -- start start 2023-11-28T18:23:23.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17f8072450 0x7f17f81f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:23.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f80cbb70 0x7f17f81ec910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:23.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f81542e0 0x7f17f81451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:23.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f815d0a0 con 0x7f17f8072450 2023-11-28T18:23:23.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f81ebec0 con 0x7f17f80cbb70 2023-11-28T18:23:23.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17fd372640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f81ec480 con 0x7f17f81542e0 2023-11-28T18:23:23.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17f8072450 0x7f17f81f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17f8072450 0x7f17f81f2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56978/0 (socket says 172.21.15.7:56978) 2023-11-28T18:23:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f6ffd640 1 -- 172.21.15.7:0/3789893443 learned_addr learned my addr 172.21.15.7:0/3789893443 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17effff640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f80cbb70 0x7f17f81ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f77fe640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f81542e0 0x7f17f81451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f6ffd640 1 -- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f81542e0 msgr2=0x7f17f81451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f6ffd640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f81542e0 0x7f17f81451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f6ffd640 1 -- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f80cbb70 msgr2=0x7f17f81ec910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f6ffd640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f80cbb70 0x7f17f81ec910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17f6ffd640 1 -- 172.21.15.7:0/3789893443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e8009420 con 0x7f17f8072450 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.488+0000 7f17effff640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f80cbb70 0x7f17f81ec910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17f6ffd640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17f8072450 0x7f17f81f2770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f17e80097f0 tx=0x7f17e8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17ef7fe640 1 -- 172.21.15.7:0/3789893443 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e80b6630 con 0x7f17f8072450 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17ef7fe640 1 -- 172.21.15.7:0/3789893443 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e80a6050 con 0x7f17f8072450 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17ef7fe640 1 -- 172.21.15.7:0/3789893443 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e80b6990 con 0x7f17f8072450 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f81f2fb0 con 0x7f17f8072450 2023-11-28T18:23:23.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f81f3570 con 0x7f17f8072450 2023-11-28T18:23:23.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f806c670 con 0x7f17f8072450 2023-11-28T18:23:23.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17ef7fe640 1 -- 172.21.15.7:0/3789893443 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f17e80b4040 con 0x7f17f8072450 2023-11-28T18:23:23.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17ef7fe640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f17d8066950 0x7f17d8068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:23.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17ef7fe640 1 -- 172.21.15.7:0/3789893443 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17e811e1a0 con 0x7f17f8072450 2023-11-28T18:23:23.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.492+0000 7f17effff640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f17d8066950 0x7f17d8068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:23.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.496+0000 7f17effff640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f17d8066950 0x7f17d8068e10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17f80cd930 tx=0x7f17e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:23.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.496+0000 7f17ef7fe640 1 -- 172.21.15.7:0/3789893443 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f17e80eb3c0 con 0x7f17f8072450 2023-11-28T18:23:23.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.692+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f17f804e890 con 0x7f17f8072450 2023-11-28T18:23:23.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.692+0000 7f17ef7fe640 1 -- 172.21.15.7:0/3789893443 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f17e80ee9e0 con 0x7f17f8072450 2023-11-28T18:23:23.699 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:23.699 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-11-28T18:23:23.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f17d8066950 msgr2=0x7f17d8068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f17d8066950 0x7f17d8068e10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17f80cd930 tx=0x7f17e0009040 comp rx=0 tx=0).stop 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17f8072450 msgr2=0x7f17f81f2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17f8072450 0x7f17f81f2770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f17e80097f0 tx=0x7f17e8019220 comp rx=0 tx=0).stop 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 shutdown_connections 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f17d8066950 0x7f17d8068e10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f17f81542e0 0x7f17f81451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f80cbb70 0x7f17f81ec910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 --2- 172.21.15.7:0/3789893443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17f8072450 0x7f17f81f2770 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 >> 172.21.15.7:0/3789893443 conn(0x7f17f80717e0 msgr2=0x7f17f815a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:23.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 shutdown_connections 2023-11-28T18:23:23.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:23.696+0000 7f17fd372640 1 -- 172.21.15.7:0/3789893443 wait complete. 2023-11-28T18:23:26.721 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T18:23:26.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.868+0000 7fbfae92a640 1 Processor -- start 2023-11-28T18:23:26.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.868+0000 7fbfae92a640 1 -- start start 2023-11-28T18:23:26.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfae92a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 0x7fbfa815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:26.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfae92a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbfa806e560 0x7fbfa8157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:26.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfae92a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa8071e60 con 0x7fbfa80701f0 2023-11-28T18:23:26.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfae92a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa8070930 con 0x7fbfa806e560 2023-11-28T18:23:26.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfae92a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8070660 con 0x7fbfa8151cd0 2023-11-28T18:23:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfa77fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbfa806e560 0x7fbfa8157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfa77fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbfa806e560 0x7fbfa8157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56994/0 (socket says 172.21.15.7:56994) 2023-11-28T18:23:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfa77fe640 1 -- 172.21.15.7:0/4244484966 learned_addr learned my addr 172.21.15.7:0/4244484966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfa7fff640 1 --2- 172.21.15.7:0/4244484966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 0x7fbfa815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfa77fe640 1 -- 172.21.15.7:0/4244484966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 msgr2=0x7fbfa815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfa77fe640 1 --2- 172.21.15.7:0/4244484966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 0x7fbfa815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfa77fe640 1 -- 172.21.15.7:0/4244484966 >> v1:172.21.15.193:6789/0 conn(0x7fbfa80701f0 legacy=0x7fbfa80c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.872+0000 7fbfa77fe640 1 -- 172.21.15.7:0/4244484966 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8070bf0 con 0x7fbfa806e560 2023-11-28T18:23:26.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfa77fe640 1 --2- 172.21.15.7:0/4244484966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa806e560 0x7fbfa8157b90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009870 tx=0x7fbf9c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34abaecf144c924 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:26.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfa6ffd640 1 -- 172.21.15.7:0/4244484966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0ac600 con 0x7fbfa806e560 2023-11-28T18:23:26.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfa6ffd640 1 -- 172.21.15.7:0/4244484966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c0a8050 con 0x7fbfa806e560 2023-11-28T18:23:26.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfa6ffd640 1 -- 172.21.15.7:0/4244484966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0ac980 con 0x7fbfa806e560 2023-11-28T18:23:26.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 -- 172.21.15.7:0/4244484966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa806e560 msgr2=0x7fbfa8157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:26.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 --2- 172.21.15.7:0/4244484966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa806e560 0x7fbfa8157b90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009870 tx=0x7fbf9c01c4b0 comp rx=0 tx=0).stop 2023-11-28T18:23:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 -- 172.21.15.7:0/4244484966 shutdown_connections 2023-11-28T18:23:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 --2- 172.21.15.7:0/4244484966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa806e560 0x7fbfa8157b90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 --2- 172.21.15.7:0/4244484966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 0x7fbfa815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 -- 172.21.15.7:0/4244484966 >> 172.21.15.7:0/4244484966 conn(0x7fbfa80662e0 msgr2=0x7fbfa80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 -- 172.21.15.7:0/4244484966 shutdown_connections 2023-11-28T18:23:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 -- 172.21.15.7:0/4244484966 wait complete. 2023-11-28T18:23:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 Processor -- start 2023-11-28T18:23:26.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.876+0000 7fbfae92a640 1 -- start start 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa806e560 0x7fbfa80ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa80701f0 0x7fbfa80ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 0x7fbfa8148010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa81574e0 con 0x7fbfa80701f0 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8157fb0 con 0x7fbfa806e560 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa81fa650 con 0x7fbfa8151cd0 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa80701f0 0x7fbfa80ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa80701f0 0x7fbfa80ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57004/0 (socket says 172.21.15.7:57004) 2023-11-28T18:23:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 -- 172.21.15.7:0/2618084913 learned_addr learned my addr 172.21.15.7:0/2618084913 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa7fff640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa806e560 0x7fbfa80ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfacec7640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 0x7fbfa8148010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 -- 172.21.15.7:0/2618084913 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 msgr2=0x7fbfa8148010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 0x7fbfa8148010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 -- 172.21.15.7:0/2618084913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa806e560 msgr2=0x7fbfa80ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa806e560 0x7fbfa80ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 -- 172.21.15.7:0/2618084913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0094d0 con 0x7fbfa80701f0 2023-11-28T18:23:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa77fe640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa80701f0 0x7fbfa80ca840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c01ab60 tx=0x7fbf9c01acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa4ff9640 1 -- 172.21.15.7:0/2618084913 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0b7450 con 0x7fbfa80701f0 2023-11-28T18:23:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa4ff9640 1 -- 172.21.15.7:0/2618084913 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c0b75b0 con 0x7fbfa80701f0 2023-11-28T18:23:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa80c4ab0 con 0x7fbfa80701f0 2023-11-28T18:23:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa4ff9640 1 -- 172.21.15.7:0/2618084913 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0b7940 con 0x7fbfa80701f0 2023-11-28T18:23:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa81fab80 con 0x7fbfa80701f0 2023-11-28T18:23:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa8151820 con 0x7fbfa80701f0 2023-11-28T18:23:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.880+0000 7fbfa4ff9640 1 -- 172.21.15.7:0/2618084913 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbf9c0b3070 con 0x7fbfa80701f0 2023-11-28T18:23:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.884+0000 7fbfa4ff9640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf94066900 0x7fbf94068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.884+0000 7fbfa4ff9640 1 -- 172.21.15.7:0/2618084913 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf9c11e9d0 con 0x7fbfa80701f0 2023-11-28T18:23:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.884+0000 7fbfa7fff640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf94066900 0x7fbf94068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:26.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.884+0000 7fbfa7fff640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf94066900 0x7fbf94068dc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009ac0 tx=0x7fbf98009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:26.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:26.888+0000 7fbfa4ff9640 1 -- 172.21.15.7:0/2618084913 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbf9c0ebc00 con 0x7fbfa80701f0 2023-11-28T18:23:26.972 INFO:tasks.ceph.mds.e.smithi007.stderr:2023-11-28T18:23:26.968+0000 7f7e1dce5640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-11-28T18:23:27.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:23:27.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:23:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.076+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbfa804e890 con 0x7fbfa80701f0 2023-11-28T18:23:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.076+0000 7fbfa4ff9640 1 -- 172.21.15.7:0/2618084913 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7fbf9c0ef220 con 0x7fbfa80701f0 2023-11-28T18:23:27.082 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:27.082 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-11-28T18:23:27.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf94066900 msgr2=0x7fbf94068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:27.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf94066900 0x7fbf94068dc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009ac0 tx=0x7fbf98009190 comp rx=0 tx=0).stop 2023-11-28T18:23:27.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa80701f0 msgr2=0x7fbfa80ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:27.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa80701f0 0x7fbfa80ca840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c01ab60 tx=0x7fbf9c01acf0 comp rx=0 tx=0).stop 2023-11-28T18:23:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 shutdown_connections 2023-11-28T18:23:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf94066900 0x7fbf94068dc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbfa8151cd0 0x7fbfa8148010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfa80701f0 0x7fbfa80ca840 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 --2- 172.21.15.7:0/2618084913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa806e560 0x7fbfa80ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 >> 172.21.15.7:0/2618084913 conn(0x7fbfa80662e0 msgr2=0x7fbfa814b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 shutdown_connections 2023-11-28T18:23:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:27.080+0000 7fbfae92a640 1 -- 172.21.15.7:0/2618084913 wait complete. 2023-11-28T18:23:30.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T18:23:30.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.260+0000 7ff732380640 1 Processor -- start 2023-11-28T18:23:30.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.260+0000 7ff732380640 1 -- start start 2023-11-28T18:23:30.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff732380640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c1540a0 0x7ff72c072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff732380640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff72c0c8ec0 0x7ff72c0c9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff732380640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0c8750 con 0x7ff72c15d350 2023-11-28T18:23:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff732380640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72c0c9890 con 0x7ff72c0c8ec0 2023-11-28T18:23:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff732380640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0c8a30 con 0x7ff72c1540a0 2023-11-28T18:23:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff72bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c1540a0 0x7ff72c072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff73091d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff72c0c8ec0 0x7ff72c0c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff72b7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff72c15d350 0x7ff72c06a5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:39126/0 (socket says 172.21.15.7:39126) 2023-11-28T18:23:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff72bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c1540a0 0x7ff72c072ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:44932/0 (socket says 172.21.15.7:44932) 2023-11-28T18:23:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff72b7fe640 1 -- 172.21.15.7:0/2996201641 learned_addr learned my addr 172.21.15.7:0/2996201641 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff72bfff640 1 -- 172.21.15.7:0/2996201641 >> v2:172.21.15.193:3300/0 conn(0x7ff72c0c8ec0 msgr2=0x7ff72c0c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff72bfff640 1 --2- 172.21.15.7:0/2996201641 >> v2:172.21.15.193:3300/0 conn(0x7ff72c0c8ec0 0x7ff72c0c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff72bfff640 1 -- 172.21.15.7:0/2996201641 >> v1:172.21.15.193:6789/0 conn(0x7ff72c15d350 legacy=0x7ff72c06a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.264+0000 7ff72bfff640 1 -- 172.21.15.7:0/2996201641 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c15d840 con 0x7ff72c1540a0 2023-11-28T18:23:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff72bfff640 1 --2- 172.21.15.7:0/2996201641 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c1540a0 0x7ff72c072ab0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0097f0 tx=0x7ff71c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28e289a3bdeb20ff server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:30.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff72affd640 1 -- 172.21.15.7:0/2996201641 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71c0ab600 con 0x7ff72c1540a0 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff72affd640 1 -- 172.21.15.7:0/2996201641 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c0a7050 con 0x7ff72c1540a0 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff72affd640 1 -- 172.21.15.7:0/2996201641 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71c0ab980 con 0x7ff72c1540a0 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 -- 172.21.15.7:0/2996201641 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c1540a0 msgr2=0x7ff72c072ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 --2- 172.21.15.7:0/2996201641 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c1540a0 0x7ff72c072ab0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0097f0 tx=0x7ff71c01c480 comp rx=0 tx=0).stop 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 -- 172.21.15.7:0/2996201641 shutdown_connections 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 --2- 172.21.15.7:0/2996201641 >> v2:172.21.15.193:3300/0 conn(0x7ff72c0c8ec0 0x7ff72c0c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 --2- 172.21.15.7:0/2996201641 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c1540a0 0x7ff72c072ab0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 -- 172.21.15.7:0/2996201641 >> 172.21.15.7:0/2996201641 conn(0x7ff72c073cd0 msgr2=0x7ff72c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 -- 172.21.15.7:0/2996201641 shutdown_connections 2023-11-28T18:23:30.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 -- 172.21.15.7:0/2996201641 wait complete. 2023-11-28T18:23:30.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 Processor -- start 2023-11-28T18:23:30.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.268+0000 7ff732380640 1 -- start start 2023-11-28T18:23:30.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff72c0c8ec0 0x7ff72c14f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72c1540a0 0x7ff72c148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c15d350 0x7ff72c149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c149df0 con 0x7ff72c1540a0 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c148980 con 0x7ff72c0c8ec0 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c1522a0 con 0x7ff72c15d350 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72c1540a0 0x7ff72c148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72c1540a0 0x7ff72c148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57008/0 (socket says 172.21.15.7:57008) 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 -- 172.21.15.7:0/2141972641 learned_addr learned my addr 172.21.15.7:0/2141972641 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72bfff640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff72c0c8ec0 0x7ff72c14f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff73091d640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c15d350 0x7ff72c149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 -- 172.21.15.7:0/2141972641 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c15d350 msgr2=0x7ff72c149310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c15d350 0x7ff72c149310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 -- 172.21.15.7:0/2141972641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff72c0c8ec0 msgr2=0x7ff72c14f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff72c0c8ec0 0x7ff72c14f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 -- 172.21.15.7:0/2141972641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c009420 con 0x7ff72c1540a0 2023-11-28T18:23:30.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff72b7fe640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72c1540a0 0x7ff72c148dd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff720001ec0 tx=0x7ff72000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:30.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff728ff9640 1 -- 172.21.15.7:0/2141972641 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720005cf0 con 0x7ff72c1540a0 2023-11-28T18:23:30.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff728ff9640 1 -- 172.21.15.7:0/2141972641 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff720005e50 con 0x7ff72c1540a0 2023-11-28T18:23:30.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff728ff9640 1 -- 172.21.15.7:0/2141972641 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7200072d0 con 0x7ff72c1540a0 2023-11-28T18:23:30.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c0cc240 con 0x7ff72c1540a0 2023-11-28T18:23:30.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c149fd0 con 0x7ff72c1540a0 2023-11-28T18:23:30.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.272+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c148af0 con 0x7ff72c1540a0 2023-11-28T18:23:30.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.276+0000 7ff728ff9640 1 -- 172.21.15.7:0/2141972641 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff720007430 con 0x7ff72c1540a0 2023-11-28T18:23:30.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.276+0000 7ff728ff9640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff718066900 0x7ff718068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.276+0000 7ff728ff9640 1 -- 172.21.15.7:0/2141972641 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff720083960 con 0x7ff72c1540a0 2023-11-28T18:23:30.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.276+0000 7ff72bfff640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff718066900 0x7ff718068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.276+0000 7ff72bfff640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff718066900 0x7ff718068dc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0ad000 tx=0x7ff71c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.280+0000 7ff728ff9640 1 -- 172.21.15.7:0/2141972641 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff72c148af0 con 0x7ff72c1540a0 2023-11-28T18:23:30.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.468+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff72c148af0 con 0x7ff72c1540a0 2023-11-28T18:23:30.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff728ff9640 1 -- 172.21.15.7:0/2141972641 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff72c148af0 con 0x7ff72c1540a0 2023-11-28T18:23:30.476 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:30.476 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff718066900 msgr2=0x7ff718068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff718066900 0x7ff718068dc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0ad000 tx=0x7ff71c0a6000 comp rx=0 tx=0).stop 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72c1540a0 msgr2=0x7ff72c148dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72c1540a0 0x7ff72c148dd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff720001ec0 tx=0x7ff72000c890 comp rx=0 tx=0).stop 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 shutdown_connections 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff718066900 0x7ff718068dc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff72c15d350 0x7ff72c149310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72c1540a0 0x7ff72c148dd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 --2- 172.21.15.7:0/2141972641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff72c0c8ec0 0x7ff72c14f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 >> 172.21.15.7:0/2141972641 conn(0x7ff72c073cd0 msgr2=0x7ff72c0c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:30.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 shutdown_connections 2023-11-28T18:23:30.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.472+0000 7ff732380640 1 -- 172.21.15.7:0/2141972641 wait complete. 2023-11-28T18:23:30.496 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-28T18:23:30.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:30.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d6e17640 1 Processor -- start 2023-11-28T18:23:30.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d6e17640 1 -- start start 2023-11-28T18:23:30.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d6e17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 0x7f63d00cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d6e17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d00ca920 con 0x7f63d01542e0 2023-11-28T18:23:30.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d6e17640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d00c4cd0 con 0x7f63d00cc810 2023-11-28T18:23:30.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d6e17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0071d50 con 0x7f63d00cbb70 2023-11-28T18:23:30.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d4bb3640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f63d01542e0 0x7f63d015e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42230/0 (socket says 172.21.15.7:42230) 2023-11-28T18:23:30.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d4bb3640 1 -- 172.21.15.7:0/153332368 learned_addr learned my addr 172.21.15.7:0/153332368 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cffff640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 0x7f63d00cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796229682 0 0) 0x7f63d00ca920 con 0x7f63d01542e0 2023-11-28T18:23:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c0003530 con 0x7f63d01542e0 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1504727297 0 0) 0x7f63c0003530 con 0x7f63d01542e0 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 msgr2=0x7f63d00cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 0x7f63d00cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 >> v1:172.21.15.193:6789/0 conn(0x7f63d00cc810 legacy=0x7f63d0072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00cab70 con 0x7f63d01542e0 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f63c4001680 con 0x7f63d01542e0 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 >> v1:172.21.15.7:6789/0 conn(0x7f63d01542e0 legacy=0x7f63d015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63c00039b0 0x7f63c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63c0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 0x7f63c0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0071d50 con 0x7f63c00047d0 2023-11-28T18:23:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c000baa0 con 0x7f63c00039b0 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c000bc60 con 0x7f63d00cbb70 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63cffff640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63c0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d4bb3640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63c00039b0 0x7f63c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.636+0000 7f63d53b4640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 0x7f63c0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cffff640 1 -- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 msgr2=0x7f63c0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cffff640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 0x7f63c0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cffff640 1 -- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63c00039b0 msgr2=0x7f63c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cffff640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63c00039b0 0x7f63c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cffff640 1 -- 172.21.15.7:0/153332368 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0003530 con 0x7f63c00047d0 2023-11-28T18:23:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63d53b4640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d00cbb70 0x7f63c0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cffff640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63c0004c50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009ab0 tx=0x7f63bc01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e04ad47f2074570 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0a5600 con 0x7f63c00047d0 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc0a5760 con 0x7f63c00047d0 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63cf7fe640 1 -- 172.21.15.7:0/153332368 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0a5b10 con 0x7f63c00047d0 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63d6e17640 1 -- 172.21.15.7:0/153332368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 msgr2=0x7f63c0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63d6e17640 1 --2- 172.21.15.7:0/153332368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63c0004c50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009ab0 tx=0x7f63bc01ca10 comp rx=0 tx=0).stop 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63d53b4640 1 -- 172.21.15.7:0/153332368 reap_dead start 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63d6e17640 1 -- 172.21.15.7:0/153332368 shutdown_connections 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63d6e17640 1 -- 172.21.15.7:0/153332368 >> 172.21.15.7:0/153332368 conn(0x7f63d00717e0 msgr2=0x7f63d00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.640+0000 7f63d6e17640 1 -- 172.21.15.7:0/153332368 shutdown_connections 2023-11-28T18:23:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 -- 172.21.15.7:0/153332368 wait complete. 2023-11-28T18:23:30.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 Processor -- start 2023-11-28T18:23:30.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 -- start start 2023-11-28T18:23:30.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63d01f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d01e56a0 0x7f63d014a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d01553f0 0x7f63d015a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0072030 con 0x7f63c00047d0 2023-11-28T18:23:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d015ae80 con 0x7f63d01e56a0 2023-11-28T18:23:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d6e17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d014ab40 con 0x7f63d01553f0 2023-11-28T18:23:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63d01f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63d01f00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57028/0 (socket says 172.21.15.7:57028) 2023-11-28T18:23:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 -- 172.21.15.7:0/4156180136 learned_addr learned my addr 172.21.15.7:0/4156180136 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63cffff640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d01e56a0 0x7f63d014a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d53b4640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d01553f0 0x7f63d015a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 -- 172.21.15.7:0/4156180136 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d01553f0 msgr2=0x7f63d015a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d01553f0 0x7f63d015a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 -- 172.21.15.7:0/4156180136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d01e56a0 msgr2=0x7f63d014a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d01e56a0 0x7f63d014a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 -- 172.21.15.7:0/4156180136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00d06b0 con 0x7f63c00047d0 2023-11-28T18:23:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.644+0000 7f63d4bb3640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63d01f00f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f63c4004880 tx=0x7f63c4012900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63cd7fa640 1 -- 172.21.15.7:0/4156180136 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4014810 con 0x7f63c00047d0 2023-11-28T18:23:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63cd7fa640 1 -- 172.21.15.7:0/4156180136 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c40149b0 con 0x7f63c00047d0 2023-11-28T18:23:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d01549e0 con 0x7f63c00047d0 2023-11-28T18:23:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63cd7fa640 1 -- 172.21.15.7:0/4156180136 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4014d60 con 0x7f63c00047d0 2023-11-28T18:23:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d0154fa0 con 0x7f63c00047d0 2023-11-28T18:23:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d006c670 con 0x7f63c00047d0 2023-11-28T18:23:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63cd7fa640 1 -- 172.21.15.7:0/4156180136 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f63c4016480 con 0x7f63c00047d0 2023-11-28T18:23:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63cd7fa640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63b40668d0 0x7f63b4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63cd7fa640 1 -- 172.21.15.7:0/4156180136 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63c40879a0 con 0x7f63c00047d0 2023-11-28T18:23:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.648+0000 7f63cffff640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63b40668d0 0x7f63b4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.652+0000 7f63cffff640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63b40668d0 0x7f63b4068d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0af000 tx=0x7f63bc0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.652+0000 7f63cd7fa640 1 -- 172.21.15.7:0/4156180136 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f63c4054b50 con 0x7f63c00047d0 2023-11-28T18:23:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.812+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63d004e890 con 0x7f63c00047d0 2023-11-28T18:23:30.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.812+0000 7f63cd7fa640 1 -- 172.21.15.7:0/4156180136 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f63c4058170 con 0x7f63c00047d0 2023-11-28T18:23:30.818 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:30.818 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63b40668d0 msgr2=0x7f63b4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63b40668d0 0x7f63b4068d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0af000 tx=0x7f63bc0046d0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 msgr2=0x7f63d01f00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63d01f00f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f63c4004880 tx=0x7f63c4012900 comp rx=0 tx=0).stop 2023-11-28T18:23:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 shutdown_connections 2023-11-28T18:23:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63b40668d0 0x7f63b4068d90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63d01553f0 0x7f63d015a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d01e56a0 0x7f63d014a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 --2- 172.21.15.7:0/4156180136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00047d0 0x7f63d01f00f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 >> 172.21.15.7:0/4156180136 conn(0x7f63d00717e0 msgr2=0x7f63d0152460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 shutdown_connections 2023-11-28T18:23:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.816+0000 7f63d6e17640 1 -- 172.21.15.7:0/4156180136 wait complete. 2023-11-28T18:23:30.840 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T18:23:05.706694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T18:23:07.048910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:23:30.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:23:30.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.980+0000 7fa90df2a640 1 Processor -- start 2023-11-28T18:23:30.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.980+0000 7fa90df2a640 1 -- start start 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.980+0000 7fa90df2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e3a0 0x7fa9080c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.980+0000 7fa90df2a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa90806e7c0 0x7fa9080c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.980+0000 7fa90df2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa908157f60 con 0x7fa908141030 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.980+0000 7fa90df2a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa908158240 con 0x7fa90806e7c0 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.980+0000 7fa90df2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa908158520 con 0x7fa90806e3a0 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa907fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa90806e7c0 0x7fa9080c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa907fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa90806e7c0 0x7fa9080c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57034/0 (socket says 172.21.15.7:57034) 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa907fff640 1 -- 172.21.15.7:0/114008437 learned_addr learned my addr 172.21.15.7:0/114008437 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:30.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa906ffd640 1 --2- 172.21.15.7:0/114008437 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e3a0 0x7fa9080c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa907fff640 1 -- 172.21.15.7:0/114008437 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e3a0 msgr2=0x7fa9080c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:30.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa907fff640 1 --2- 172.21.15.7:0/114008437 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e3a0 0x7fa9080c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa907fff640 1 -- 172.21.15.7:0/114008437 >> v1:172.21.15.193:6789/0 conn(0x7fa908141030 legacy=0x7fa90815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa907fff640 1 -- 172.21.15.7:0/114008437 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908158800 con 0x7fa90806e7c0 2023-11-28T18:23:30.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa907fff640 1 --2- 172.21.15.7:0/114008437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90806e7c0 0x7fa9080c50d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa900009e50 tx=0x7fa90001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3099f8ced74c92 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:30.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa9067fc640 1 -- 172.21.15.7:0/114008437 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9000af400 con 0x7fa90806e7c0 2023-11-28T18:23:30.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa9067fc640 1 -- 172.21.15.7:0/114008437 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9000ab050 con 0x7fa90806e7c0 2023-11-28T18:23:30.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.984+0000 7fa9067fc640 1 -- 172.21.15.7:0/114008437 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9000af780 con 0x7fa90806e7c0 2023-11-28T18:23:30.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- 172.21.15.7:0/114008437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90806e7c0 msgr2=0x7fa9080c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 --2- 172.21.15.7:0/114008437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90806e7c0 0x7fa9080c50d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa900009e50 tx=0x7fa90001f1f0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- 172.21.15.7:0/114008437 shutdown_connections 2023-11-28T18:23:30.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 --2- 172.21.15.7:0/114008437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90806e7c0 0x7fa9080c50d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 --2- 172.21.15.7:0/114008437 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e3a0 0x7fa9080c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- 172.21.15.7:0/114008437 >> 172.21.15.7:0/114008437 conn(0x7fa908073f00 msgr2=0x7fa9080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:30.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- 172.21.15.7:0/114008437 shutdown_connections 2023-11-28T18:23:30.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- 172.21.15.7:0/114008437 wait complete. 2023-11-28T18:23:30.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 Processor -- start 2023-11-28T18:23:30.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- start start 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa90806e3a0 0x7fa9080cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e7c0 0x7fa9080cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa908141030 0x7fa908153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9080c9da0 con 0x7fa908141030 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9080ca040 con 0x7fa90806e3a0 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa90df2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9080ca2e0 con 0x7fa90806e7c0 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa907fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa908141030 0x7fa908153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa907fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa908141030 0x7fa908153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57042/0 (socket says 172.21.15.7:57042) 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa907fff640 1 -- 172.21.15.7:0/1209952448 learned_addr learned my addr 172.21.15.7:0/1209952448 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa906ffd640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e7c0 0x7fa9080cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.988+0000 7fa9077fe640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa90806e3a0 0x7fa9080cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa907fff640 1 -- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e7c0 msgr2=0x7fa9080cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa907fff640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e7c0 0x7fa9080cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa907fff640 1 -- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa90806e3a0 msgr2=0x7fa9080cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa907fff640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa90806e3a0 0x7fa9080cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa907fff640 1 -- 172.21.15.7:0/1209952448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa900009b30 con 0x7fa908141030 2023-11-28T18:23:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa9077fe640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa90806e3a0 0x7fa9080cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa907fff640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa908141030 0x7fa908153250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa90001ce20 tx=0x7fa90001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa8effff640 1 -- 172.21.15.7:0/1209952448 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9000c0070 con 0x7fa908141030 2023-11-28T18:23:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa8effff640 1 -- 172.21.15.7:0/1209952448 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa900009510 con 0x7fa908141030 2023-11-28T18:23:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9080ca5a0 con 0x7fa908141030 2023-11-28T18:23:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa8effff640 1 -- 172.21.15.7:0/1209952448 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9000afcf0 con 0x7fa908141030 2023-11-28T18:23:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9080cab60 con 0x7fa908141030 2023-11-28T18:23:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90804e890 con 0x7fa908141030 2023-11-28T18:23:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa8effff640 1 -- 172.21.15.7:0/1209952448 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa9000b6040 con 0x7fa908141030 2023-11-28T18:23:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa8effff640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8f4066900 0x7fa8f4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa8effff640 1 -- 172.21.15.7:0/1209952448 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa900121f00 con 0x7fa908141030 2023-11-28T18:23:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.992+0000 7fa9077fe640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8f4066900 0x7fa8f4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.996+0000 7fa9077fe640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8f4066900 0x7fa8f4068dc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8000f70 tx=0x7fa8f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:30.996+0000 7fa8effff640 1 -- 172.21.15.7:0/1209952448 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa9000ef170 con 0x7fa908141030 2023-11-28T18:23:31.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9080c6030 con 0x7fa908141030 2023-11-28T18:23:31.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa8effff640 1 -- 172.21.15.7:0/1209952448 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9000f2790 con 0x7fa908141030 2023-11-28T18:23:31.165 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:23:31.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8f4066900 msgr2=0x7fa8f4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8f4066900 0x7fa8f4068dc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8000f70 tx=0x7fa8f8009040 comp rx=0 tx=0).stop 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa908141030 msgr2=0x7fa908153250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa908141030 0x7fa908153250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa90001ce20 tx=0x7fa90001bf90 comp rx=0 tx=0).stop 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 shutdown_connections 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8f4066900 0x7fa8f4068dc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa908141030 0x7fa908153250 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa90806e7c0 0x7fa9080cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 --2- 172.21.15.7:0/1209952448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa90806e3a0 0x7fa9080cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 >> 172.21.15.7:0/1209952448 conn(0x7fa908073f00 msgr2=0x7fa90813b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 shutdown_connections 2023-11-28T18:23:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.160+0000 7fa90df2a640 1 -- 172.21.15.7:0/1209952448 wait complete. 2023-11-28T18:23:31.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T18:23:31.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.328+0000 7fcfab09f640 1 Processor -- start 2023-11-28T18:23:31.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.328+0000 7fcfab09f640 1 -- start start 2023-11-28T18:23:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfab09f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcfa40ca280 0x7fcfa40cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfab09f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fcfa40cf5e0 0x7fcfa40cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfab09f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa415ddb0 con 0x7fcfa40b3da0 2023-11-28T18:23:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfab09f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa40cff30 con 0x7fcfa40ca280 2023-11-28T18:23:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfab09f640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa40ca800 con 0x7fcfa40cf5e0 2023-11-28T18:23:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa989c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcfa40ca280 0x7fcfa40cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa989c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcfa40ca280 0x7fcfa40cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57050/0 (socket says 172.21.15.7:57050) 2023-11-28T18:23:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa989c640 1 -- 172.21.15.7:0/448557146 learned_addr learned my addr 172.21.15.7:0/448557146 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa989c640 1 -- 172.21.15.7:0/448557146 >> v2:172.21.15.193:3300/0 conn(0x7fcfa40cf5e0 msgr2=0x7fcfa40cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa989c640 1 --2- 172.21.15.7:0/448557146 >> v2:172.21.15.193:3300/0 conn(0x7fcfa40cf5e0 0x7fcfa40cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa989c640 1 -- 172.21.15.7:0/448557146 >> v1:172.21.15.7:6789/0 conn(0x7fcfa40b3da0 legacy=0x7fcfa415d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa989c640 1 -- 172.21.15.7:0/448557146 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa415e020 con 0x7fcfa40ca280 2023-11-28T18:23:31.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa989c640 1 --2- 172.21.15.7:0/448557146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa40cba10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40c9610 tx=0x7fcf9401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d15e863be5447 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:31.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa909b640 1 -- 172.21.15.7:0/448557146 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940aa430 con 0x7fcfa40ca280 2023-11-28T18:23:31.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa909b640 1 -- 172.21.15.7:0/448557146 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf940aa590 con 0x7fcfa40ca280 2023-11-28T18:23:31.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.332+0000 7fcfa909b640 1 -- 172.21.15.7:0/448557146 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940aa920 con 0x7fcfa40ca280 2023-11-28T18:23:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- 172.21.15.7:0/448557146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 msgr2=0x7fcfa40cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 --2- 172.21.15.7:0/448557146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa40cba10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40c9610 tx=0x7fcf9401c4d0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- 172.21.15.7:0/448557146 shutdown_connections 2023-11-28T18:23:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 --2- 172.21.15.7:0/448557146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa40cba10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 --2- 172.21.15.7:0/448557146 >> v2:172.21.15.193:3300/0 conn(0x7fcfa40cf5e0 0x7fcfa40cf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- 172.21.15.7:0/448557146 >> 172.21.15.7:0/448557146 conn(0x7fcfa4071210 msgr2=0x7fcfa40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- 172.21.15.7:0/448557146 shutdown_connections 2023-11-28T18:23:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- 172.21.15.7:0/448557146 wait complete. 2023-11-28T18:23:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 Processor -- start 2023-11-28T18:23:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- start start 2023-11-28T18:23:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa40b3da0 0x7fcfa41ec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa41ec8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfa40cf5e0 0x7fcfa4152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa41ebc40 con 0x7fcfa40ca280 2023-11-28T18:23:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa41ebf20 con 0x7fcfa40b3da0 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfab09f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa41f29e0 con 0x7fcfa40cf5e0 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfa989c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa41ec8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfa989c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa41ec8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57064/0 (socket says 172.21.15.7:57064) 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfa989c640 1 -- 172.21.15.7:0/361281228 learned_addr learned my addr 172.21.15.7:0/361281228 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfaa89e640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfa40cf5e0 0x7fcfa4152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.336+0000 7fcfaa09d640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa40b3da0 0x7fcfa41ec3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfa989c640 1 -- 172.21.15.7:0/361281228 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfa40cf5e0 msgr2=0x7fcfa4152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfa989c640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfa40cf5e0 0x7fcfa4152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfa989c640 1 -- 172.21.15.7:0/361281228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa40b3da0 msgr2=0x7fcfa41ec3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfa989c640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa40b3da0 0x7fcfa41ec3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfa989c640 1 -- 172.21.15.7:0/361281228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf940094d0 con 0x7fcfa40ca280 2023-11-28T18:23:31.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfa989c640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa41ec8f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcf9401c9b0 tx=0x7fcf9401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcf9affd640 1 -- 172.21.15.7:0/361281228 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940b3800 con 0x7fcfa40ca280 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcf9affd640 1 -- 172.21.15.7:0/361281228 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf940b39a0 con 0x7fcfa40ca280 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcf9affd640 1 -- 172.21.15.7:0/361281228 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940b3d50 con 0x7fcfa40ca280 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa41f2cc0 con 0x7fcfa40ca280 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa41f3280 con 0x7fcfa40ca280 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa404e890 con 0x7fcfa40ca280 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcf9affd640 1 -- 172.21.15.7:0/361281228 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fcf940b2380 con 0x7fcfa40ca280 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcf9affd640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf880668d0 0x7fcf88068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcf9affd640 1 -- 172.21.15.7:0/361281228 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf9411e8f0 con 0x7fcfa40ca280 2023-11-28T18:23:31.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.340+0000 7fcfaa09d640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf880668d0 0x7fcf88068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.344+0000 7fcfaa09d640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf880668d0 0x7fcf88068d90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0000f70 tx=0x7fcfa0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:31.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.344+0000 7fcf9affd640 1 -- 172.21.15.7:0/361281228 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcf940ebbe0 con 0x7fcfa40ca280 2023-11-28T18:23:31.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.504+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcfa40c8700 con 0x7fcfa40ca280 2023-11-28T18:23:31.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.504+0000 7fcf9affd640 1 -- 172.21.15.7:0/361281228 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcf940ef200 con 0x7fcfa40ca280 2023-11-28T18:23:31.510 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-11-28T18:23:31.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf880668d0 msgr2=0x7fcf88068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf880668d0 0x7fcf88068d90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0000f70 tx=0x7fcfa0009040 comp rx=0 tx=0).stop 2023-11-28T18:23:31.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 msgr2=0x7fcfa41ec8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa41ec8f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcf9401c9b0 tx=0x7fcf9401ad10 comp rx=0 tx=0).stop 2023-11-28T18:23:31.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 shutdown_connections 2023-11-28T18:23:31.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcf880668d0 0x7fcf88068d90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfa40cf5e0 0x7fcfa4152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa40ca280 0x7fcfa41ec8f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 --2- 172.21.15.7:0/361281228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa40b3da0 0x7fcfa41ec3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 >> 172.21.15.7:0/361281228 conn(0x7fcfa4071210 msgr2=0x7fcfa40b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:31.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 shutdown_connections 2023-11-28T18:23:31.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.508+0000 7fcfab09f640 1 -- 172.21.15.7:0/361281228 wait complete. 2023-11-28T18:23:31.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T18:23:31.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.672+0000 7f874458d640 1 Processor -- start 2023-11-28T18:23:31.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.672+0000 7f874458d640 1 -- start start 2023-11-28T18:23:31.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f874458d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f873c154130 0x7f873c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f874458d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c0caeb0 con 0x7f873c0ca530 2023-11-28T18:23:31.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f874458d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c15d9b0 con 0x7f873c0c4c80 2023-11-28T18:23:31.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f874458d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873c15a3f0 con 0x7f873c154130 2023-11-28T18:23:31.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8742b2a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f873c0ca530 0x7f873c0cb2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42256/0 (socket says 172.21.15.7:42256) 2023-11-28T18:23:31.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8742b2a640 1 -- 172.21.15.7:0/709786276 learned_addr learned my addr 172.21.15.7:0/709786276 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:31.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8742329640 1 --2- 172.21.15.7:0/709786276 >> v2:172.21.15.193:3300/0 conn(0x7f873c154130 0x7f873c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588450274 0 0) 0x7f873c0caeb0 con 0x7f873c0ca530 2023-11-28T18:23:31.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8728003530 con 0x7f873c0ca530 2023-11-28T18:23:31.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3406412722 0 0) 0x7f8728003530 con 0x7f873c0ca530 2023-11-28T18:23:31.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 >> v2:172.21.15.193:3300/0 conn(0x7f873c154130 msgr2=0x7f873c159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 --2- 172.21.15.7:0/709786276 >> v2:172.21.15.193:3300/0 conn(0x7f873c154130 0x7f873c159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 >> v1:172.21.15.193:6789/0 conn(0x7f873c0c4c80 legacy=0x7f873c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c15a5d0 con 0x7f873c0ca530 2023-11-28T18:23:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f8734001a00 con 0x7f873c0ca530 2023-11-28T18:23:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 >> v1:172.21.15.7:6789/0 conn(0x7f873c0ca530 legacy=0x7f873c0cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8728003980 0x7f8728003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87280047d0 0x7f8728004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8728005190 0x7f872800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c15a3f0 con 0x7f87280047d0 2023-11-28T18:23:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872800c040 con 0x7f8728005190 2023-11-28T18:23:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872800c200 con 0x7f8728003980 2023-11-28T18:23:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741b28640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87280047d0 0x7f8728004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8742329640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8728003980 0x7f8728003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8742b2a640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8728005190 0x7f872800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741b28640 1 -- 172.21.15.7:0/709786276 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8728003980 msgr2=0x7f8728003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741b28640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8728003980 0x7f8728003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741b28640 1 -- 172.21.15.7:0/709786276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8728005190 msgr2=0x7f872800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741b28640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8728005190 0x7f872800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8741b28640 1 -- 172.21.15.7:0/709786276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728003530 con 0x7f87280047d0 2023-11-28T18:23:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.676+0000 7f8742b2a640 1 -- 172.21.15.7:0/709786276 reap_dead start 2023-11-28T18:23:31.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f8741b28640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87280047d0 0x7f8728004c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f872c001780 tx=0x7f872c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420d0c0d44f16d1b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c0052d0 con 0x7f87280047d0 2023-11-28T18:23:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c0a7050 con 0x7f87280047d0 2023-11-28T18:23:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f8741327640 1 -- 172.21.15.7:0/709786276 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c003600 con 0x7f87280047d0 2023-11-28T18:23:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f874458d640 1 -- 172.21.15.7:0/709786276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87280047d0 msgr2=0x7f8728004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f874458d640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87280047d0 0x7f8728004c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f872c001780 tx=0x7f872c01c3e0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f874458d640 1 -- 172.21.15.7:0/709786276 shutdown_connections 2023-11-28T18:23:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f874458d640 1 --2- 172.21.15.7:0/709786276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87280047d0 0x7f8728004c50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f874458d640 1 -- 172.21.15.7:0/709786276 >> 172.21.15.7:0/709786276 conn(0x7f873c073ca0 msgr2=0x7f873c071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f874458d640 1 -- 172.21.15.7:0/709786276 shutdown_connections 2023-11-28T18:23:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f874458d640 1 -- 172.21.15.7:0/709786276 wait complete. 2023-11-28T18:23:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.680+0000 7f874458d640 1 Processor -- start 2023-11-28T18:23:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 -- start start 2023-11-28T18:23:31.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87280047d0 0x7f873c146770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f873c144c30 0x7f873c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f873c145070 0x7f873c0c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0c8ea0 con 0x7f873c144c30 2023-11-28T18:23:31.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c15a160 con 0x7f87280047d0 2023-11-28T18:23:31.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0cf180 con 0x7f873c145070 2023-11-28T18:23:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f873c144c30 0x7f873c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f873c144c30 0x7f873c0c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57084/0 (socket says 172.21.15.7:57084) 2023-11-28T18:23:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8742329640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87280047d0 0x7f873c146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 -- 172.21.15.7:0/2171079039 learned_addr learned my addr 172.21.15.7:0/2171079039 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8742b2a640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f873c145070 0x7f873c0c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 -- 172.21.15.7:0/2171079039 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f873c145070 msgr2=0x7f873c0c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f873c145070 0x7f873c0c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 -- 172.21.15.7:0/2171079039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87280047d0 msgr2=0x7f873c146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87280047d0 0x7f873c146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 -- 172.21.15.7:0/2171079039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c159bc0 con 0x7f873c144c30 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8742329640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87280047d0 0x7f873c146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8741b28640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f873c144c30 0x7f873c0c7420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f872c01c210 tx=0x7f8728005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8732ffd640 1 -- 172.21.15.7:0/2171079039 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c007440 con 0x7f873c144c30 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8732ffd640 1 -- 172.21.15.7:0/2171079039 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c0075a0 con 0x7f873c144c30 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c159e80 con 0x7f873c144c30 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f8732ffd640 1 -- 172.21.15.7:0/2171079039 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c006a20 con 0x7f873c144c30 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c0cee10 con 0x7f873c144c30 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.684+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873c0ce990 con 0x7f873c144c30 2023-11-28T18:23:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.688+0000 7f8732ffd640 1 -- 172.21.15.7:0/2171079039 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f872c006b80 con 0x7f873c144c30 2023-11-28T18:23:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.688+0000 7f8732ffd640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8720066900 0x7f8720068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.688+0000 7f8732ffd640 1 -- 172.21.15.7:0/2171079039 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f872c11e540 con 0x7f873c144c30 2023-11-28T18:23:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.688+0000 7f8742329640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8720066900 0x7f8720068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.688+0000 7f8742329640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8720066900 0x7f8720068dc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f873c074f10 tx=0x7f8738009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.692+0000 7f8732ffd640 1 -- 172.21.15.7:0/2171079039 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f873c0ce990 con 0x7f873c144c30 2023-11-28T18:23:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.848+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f873c0ce990 con 0x7f873c144c30 2023-11-28T18:23:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.848+0000 7f8732ffd640 1 -- 172.21.15.7:0/2171079039 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f873c0ce990 con 0x7f873c144c30 2023-11-28T18:23:31.855 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-11-28T18:23:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8720066900 msgr2=0x7f8720068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8720066900 0x7f8720068dc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f873c074f10 tx=0x7f8738009190 comp rx=0 tx=0).stop 2023-11-28T18:23:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f873c144c30 msgr2=0x7f873c0c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f873c144c30 0x7f873c0c7420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f872c01c210 tx=0x7f8728005800 comp rx=0 tx=0).stop 2023-11-28T18:23:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 shutdown_connections 2023-11-28T18:23:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8720066900 0x7f8720068dc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f873c145070 0x7f873c0c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f873c144c30 0x7f873c0c7420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 --2- 172.21.15.7:0/2171079039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87280047d0 0x7f873c146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 >> 172.21.15.7:0/2171079039 conn(0x7f873c073ca0 msgr2=0x7f873c0cd910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 shutdown_connections 2023-11-28T18:23:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:31.852+0000 7f874458d640 1 -- 172.21.15.7:0/2171079039 wait complete. 2023-11-28T18:23:31.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:31.974 INFO:tasks.ceph.mds.e.smithi007.stderr:2023-11-28T18:23:31.968+0000 7f7e1dce5640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-11-28T18:23:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51d172640 1 Processor -- start 2023-11-28T18:23:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51d172640 1 -- start start 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51d172640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff518151cd0 0x7ff51815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51d172640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff51806e560 0x7ff518157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51d172640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff518071e60 con 0x7ff5180701f0 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51d172640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff518070930 con 0x7ff51806e560 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51d172640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff518070660 con 0x7ff518151cd0 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff51806e560 0x7ff518157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff51806e560 0x7ff518157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57096/0 (socket says 172.21.15.7:57096) 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51659c640 1 -- 172.21.15.7:0/729222753 learned_addr learned my addr 172.21.15.7:0/729222753 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff516d9d640 1 --2- 172.21.15.7:0/729222753 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff518151cd0 0x7ff51815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51659c640 1 -- 172.21.15.7:0/729222753 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff518151cd0 msgr2=0x7ff51815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51659c640 1 --2- 172.21.15.7:0/729222753 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff518151cd0 0x7ff51815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51659c640 1 -- 172.21.15.7:0/729222753 >> v1:172.21.15.193:6789/0 conn(0x7ff5180701f0 legacy=0x7ff5180c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.016+0000 7ff51659c640 1 -- 172.21.15.7:0/729222753 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518070bf0 con 0x7ff51806e560 2023-11-28T18:23:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51659c640 1 --2- 172.21.15.7:0/729222753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff518157b90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009870 tx=0x7ff50c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a6bbd8565381df server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff515d9b640 1 -- 172.21.15.7:0/729222753 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c0ac600 con 0x7ff51806e560 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff515d9b640 1 -- 172.21.15.7:0/729222753 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c0a8050 con 0x7ff51806e560 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff515d9b640 1 -- 172.21.15.7:0/729222753 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c0ac980 con 0x7ff51806e560 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51d172640 1 -- 172.21.15.7:0/729222753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 msgr2=0x7ff518157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51d172640 1 --2- 172.21.15.7:0/729222753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff518157b90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009870 tx=0x7ff50c01c4b0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51d172640 1 -- 172.21.15.7:0/729222753 shutdown_connections 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51d172640 1 --2- 172.21.15.7:0/729222753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff518157b90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51d172640 1 --2- 172.21.15.7:0/729222753 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff518151cd0 0x7ff51815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51d172640 1 -- 172.21.15.7:0/729222753 >> 172.21.15.7:0/729222753 conn(0x7ff5180662e0 msgr2=0x7ff5180666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51d172640 1 -- 172.21.15.7:0/729222753 shutdown_connections 2023-11-28T18:23:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.020+0000 7ff51d172640 1 -- 172.21.15.7:0/729222753 wait complete. 2023-11-28T18:23:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 Processor -- start 2023-11-28T18:23:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 -- start start 2023-11-28T18:23:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff5180c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5180701f0 0x7ff5180c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff518151cd0 0x7ff5181433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180c4d10 con 0x7ff51806e560 2023-11-28T18:23:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180c39e0 con 0x7ff518151cd0 2023-11-28T18:23:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180c2ce0 con 0x7ff5180701f0 2023-11-28T18:23:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff5180c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff5180c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57112/0 (socket says 172.21.15.7:57112) 2023-11-28T18:23:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 -- 172.21.15.7:0/1548987457 learned_addr learned my addr 172.21.15.7:0/1548987457 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51759e640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff518151cd0 0x7ff5181433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51659c640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5180701f0 0x7ff5180c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 -- 172.21.15.7:0/1548987457 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5180701f0 msgr2=0x7ff5180c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5180701f0 0x7ff5180c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 -- 172.21.15.7:0/1548987457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff518151cd0 msgr2=0x7ff5181433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff518151cd0 0x7ff5181433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 -- 172.21.15.7:0/1548987457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c0094d0 con 0x7ff51806e560 2023-11-28T18:23:32.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff516d9d640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff5180c8cd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff50800d620 tx=0x7ff50800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff4ff7fe640 1 -- 172.21.15.7:0/1548987457 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800b620 con 0x7ff51806e560 2023-11-28T18:23:32.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff4ff7fe640 1 -- 172.21.15.7:0/1548987457 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50800b7a0 con 0x7ff51806e560 2023-11-28T18:23:32.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff518144210 con 0x7ff51806e560 2023-11-28T18:23:32.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff4ff7fe640 1 -- 172.21.15.7:0/1548987457 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800bb50 con 0x7ff51806e560 2023-11-28T18:23:32.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5181619b0 con 0x7ff51806e560 2023-11-28T18:23:32.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.024+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51804e890 con 0x7ff51806e560 2023-11-28T18:23:32.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.028+0000 7ff4ff7fe640 1 -- 172.21.15.7:0/1548987457 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff50800bcb0 con 0x7ff51806e560 2023-11-28T18:23:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.028+0000 7ff4ff7fe640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff504066900 0x7ff504068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.028+0000 7ff4ff7fe640 1 -- 172.21.15.7:0/1548987457 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff508084830 con 0x7ff51806e560 2023-11-28T18:23:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.028+0000 7ff51659c640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff504066900 0x7ff504068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.028+0000 7ff51659c640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff504066900 0x7ff504068dc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0ae000 tx=0x7ff50c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.032+0000 7ff4ff7fe640 1 -- 172.21.15.7:0/1548987457 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff508051b20 con 0x7ff51806e560 2023-11-28T18:23:32.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5180ca4a0 con 0x7ff51806e560 2023-11-28T18:23:32.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff4ff7fe640 1 -- 172.21.15.7:0/1548987457 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6551 (secure 0 0 0) 0x7ff508055140 con 0x7ff51806e560 2023-11-28T18:23:32.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:32.233 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:23:30.233267+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:23:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 15 2023-11-28T18:23:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff504066900 msgr2=0x7ff504068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff504066900 0x7ff504068dc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0ae000 tx=0x7ff50c0a7000 comp rx=0 tx=0).stop 2023-11-28T18:23:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 msgr2=0x7ff5180c8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff5180c8cd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff50800d620 tx=0x7ff50800dae0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 shutdown_connections 2023-11-28T18:23:32.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff504066900 0x7ff504068dc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff518151cd0 0x7ff5181433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5180701f0 0x7ff5180c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 --2- 172.21.15.7:0/1548987457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51806e560 0x7ff5180c8cd0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 >> 172.21.15.7:0/1548987457 conn(0x7ff5180662e0 msgr2=0x7ff518144e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:32.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 shutdown_connections 2023-11-28T18:23:32.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.228+0000 7ff51d172640 1 -- 172.21.15.7:0/1548987457 wait complete. 2023-11-28T18:23:32.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:32.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.392+0000 7fa1b3d1d640 1 Processor -- start 2023-11-28T18:23:32.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.392+0000 7fa1b3d1d640 1 -- start start 2023-11-28T18:23:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b3d1d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa1ac1541a0 0x7fa1ac0c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b3d1d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1ac0c7db0 0x7fa1ac0c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b3d1d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac15e190 con 0x7fa1ac0c7200 2023-11-28T18:23:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b3d1d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac15d330 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b3d1d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac15d580 con 0x7fa1ac1541a0 2023-11-28T18:23:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b12b8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1ac0c7db0 0x7fa1ac0c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b22ba640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa1ac0c7200 0x7fa1ac159f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42270/0 (socket says 172.21.15.7:42270) 2023-11-28T18:23:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b22ba640 1 -- 172.21.15.7:0/1701138170 learned_addr learned my addr 172.21.15.7:0/1701138170 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b1ab9640 1 --2- 172.21.15.7:0/1701138170 >> v2:172.21.15.193:3300/0 conn(0x7fa1ac1541a0 0x7fa1ac0c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b12b8640 1 -- 172.21.15.7:0/1701138170 >> v2:172.21.15.193:3300/0 conn(0x7fa1ac1541a0 msgr2=0x7fa1ac0c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b12b8640 1 --2- 172.21.15.7:0/1701138170 >> v2:172.21.15.193:3300/0 conn(0x7fa1ac1541a0 0x7fa1ac0c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b12b8640 1 -- 172.21.15.7:0/1701138170 >> v1:172.21.15.7:6789/0 conn(0x7fa1ac0c7200 legacy=0x7fa1ac159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b12b8640 1 -- 172.21.15.7:0/1701138170 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac15d860 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.396+0000 7fa1b0ab7640 1 -- 172.21.15.7:0/1701138170 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197293622 0 0) 0x7fa1ac15e190 con 0x7fa1ac0c7200 2023-11-28T18:23:32.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b12b8640 1 --2- 172.21.15.7:0/1701138170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac0c68e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0ccc20 tx=0x7fa19c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a78c67ab47c166d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b0ab7640 1 -- 172.21.15.7:0/1701138170 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c0a8950 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b0ab7640 1 -- 172.21.15.7:0/1701138170 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c0a8ab0 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b0ab7640 1 -- 172.21.15.7:0/1701138170 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c0a8e40 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/1701138170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 msgr2=0x7fa1ac0c68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/1701138170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac0c68e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0ccc20 tx=0x7fa19c01ec50 comp rx=0 tx=0).stop 2023-11-28T18:23:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/1701138170 shutdown_connections 2023-11-28T18:23:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/1701138170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac0c68e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/1701138170 >> v2:172.21.15.193:3300/0 conn(0x7fa1ac1541a0 0x7fa1ac0c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/1701138170 >> 172.21.15.7:0/1701138170 conn(0x7fa1ac0662e0 msgr2=0x7fa1ac0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/1701138170 shutdown_connections 2023-11-28T18:23:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/1701138170 wait complete. 2023-11-28T18:23:32.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 Processor -- start 2023-11-28T18:23:32.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.400+0000 7fa1b3d1d640 1 -- start start 2023-11-28T18:23:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1ac0c7200 0x7fa1ac148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1ac1541a0 0x7fa1ac1efc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0710a0 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac070770 con 0x7fa1ac0c7200 2023-11-28T18:23:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac070a50 con 0x7fa1ac1541a0 2023-11-28T18:23:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac148bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57122/0 (socket says 172.21.15.7:57122) 2023-11-28T18:23:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 -- 172.21.15.7:0/3665839144 learned_addr learned my addr 172.21.15.7:0/3665839144 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b1ab9640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1ac0c7200 0x7fa1ac148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b22ba640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1ac1541a0 0x7fa1ac1efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 -- 172.21.15.7:0/3665839144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1ac1541a0 msgr2=0x7fa1ac1efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1ac1541a0 0x7fa1ac1efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 -- 172.21.15.7:0/3665839144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1ac0c7200 msgr2=0x7fa1ac148670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1ac0c7200 0x7fa1ac148670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 -- 172.21.15.7:0/3665839144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c009420 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b12b8640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac148bb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa19c0b3040 tx=0x7fa19c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1a27fc640 1 -- 172.21.15.7:0/3665839144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c0b9630 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1a27fc640 1 -- 172.21.15.7:0/3665839144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c0b97b0 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1a27fc640 1 -- 172.21.15.7:0/3665839144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c0b9b60 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac070d30 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac150780 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac070ec0 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.404+0000 7fa1a27fc640 1 -- 172.21.15.7:0/3665839144 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa19c0b7040 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.408+0000 7fa1a27fc640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1900668d0 0x7fa190068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.408+0000 7fa1a27fc640 1 -- 172.21.15.7:0/3665839144 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa19c122440 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.408+0000 7fa1b1ab9640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1900668d0 0x7fa190068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.408+0000 7fa1b1ab9640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1900668d0 0x7fa190068d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8007a60 tx=0x7fa1a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.412+0000 7fa1a27fc640 1 -- 172.21.15.7:0/3665839144 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa19c0ef700 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.568+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1ac0711e0 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1a27fc640 1 -- 172.21.15.7:0/3665839144 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fa1ac0711e0 con 0x7fa1ac0c7db0 2023-11-28T18:23:32.575 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:32.575 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1900668d0 msgr2=0x7fa190068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1900668d0 0x7fa190068d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8007a60 tx=0x7fa1a8008040 comp rx=0 tx=0).stop 2023-11-28T18:23:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 msgr2=0x7fa1ac148bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac148bb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa19c0b3040 tx=0x7fa19c01b9f0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 shutdown_connections 2023-11-28T18:23:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1900668d0 0x7fa190068d90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1ac1541a0 0x7fa1ac1efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1ac0c7db0 0x7fa1ac148bb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 --2- 172.21.15.7:0/3665839144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1ac0c7200 0x7fa1ac148670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 >> 172.21.15.7:0/3665839144 conn(0x7fa1ac0662e0 msgr2=0x7fa1ac0c0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 shutdown_connections 2023-11-28T18:23:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.572+0000 7fa1b3d1d640 1 -- 172.21.15.7:0/3665839144 wait complete. 2023-11-28T18:23:32.596 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T18:23:05.706694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T18:23:07.048910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:23:32.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:23:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.736+0000 7f81c7bd1640 1 Processor -- start 2023-11-28T18:23:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.736+0000 7f81c7bd1640 1 -- start start 2023-11-28T18:23:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c7bd1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81c0154210 0x7f81c00cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c7bd1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c015ad50 con 0x7f81c00c6ee0 2023-11-28T18:23:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c7bd1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c00c6ca0 con 0x7f81c00cbf50 2023-11-28T18:23:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c7bd1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c00c9090 con 0x7f81c0154210 2023-11-28T18:23:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c596d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81c0154210 0x7f81c00cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c596d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81c0154210 0x7f81c00cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57124/0 (socket says 172.21.15.7:57124) 2023-11-28T18:23:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c596d640 1 -- 172.21.15.7:0/4069993447 learned_addr learned my addr 172.21.15.7:0/4069993447 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c596d640 1 -- 172.21.15.7:0/4069993447 >> v1:172.21.15.193:6789/0 conn(0x7f81c00cbf50 legacy=0x7f81c01648d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c596d640 1 -- 172.21.15.7:0/4069993447 >> v1:172.21.15.7:6789/0 conn(0x7f81c00c6ee0 legacy=0x7f81c00ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c596d640 1 -- 172.21.15.7:0/4069993447 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c00cb9a0 con 0x7f81c0154210 2023-11-28T18:23:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.740+0000 7f81c496b640 1 -- 172.21.15.7:0/4069993447 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741558872 0 0) 0x7f81c015ad50 con 0x7f81c00c6ee0 2023-11-28T18:23:32.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c596d640 1 --2- 172.21.15.7:0/4069993447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c0154210 0x7f81c00cb380 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009870 tx=0x7f81bc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7a4a74c3747a45 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c496b640 1 -- 172.21.15.7:0/4069993447 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0aa670 con 0x7f81c0154210 2023-11-28T18:23:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c496b640 1 -- 172.21.15.7:0/4069993447 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc0aa7d0 con 0x7f81c0154210 2023-11-28T18:23:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c496b640 1 -- 172.21.15.7:0/4069993447 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0aab60 con 0x7f81c0154210 2023-11-28T18:23:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 -- 172.21.15.7:0/4069993447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c0154210 msgr2=0x7f81c00cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 --2- 172.21.15.7:0/4069993447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c0154210 0x7f81c00cb380 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009870 tx=0x7f81bc01c510 comp rx=0 tx=0).stop 2023-11-28T18:23:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 -- 172.21.15.7:0/4069993447 shutdown_connections 2023-11-28T18:23:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 --2- 172.21.15.7:0/4069993447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c0154210 0x7f81c00cb380 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 -- 172.21.15.7:0/4069993447 >> 172.21.15.7:0/4069993447 conn(0x7f81c00662e0 msgr2=0x7f81c00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 -- 172.21.15.7:0/4069993447 shutdown_connections 2023-11-28T18:23:32.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 -- 172.21.15.7:0/4069993447 wait complete. 2023-11-28T18:23:32.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 Processor -- start 2023-11-28T18:23:32.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.744+0000 7f81c7bd1640 1 -- start start 2023-11-28T18:23:32.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c00c6ee0 0x7f81c00c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81c00cbf50 0x7f81c0144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f81c0154210 0x7f81c00c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c01599e0 con 0x7f81c00c6ee0 2023-11-28T18:23:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00cd730 con 0x7f81c00cbf50 2023-11-28T18:23:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00c5ca0 con 0x7f81c0154210 2023-11-28T18:23:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c00c6ee0 0x7f81c00c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c00c6ee0 0x7f81c00c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57132/0 (socket says 172.21.15.7:57132) 2023-11-28T18:23:32.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 -- 172.21.15.7:0/1781740572 learned_addr learned my addr 172.21.15.7:0/1781740572 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:32.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c516c640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81c00cbf50 0x7f81c0144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c616e640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f81c0154210 0x7f81c00c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 -- 172.21.15.7:0/1781740572 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f81c0154210 msgr2=0x7f81c00c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f81c0154210 0x7f81c00c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 -- 172.21.15.7:0/1781740572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81c00cbf50 msgr2=0x7f81c0144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81c00cbf50 0x7f81c0144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 -- 172.21.15.7:0/1781740572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc0094d0 con 0x7f81c00c6ee0 2023-11-28T18:23:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c596d640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c00c6ee0 0x7f81c00c5510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0ac040 tx=0x7f81bc01ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81b67fc640 1 -- 172.21.15.7:0/1781740572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0b38c0 con 0x7f81c00c6ee0 2023-11-28T18:23:32.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81b67fc640 1 -- 172.21.15.7:0/1781740572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc0b3a20 con 0x7f81c00c6ee0 2023-11-28T18:23:32.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c0158b00 con 0x7f81c00c6ee0 2023-11-28T18:23:32.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81b67fc640 1 -- 172.21.15.7:0/1781740572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0b3db0 con 0x7f81c00c6ee0 2023-11-28T18:23:32.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c0158d50 con 0x7f81c00c6ee0 2023-11-28T18:23:32.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c00c3d90 con 0x7f81c00c6ee0 2023-11-28T18:23:32.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.748+0000 7f81b67fc640 1 -- 172.21.15.7:0/1781740572 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f81bc0c6390 con 0x7f81c00c6ee0 2023-11-28T18:23:32.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.752+0000 7f81b67fc640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f81a4066920 0x7f81a4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:32.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.752+0000 7f81b67fc640 1 -- 172.21.15.7:0/1781740572 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81bc11f060 con 0x7f81c00c6ee0 2023-11-28T18:23:32.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.752+0000 7f81c516c640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f81a4066920 0x7f81a4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:32.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.752+0000 7f81c516c640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f81a4066920 0x7f81a4068de0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f81c0067ab0 tx=0x7f81b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:32.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.756+0000 7f81b67fc640 1 -- 172.21.15.7:0/1781740572 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f81c00c3d90 con 0x7f81c00c6ee0 2023-11-28T18:23:32.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f81c00c3d90 con 0x7f81c00c6ee0 2023-11-28T18:23:32.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81b67fc640 1 -- 172.21.15.7:0/1781740572 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f81c00c3d90 con 0x7f81c00c6ee0 2023-11-28T18:23:32.917 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:23:32.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f81a4066920 msgr2=0x7f81a4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f81a4066920 0x7f81a4068de0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f81c0067ab0 tx=0x7f81b0009040 comp rx=0 tx=0).stop 2023-11-28T18:23:32.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c00c6ee0 msgr2=0x7f81c00c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:32.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c00c6ee0 0x7f81c00c5510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0ac040 tx=0x7f81bc01ad50 comp rx=0 tx=0).stop 2023-11-28T18:23:32.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 shutdown_connections 2023-11-28T18:23:32.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f81a4066920 0x7f81a4068de0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f81c0154210 0x7f81c00c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81c00cbf50 0x7f81c0144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 --2- 172.21.15.7:0/1781740572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c00c6ee0 0x7f81c00c5510 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:32.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 >> 172.21.15.7:0/1781740572 conn(0x7f81c00662e0 msgr2=0x7f81c0151a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:32.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 shutdown_connections 2023-11-28T18:23:32.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:32.912+0000 7f81c7bd1640 1 -- 172.21.15.7:0/1781740572 wait complete. 2023-11-28T18:23:32.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T18:23:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb569b8640 1 Processor -- start 2023-11-28T18:23:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb569b8640 1 -- start start 2023-11-28T18:23:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb569b8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb501540d0 0x7fdb50072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb569b8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5015ae90 con 0x7fdb5015df80 2023-11-28T18:23:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb569b8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5015a370 con 0x7fdb5015db70 2023-11-28T18:23:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb569b8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5015a610 con 0x7fdb501540d0 2023-11-28T18:23:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb4ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb501540d0 0x7fdb50072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb54f55640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdb5015df80 0x7fdb501648d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42302/0 (socket says 172.21.15.7:42302) 2023-11-28T18:23:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.080+0000 7fdb54f55640 1 -- 172.21.15.7:0/4200314916 learned_addr learned my addr 172.21.15.7:0/4200314916 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb4ffff640 1 -- 172.21.15.7:0/4200314916 >> v1:172.21.15.193:6789/0 conn(0x7fdb5015db70 legacy=0x7fdb500cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb4ffff640 1 -- 172.21.15.7:0/4200314916 >> v1:172.21.15.7:6789/0 conn(0x7fdb5015df80 legacy=0x7fdb501648d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb4ffff640 1 -- 172.21.15.7:0/4200314916 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5015a8d0 con 0x7fdb501540d0 2023-11-28T18:23:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb4effd640 1 -- 172.21.15.7:0/4200314916 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468374718 0 0) 0x7fdb5015ae90 con 0x7fdb5015df80 2023-11-28T18:23:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb4ffff640 1 --2- 172.21.15.7:0/4200314916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb501540d0 0x7fdb50072940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb400097c0 tx=0x7fdb4001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3783ed648f6e45 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb4effd640 1 -- 172.21.15.7:0/4200314916 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb400ab600 con 0x7fdb501540d0 2023-11-28T18:23:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb4effd640 1 -- 172.21.15.7:0/4200314916 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb400a7050 con 0x7fdb501540d0 2023-11-28T18:23:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb4effd640 1 -- 172.21.15.7:0/4200314916 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb400ab980 con 0x7fdb501540d0 2023-11-28T18:23:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb569b8640 1 -- 172.21.15.7:0/4200314916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb501540d0 msgr2=0x7fdb50072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb569b8640 1 --2- 172.21.15.7:0/4200314916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb501540d0 0x7fdb50072940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb400097c0 tx=0x7fdb4001c440 comp rx=0 tx=0).stop 2023-11-28T18:23:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb569b8640 1 -- 172.21.15.7:0/4200314916 shutdown_connections 2023-11-28T18:23:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb569b8640 1 --2- 172.21.15.7:0/4200314916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb501540d0 0x7fdb50072940 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb569b8640 1 -- 172.21.15.7:0/4200314916 >> 172.21.15.7:0/4200314916 conn(0x7fdb500662e0 msgr2=0x7fdb500666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb569b8640 1 -- 172.21.15.7:0/4200314916 shutdown_connections 2023-11-28T18:23:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.084+0000 7fdb569b8640 1 -- 172.21.15.7:0/4200314916 wait complete. 2023-11-28T18:23:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb569b8640 1 Processor -- start 2023-11-28T18:23:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb569b8640 1 -- start start 2023-11-28T18:23:33.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb569b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb501540d0 0x7fdb50155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb569b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb5015db70 0x7fdb500cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb569b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdb5015df80 0x7fdb501f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb569b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb50156700 con 0x7fdb5015db70 2023-11-28T18:23:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb569b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb501f3680 con 0x7fdb501540d0 2023-11-28T18:23:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb569b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb501f3f20 con 0x7fdb5015df80 2023-11-28T18:23:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb5015db70 0x7fdb500cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb5015db70 0x7fdb500cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57162/0 (socket says 172.21.15.7:57162) 2023-11-28T18:23:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 -- 172.21.15.7:0/2390418567 learned_addr learned my addr 172.21.15.7:0/2390418567 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4ffff640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb501540d0 0x7fdb50155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb54f55640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdb5015df80 0x7fdb501f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 -- 172.21.15.7:0/2390418567 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdb5015df80 msgr2=0x7fdb501f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdb5015df80 0x7fdb501f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 -- 172.21.15.7:0/2390418567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb501540d0 msgr2=0x7fdb50155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb501540d0 0x7fdb50155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 -- 172.21.15.7:0/2390418567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb40009420 con 0x7fdb5015db70 2023-11-28T18:23:33.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.088+0000 7fdb4f7fe640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb5015db70 0x7fdb500cc7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400c480 tx=0x7fdb4400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb4cff9640 1 -- 172.21.15.7:0/2390418567 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb44012830 con 0x7fdb5015db70 2023-11-28T18:23:33.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb4cff9640 1 -- 172.21.15.7:0/2390418567 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb440129b0 con 0x7fdb5015db70 2023-11-28T18:23:33.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb4cff9640 1 -- 172.21.15.7:0/2390418567 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb44012d60 con 0x7fdb5015db70 2023-11-28T18:23:33.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb501578f0 con 0x7fdb5015db70 2023-11-28T18:23:33.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb501560f0 con 0x7fdb5015db70 2023-11-28T18:23:33.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5004e890 con 0x7fdb5015db70 2023-11-28T18:23:33.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb4cff9640 1 -- 172.21.15.7:0/2390418567 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fdb4401c460 con 0x7fdb5015db70 2023-11-28T18:23:33.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb4cff9640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdb3c0668d0 0x7fdb3c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb4cff9640 1 -- 172.21.15.7:0/2390418567 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb44084290 con 0x7fdb5015db70 2023-11-28T18:23:33.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.092+0000 7fdb4ffff640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdb3c0668d0 0x7fdb3c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.096+0000 7fdb4ffff640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdb3c0668d0 0x7fdb3c068d90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdb400ad000 tx=0x7fdb400a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.096+0000 7fdb4cff9640 1 -- 172.21.15.7:0/2390418567 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdb440514e0 con 0x7fdb5015db70 2023-11-28T18:23:33.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdb501f4060 con 0x7fdb5015db70 2023-11-28T18:23:33.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb4cff9640 1 -- 172.21.15.7:0/2390418567 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdb44054b00 con 0x7fdb5015db70 2023-11-28T18:23:33.261 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-11-28T18:23:33.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdb3c0668d0 msgr2=0x7fdb3c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdb3c0668d0 0x7fdb3c068d90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdb400ad000 tx=0x7fdb400a6000 comp rx=0 tx=0).stop 2023-11-28T18:23:33.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb5015db70 msgr2=0x7fdb500cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb5015db70 0x7fdb500cc7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400c480 tx=0x7fdb4400c940 comp rx=0 tx=0).stop 2023-11-28T18:23:33.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 shutdown_connections 2023-11-28T18:23:33.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdb3c0668d0 0x7fdb3c068d90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdb5015df80 0x7fdb501f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb5015db70 0x7fdb500cc7a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 --2- 172.21.15.7:0/2390418567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb501540d0 0x7fdb50155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 >> 172.21.15.7:0/2390418567 conn(0x7fdb500662e0 msgr2=0x7fdb50158560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:33.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 shutdown_connections 2023-11-28T18:23:33.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.256+0000 7fdb569b8640 1 -- 172.21.15.7:0/2390418567 wait complete. 2023-11-28T18:23:33.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T18:23:33.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa935479640 1 Processor -- start 2023-11-28T18:23:33.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa935479640 1 -- start start 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa935479640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa93015a5c0 0x7fa9300c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa935479640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 0x7fa93015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa935479640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9300c8530 con 0x7fa9300c3ba0 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa935479640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9300c7680 con 0x7fa93015a5c0 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa935479640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300c7920 con 0x7fa93015a9a0 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa92effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa93015a5c0 0x7fa9300c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa92effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa93015a5c0 0x7fa9300c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57168/0 (socket says 172.21.15.7:57168) 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa92effd640 1 -- 172.21.15.7:0/1336321189 learned_addr learned my addr 172.21.15.7:0/1336321189 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa927fff640 1 --2- 172.21.15.7:0/1336321189 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 0x7fa93015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa92effd640 1 -- 172.21.15.7:0/1336321189 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 msgr2=0x7fa93015adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:33.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa92effd640 1 --2- 172.21.15.7:0/1336321189 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 0x7fa93015adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa92effd640 1 -- 172.21.15.7:0/1336321189 >> v1:172.21.15.193:6789/0 conn(0x7fa9300c3ba0 legacy=0x7fa9300c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:33.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.424+0000 7fa92effd640 1 -- 172.21.15.7:0/1336321189 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9300c7c00 con 0x7fa93015a5c0 2023-11-28T18:23:33.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa92effd640 1 --2- 172.21.15.7:0/1336321189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9300c6890 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa920009870 tx=0x7fa92001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832820ecf274dbf0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa92e7fc640 1 -- 172.21.15.7:0/1336321189 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9200a89b0 con 0x7fa93015a5c0 2023-11-28T18:23:33.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa92e7fc640 1 -- 172.21.15.7:0/1336321189 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9200a8b10 con 0x7fa93015a5c0 2023-11-28T18:23:33.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa92e7fc640 1 -- 172.21.15.7:0/1336321189 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9200a8ea0 con 0x7fa93015a5c0 2023-11-28T18:23:33.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa935479640 1 -- 172.21.15.7:0/1336321189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 msgr2=0x7fa9300c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa935479640 1 --2- 172.21.15.7:0/1336321189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9300c6890 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa920009870 tx=0x7fa92001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa935479640 1 -- 172.21.15.7:0/1336321189 shutdown_connections 2023-11-28T18:23:33.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa935479640 1 --2- 172.21.15.7:0/1336321189 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 0x7fa93015adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa935479640 1 --2- 172.21.15.7:0/1336321189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9300c6890 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa935479640 1 -- 172.21.15.7:0/1336321189 >> 172.21.15.7:0/1336321189 conn(0x7fa930073f00 msgr2=0x7fa9300662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:33.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa935479640 1 -- 172.21.15.7:0/1336321189 shutdown_connections 2023-11-28T18:23:33.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.428+0000 7fa935479640 1 -- 172.21.15.7:0/1336321189 wait complete. 2023-11-28T18:23:33.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 Processor -- start 2023-11-28T18:23:33.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 -- start start 2023-11-28T18:23:33.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9300c3ba0 0x7fa9300cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9301e3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 0x7fa9301e0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300cb7d0 con 0x7fa93015a5c0 2023-11-28T18:23:33.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300cbbf0 con 0x7fa9300c3ba0 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300cc140 con 0x7fa93015a9a0 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9301e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9301e3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57176/0 (socket says 172.21.15.7:57176) 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 -- 172.21.15.7:0/2751265085 learned_addr learned my addr 172.21.15.7:0/2751265085 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa92effd640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9300c3ba0 0x7fa9300cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa92f7fe640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 0x7fa9301e0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 -- 172.21.15.7:0/2751265085 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 msgr2=0x7fa9301e0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 0x7fa9301e0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 -- 172.21.15.7:0/2751265085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9300c3ba0 msgr2=0x7fa9300cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9300c3ba0 0x7fa9300cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 -- 172.21.15.7:0/2751265085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9200094d0 con 0x7fa93015a5c0 2023-11-28T18:23:33.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa927fff640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9301e3810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa91800d650 tx=0x7fa91800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa9277fe640 1 -- 172.21.15.7:0/2751265085 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91800b6c0 con 0x7fa93015a5c0 2023-11-28T18:23:33.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa9277fe640 1 -- 172.21.15.7:0/2751265085 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91800b820 con 0x7fa93015a5c0 2023-11-28T18:23:33.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9300c28f0 con 0x7fa93015a5c0 2023-11-28T18:23:33.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa9277fe640 1 -- 172.21.15.7:0/2751265085 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91800bbb0 con 0x7fa93015a5c0 2023-11-28T18:23:33.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.432+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9301e0a10 con 0x7fa93015a5c0 2023-11-28T18:23:33.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.436+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa930070c00 con 0x7fa93015a5c0 2023-11-28T18:23:33.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.436+0000 7fa9277fe640 1 -- 172.21.15.7:0/2751265085 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa91800bd10 con 0x7fa93015a5c0 2023-11-28T18:23:33.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.436+0000 7fa9277fe640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9100668d0 0x7fa910068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.436+0000 7fa9277fe640 1 -- 172.21.15.7:0/2751265085 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa918084870 con 0x7fa93015a5c0 2023-11-28T18:23:33.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.436+0000 7fa92effd640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9100668d0 0x7fa910068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.436+0000 7fa92effd640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9100668d0 0x7fa910068d90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa9200b2000 tx=0x7fa920009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.440+0000 7fa9277fe640 1 -- 172.21.15.7:0/2751265085 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa918051a30 con 0x7fa93015a5c0 2023-11-28T18:23:33.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa93004e890 con 0x7fa93015a5c0 2023-11-28T18:23:33.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa9277fe640 1 -- 172.21.15.7:0/2751265085 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa918007380 con 0x7fa93015a5c0 2023-11-28T18:23:33.608 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-11-28T18:23:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9100668d0 msgr2=0x7fa910068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9100668d0 0x7fa910068d90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa9200b2000 tx=0x7fa920009040 comp rx=0 tx=0).stop 2023-11-28T18:23:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 msgr2=0x7fa9301e3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9301e3810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa91800d650 tx=0x7fa91800db10 comp rx=0 tx=0).stop 2023-11-28T18:23:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 shutdown_connections 2023-11-28T18:23:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9100668d0 0x7fa910068d90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa93015a9a0 0x7fa9301e0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa93015a5c0 0x7fa9301e3810 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 --2- 172.21.15.7:0/2751265085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9300c3ba0 0x7fa9300cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 >> 172.21.15.7:0/2751265085 conn(0x7fa930073f00 msgr2=0x7fa9300c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 shutdown_connections 2023-11-28T18:23:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.604+0000 7fa935479640 1 -- 172.21.15.7:0/2751265085 wait complete. 2023-11-28T18:23:33.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:33.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.768+0000 7f054092a640 1 Processor -- start 2023-11-28T18:23:33.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.768+0000 7f054092a640 1 -- start start 2023-11-28T18:23:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f054092a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f053c15bd00 0x7f053c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f054092a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c15c110 con 0x7f053c151cd0 2023-11-28T18:23:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f054092a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c071e60 con 0x7f053c06e560 2023-11-28T18:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f054092a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c070930 con 0x7f053c15bd00 2023-11-28T18:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f053a59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f053c151cd0 0x7f053c0c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44810/0 (socket says 172.21.15.7:44810) 2023-11-28T18:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f053a59c640 1 -- 172.21.15.7:0/291667808 learned_addr learned my addr 172.21.15.7:0/291667808 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f053ad9d640 1 --2- 172.21.15.7:0/291667808 >> v2:172.21.15.193:3300/0 conn(0x7f053c15bd00 0x7f053c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f0539d9b640 1 -- 172.21.15.7:0/291667808 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438800271 0 0) 0x7f053c15c110 con 0x7f053c151cd0 2023-11-28T18:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f0539d9b640 1 -- 172.21.15.7:0/291667808 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0528003530 con 0x7f053c151cd0 2023-11-28T18:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f0539d9b640 1 -- 172.21.15.7:0/291667808 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648688859 0 0) 0x7f053c071e60 con 0x7f053c06e560 2023-11-28T18:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f0539d9b640 1 -- 172.21.15.7:0/291667808 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053c15c110 con 0x7f053c06e560 2023-11-28T18:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f053ad9d640 1 -- 172.21.15.7:0/291667808 >> v1:172.21.15.193:6789/0 conn(0x7f053c06e560 legacy=0x7f053c160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f053ad9d640 1 -- 172.21.15.7:0/291667808 >> v1:172.21.15.7:6789/0 conn(0x7f053c151cd0 legacy=0x7f053c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.772+0000 7f053ad9d640 1 -- 172.21.15.7:0/291667808 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c070bf0 con 0x7f053c15bd00 2023-11-28T18:23:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f053ad9d640 1 --2- 172.21.15.7:0/291667808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c15bd00 0x7f053c157b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0534009d90 tx=0x7f053401c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c71057c8ec02a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f0539d9b640 1 -- 172.21.15.7:0/291667808 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05340a5900 con 0x7f053c15bd00 2023-11-28T18:23:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f0539d9b640 1 -- 172.21.15.7:0/291667808 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05340a5a60 con 0x7f053c15bd00 2023-11-28T18:23:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f0539d9b640 1 -- 172.21.15.7:0/291667808 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05340a5df0 con 0x7f053c15bd00 2023-11-28T18:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 -- 172.21.15.7:0/291667808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c15bd00 msgr2=0x7f053c157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 --2- 172.21.15.7:0/291667808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c15bd00 0x7f053c157b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0534009d90 tx=0x7f053401c950 comp rx=0 tx=0).stop 2023-11-28T18:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 -- 172.21.15.7:0/291667808 shutdown_connections 2023-11-28T18:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 --2- 172.21.15.7:0/291667808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c15bd00 0x7f053c157b90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 -- 172.21.15.7:0/291667808 >> 172.21.15.7:0/291667808 conn(0x7f053c0662e0 msgr2=0x7f053c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 -- 172.21.15.7:0/291667808 shutdown_connections 2023-11-28T18:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 -- 172.21.15.7:0/291667808 wait complete. 2023-11-28T18:23:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 Processor -- start 2023-11-28T18:23:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 -- start start 2023-11-28T18:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c06e560 0x7f053c156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.776+0000 7f054092a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053c151cd0 0x7f053c0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f054092a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c15bd00 0x7f053c0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f054092a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c157bf0 con 0x7f053c151cd0 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f054092a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c0c78c0 con 0x7f053c06e560 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f054092a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c157630 con 0x7f053c15bd00 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053c151cd0 0x7f053c0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053c151cd0 0x7f053c0c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40014/0 (socket says 172.21.15.7:40014) 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 -- 172.21.15.7:0/1130754178 learned_addr learned my addr 172.21.15.7:0/1130754178 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f053a59c640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c06e560 0x7f053c156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f053ad9d640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c15bd00 0x7f053c0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 -- 172.21.15.7:0/1130754178 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c15bd00 msgr2=0x7f053c0c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c15bd00 0x7f053c0c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 -- 172.21.15.7:0/1130754178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c06e560 msgr2=0x7f053c156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c06e560 0x7f053c156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 -- 172.21.15.7:0/1130754178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534009a70 con 0x7f053c151cd0 2023-11-28T18:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0533fff640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053c151cd0 0x7f053c0c8800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f052400c490 tx=0x7f052400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0532ffd640 1 -- 172.21.15.7:0/1130754178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05240052d0 con 0x7f053c151cd0 2023-11-28T18:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0532ffd640 1 -- 172.21.15.7:0/1130754178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0524005430 con 0x7f053c151cd0 2023-11-28T18:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0532ffd640 1 -- 172.21.15.7:0/1130754178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524012870 con 0x7f053c151cd0 2023-11-28T18:23:33.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c157910 con 0x7f053c151cd0 2023-11-28T18:23:33.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c155c50 con 0x7f053c151cd0 2023-11-28T18:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c0c7a10 con 0x7f053c151cd0 2023-11-28T18:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.780+0000 7f0532ffd640 1 -- 172.21.15.7:0/1130754178 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f05240129d0 con 0x7f053c151cd0 2023-11-28T18:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.784+0000 7f0532ffd640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c0668d0 0x7f051c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.784+0000 7f0532ffd640 1 -- 172.21.15.7:0/1130754178 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0524083e90 con 0x7f053c151cd0 2023-11-28T18:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.784+0000 7f053a59c640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c0668d0 0x7f051c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.784+0000 7f053a59c640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c0668d0 0x7f051c068d90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f052c0011f0 tx=0x7f052c007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:33.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.788+0000 7f0532ffd640 1 -- 172.21.15.7:0/1130754178 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f053c0c7a10 con 0x7f053c151cd0 2023-11-28T18:23:33.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.980+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f053c0c7a10 con 0x7f053c151cd0 2023-11-28T18:23:33.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.980+0000 7f0532ffd640 1 -- 172.21.15.7:0/1130754178 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f053c0c7a10 con 0x7f053c151cd0 2023-11-28T18:23:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:33.986 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:23:30.233267+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:23:33.986 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 15 2023-11-28T18:23:33.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c0668d0 msgr2=0x7f051c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c0668d0 0x7f051c068d90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f052c0011f0 tx=0x7f052c007d40 comp rx=0 tx=0).stop 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053c151cd0 msgr2=0x7f053c0c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053c151cd0 0x7f053c0c8800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f052400c490 tx=0x7f052400c950 comp rx=0 tx=0).stop 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 shutdown_connections 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c0668d0 0x7f051c068d90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c15bd00 0x7f053c0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053c151cd0 0x7f053c0c8800 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 --2- 172.21.15.7:0/1130754178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c06e560 0x7f053c156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 >> 172.21.15.7:0/1130754178 conn(0x7f053c0662e0 msgr2=0x7f053c14c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 shutdown_connections 2023-11-28T18:23:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:33.984+0000 7f054092a640 1 -- 172.21.15.7:0/1130754178 wait complete. 2023-11-28T18:23:34.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:34.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.144+0000 7fdf8149a640 1 Processor -- start 2023-11-28T18:23:34.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.144+0000 7fdf8149a640 1 -- start start 2023-11-28T18:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf8149a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fdf7c0c24f0 0x7fdf7c1575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf8149a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf7c0c7980 0x7fdf7c0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf8149a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c0c6140 con 0x7fdf7c150b30 2023-11-28T18:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf8149a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c0c72b0 con 0x7fdf7c0c7980 2023-11-28T18:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf8149a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c070c00 con 0x7fdf7c0c24f0 2023-11-28T18:23:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf7b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf7c0c7980 0x7fdf7c0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf7b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf7c0c7980 0x7fdf7c0c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40022/0 (socket says 172.21.15.7:40022) 2023-11-28T18:23:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf7b7fe640 1 -- 172.21.15.7:0/4206351792 learned_addr learned my addr 172.21.15.7:0/4206351792 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf73fff640 1 --2- 172.21.15.7:0/4206351792 >> v2:172.21.15.193:3300/0 conn(0x7fdf7c0c24f0 0x7fdf7c1575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf7b7fe640 1 -- 172.21.15.7:0/4206351792 >> v2:172.21.15.193:3300/0 conn(0x7fdf7c0c24f0 msgr2=0x7fdf7c1575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:34.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf7b7fe640 1 --2- 172.21.15.7:0/4206351792 >> v2:172.21.15.193:3300/0 conn(0x7fdf7c0c24f0 0x7fdf7c1575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf7b7fe640 1 -- 172.21.15.7:0/4206351792 >> v1:172.21.15.193:6789/0 conn(0x7fdf7c150b30 legacy=0x7fdf7c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.148+0000 7fdf7b7fe640 1 -- 172.21.15.7:0/4206351792 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c157b00 con 0x7fdf7c0c7980 2023-11-28T18:23:34.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf7b7fe640 1 --2- 172.21.15.7:0/4206351792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c0c7980 0x7fdf7c0c7dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c06fee0 tx=0x7fdf740213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32ce46503e0312 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:34.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf7a7fc640 1 -- 172.21.15.7:0/4206351792 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7400b2d0 con 0x7fdf7c0c7980 2023-11-28T18:23:34.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf7a7fc640 1 -- 172.21.15.7:0/4206351792 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf740ad050 con 0x7fdf7c0c7980 2023-11-28T18:23:34.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf7a7fc640 1 -- 172.21.15.7:0/4206351792 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74009820 con 0x7fdf7c0c7980 2023-11-28T18:23:34.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 -- 172.21.15.7:0/4206351792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c0c7980 msgr2=0x7fdf7c0c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4206351792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c0c7980 0x7fdf7c0c7dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c06fee0 tx=0x7fdf740213c0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 -- 172.21.15.7:0/4206351792 shutdown_connections 2023-11-28T18:23:34.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4206351792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c0c7980 0x7fdf7c0c7dd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4206351792 >> v2:172.21.15.193:3300/0 conn(0x7fdf7c0c24f0 0x7fdf7c1575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 -- 172.21.15.7:0/4206351792 >> 172.21.15.7:0/4206351792 conn(0x7fdf7c0662e0 msgr2=0x7fdf7c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:34.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 -- 172.21.15.7:0/4206351792 shutdown_connections 2023-11-28T18:23:34.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 -- 172.21.15.7:0/4206351792 wait complete. 2023-11-28T18:23:34.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 Processor -- start 2023-11-28T18:23:34.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.152+0000 7fdf8149a640 1 -- start start 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdf7c0c24f0 0x7fdf7c072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c0c7980 0x7fdf7c0713e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c150b30 0x7fdf7c14d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c14d320 con 0x7fdf7c150b30 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c14b6e0 con 0x7fdf7c0c7980 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c14b9c0 con 0x7fdf7c0c24f0 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c150b30 0x7fdf7c14d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c150b30 0x7fdf7c14d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40024/0 (socket says 172.21.15.7:40024) 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 -- 172.21.15.7:0/4278971813 learned_addr learned my addr 172.21.15.7:0/4278971813 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:34.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7affd640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdf7c0c24f0 0x7fdf7c072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf73fff640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c0c7980 0x7fdf7c0713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 -- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdf7c0c24f0 msgr2=0x7fdf7c072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdf7c0c24f0 0x7fdf7c072370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 -- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c0c7980 msgr2=0x7fdf7c0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c0c7980 0x7fdf7c0713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 -- 172.21.15.7:0/4278971813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7400bd40 con 0x7fdf7c150b30 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf73fff640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c0c7980 0x7fdf7c0713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf7b7fe640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c150b30 0x7fdf7c14d8c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdf7401dfd0 tx=0x7fdf7401e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf737fe640 1 -- 172.21.15.7:0/4278971813 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf740c0070 con 0x7fdf7c150b30 2023-11-28T18:23:34.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf737fe640 1 -- 172.21.15.7:0/4278971813 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf74009810 con 0x7fdf7c150b30 2023-11-28T18:23:34.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf737fe640 1 -- 172.21.15.7:0/4278971813 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74009bc0 con 0x7fdf7c150b30 2023-11-28T18:23:34.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c14bca0 con 0x7fdf7c150b30 2023-11-28T18:23:34.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c073570 con 0x7fdf7c150b30 2023-11-28T18:23:34.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c072090 con 0x7fdf7c150b30 2023-11-28T18:23:34.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.156+0000 7fdf737fe640 1 -- 172.21.15.7:0/4278971813 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fdf74009d20 con 0x7fdf7c150b30 2023-11-28T18:23:34.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.160+0000 7fdf737fe640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdf5c0668d0 0x7fdf5c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.160+0000 7fdf737fe640 1 -- 172.21.15.7:0/4278971813 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf74122680 con 0x7fdf7c150b30 2023-11-28T18:23:34.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.160+0000 7fdf7affd640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdf5c0668d0 0x7fdf5c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.160+0000 7fdf7affd640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdf5c0668d0 0x7fdf5c068d90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c007fd0 tx=0x7fdf6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:34.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.164+0000 7fdf737fe640 1 -- 172.21.15.7:0/4278971813 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdf740f3340 con 0x7fdf7c150b30 2023-11-28T18:23:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.324+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf7c04e890 con 0x7fdf7c150b30 2023-11-28T18:23:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.324+0000 7fdf737fe640 1 -- 172.21.15.7:0/4278971813 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fdf740f2ed0 con 0x7fdf7c150b30 2023-11-28T18:23:34.330 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:34.330 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:34.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdf5c0668d0 msgr2=0x7fdf5c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdf5c0668d0 0x7fdf5c068d90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c007fd0 tx=0x7fdf6c009040 comp rx=0 tx=0).stop 2023-11-28T18:23:34.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c150b30 msgr2=0x7fdf7c14d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c150b30 0x7fdf7c14d8c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdf7401dfd0 tx=0x7fdf7401e160 comp rx=0 tx=0).stop 2023-11-28T18:23:34.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 shutdown_connections 2023-11-28T18:23:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdf5c0668d0 0x7fdf5c068d90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf7c150b30 0x7fdf7c14d8c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c0c7980 0x7fdf7c0713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 --2- 172.21.15.7:0/4278971813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdf7c0c24f0 0x7fdf7c072370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 >> 172.21.15.7:0/4278971813 conn(0x7fdf7c0662e0 msgr2=0x7fdf7c14f370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 shutdown_connections 2023-11-28T18:23:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.328+0000 7fdf8149a640 1 -- 172.21.15.7:0/4278971813 wait complete. 2023-11-28T18:23:34.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.488+0000 7f75fa7b8640 1 Processor -- start 2023-11-28T18:23:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.488+0000 7f75fa7b8640 1 -- start start 2023-11-28T18:23:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75fa7b8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f75f4072450 0x7f75f4072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75fa7b8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40ca920 con 0x7f75f40cbb70 2023-11-28T18:23:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75fa7b8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40c4cd0 con 0x7f75f41542e0 2023-11-28T18:23:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75fa7b8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f4071d50 con 0x7f75f4072450 2023-11-28T18:23:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75f37fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f75f40cbb70 0x7f75f40cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44820/0 (socket says 172.21.15.7:44820) 2023-11-28T18:23:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75f37fe640 1 -- 172.21.15.7:0/2590167010 learned_addr learned my addr 172.21.15.7:0/2590167010 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75f8d55640 1 --2- 172.21.15.7:0/2590167010 >> v2:172.21.15.7:3300/0 conn(0x7f75f4072450 0x7f75f4072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75f8d55640 1 -- 172.21.15.7:0/2590167010 >> v1:172.21.15.193:6789/0 conn(0x7f75f41542e0 legacy=0x7f75f415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75f8d55640 1 -- 172.21.15.7:0/2590167010 >> v1:172.21.15.7:6789/0 conn(0x7f75f40cbb70 legacy=0x7f75f40cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.492+0000 7f75f8d55640 1 -- 172.21.15.7:0/2590167010 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40cab70 con 0x7f75f4072450 2023-11-28T18:23:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75f8d55640 1 --2- 172.21.15.7:0/2590167010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f4072450 0x7f75f4072870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009d90 tx=0x7f75ec01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d741e76af2cd1d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75f2ffd640 1 -- 172.21.15.7:0/2590167010 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0a5920 con 0x7f75f4072450 2023-11-28T18:23:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75f2ffd640 1 -- 172.21.15.7:0/2590167010 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec0a5a80 con 0x7f75f4072450 2023-11-28T18:23:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75f2ffd640 1 -- 172.21.15.7:0/2590167010 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0a5e10 con 0x7f75f4072450 2023-11-28T18:23:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 -- 172.21.15.7:0/2590167010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f4072450 msgr2=0x7f75f4072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 --2- 172.21.15.7:0/2590167010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f4072450 0x7f75f4072870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009d90 tx=0x7f75ec01c950 comp rx=0 tx=0).stop 2023-11-28T18:23:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 -- 172.21.15.7:0/2590167010 shutdown_connections 2023-11-28T18:23:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 --2- 172.21.15.7:0/2590167010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f4072450 0x7f75f4072870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 -- 172.21.15.7:0/2590167010 >> 172.21.15.7:0/2590167010 conn(0x7f75f40717e0 msgr2=0x7f75f40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 -- 172.21.15.7:0/2590167010 shutdown_connections 2023-11-28T18:23:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 -- 172.21.15.7:0/2590167010 wait complete. 2023-11-28T18:23:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 Processor -- start 2023-11-28T18:23:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.496+0000 7f75fa7b8640 1 -- start start 2023-11-28T18:23:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f75f4072450 0x7f75f41f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f40cbb70 0x7f75f4156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75f41542e0 0x7f75f41569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4072030 con 0x7f75f40cbb70 2023-11-28T18:23:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f41f39b0 con 0x7f75f41542e0 2023-11-28T18:23:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4154df0 con 0x7f75f4072450 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f40cbb70 0x7f75f4156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f40cbb70 0x7f75f4156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40046/0 (socket says 172.21.15.7:40046) 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 -- 172.21.15.7:0/1280681776 learned_addr learned my addr 172.21.15.7:0/1280681776 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f3fff640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f75f4072450 0x7f75f41f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f8d55640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75f41542e0 0x7f75f41569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 -- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f75f4072450 msgr2=0x7f75f41f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f75f4072450 0x7f75f41f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 -- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75f41542e0 msgr2=0x7f75f41569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75f41542e0 0x7f75f41569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 -- 172.21.15.7:0/1280681776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec009a70 con 0x7f75f40cbb70 2023-11-28T18:23:34.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f8d55640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75f41542e0 0x7f75f41569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:34.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f37fe640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f40cbb70 0x7f75f4156480 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c530 tx=0x7f75e800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:34.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f0ff9640 1 -- 172.21.15.7:0/1280681776 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8005cf0 con 0x7f75f40cbb70 2023-11-28T18:23:34.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f0ff9640 1 -- 172.21.15.7:0/1280681776 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e8005e50 con 0x7f75f40cbb70 2023-11-28T18:23:34.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f0ff9640 1 -- 172.21.15.7:0/1280681776 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8007270 con 0x7f75f40cbb70 2023-11-28T18:23:34.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f4155040 con 0x7f75f40cbb70 2023-11-28T18:23:34.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f4155600 con 0x7f75f40cbb70 2023-11-28T18:23:34.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f406c670 con 0x7f75f40cbb70 2023-11-28T18:23:34.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.500+0000 7f75f0ff9640 1 -- 172.21.15.7:0/1280681776 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f75e80073d0 con 0x7f75f40cbb70 2023-11-28T18:23:34.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.504+0000 7f75f0ff9640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f75e00668d0 0x7f75e0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:34.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.504+0000 7f75f0ff9640 1 -- 172.21.15.7:0/1280681776 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75e8083950 con 0x7f75f40cbb70 2023-11-28T18:23:34.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.504+0000 7f75f3fff640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f75e00668d0 0x7f75e0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:34.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.504+0000 7f75f3fff640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f75e00668d0 0x7f75e0068d90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f75e4007fd0 tx=0x7f75e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:34.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.508+0000 7f75f0ff9640 1 -- 172.21.15.7:0/1280681776 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f75e8050b90 con 0x7f75f40cbb70 2023-11-28T18:23:34.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.700+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75f404e890 con 0x7f75f40cbb70 2023-11-28T18:23:34.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.700+0000 7f75f0ff9640 1 -- 172.21.15.7:0/1280681776 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f75e8017020 con 0x7f75f40cbb70 2023-11-28T18:23:34.705 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:34.705 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:23:30.233267+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:23:34.706 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 15 2023-11-28T18:23:34.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f75e00668d0 msgr2=0x7f75e0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f75e00668d0 0x7f75e0068d90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f75e4007fd0 tx=0x7f75e4009040 comp rx=0 tx=0).stop 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f40cbb70 msgr2=0x7f75f4156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f40cbb70 0x7f75f4156480 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c530 tx=0x7f75e800c9f0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 shutdown_connections 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f75e00668d0 0x7f75e0068d90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75f41542e0 0x7f75f41569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75f40cbb70 0x7f75f4156480 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 --2- 172.21.15.7:0/1280681776 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f75f4072450 0x7f75f41f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 >> 172.21.15.7:0/1280681776 conn(0x7f75f40717e0 msgr2=0x7f75f40ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 shutdown_connections 2023-11-28T18:23:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:34.704+0000 7f75fa7b8640 1 -- 172.21.15.7:0/1280681776 wait complete. 2023-11-28T18:23:35.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T18:23:08.041625+0000', 'modified': '2023-11-28T18:23:30.233267+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4291, 'mds_1': 4288, 'mds_2': 4285, 'mds_3': 4284, 'mds_4': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 4, 'incarnation': 15, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.193:6836/1372595381', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6833', 'nonce': 1372595381}, {'type': 'v1', 'addr': '172.21.15.193:6836', 'nonce': 1372595381}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4284': {'gid': 4284, 'name': 'a', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.7:6835/1419029356', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6834', 'nonce': 1419029356}, {'type': 'v1', 'addr': '172.21.15.7:6835', 'nonce': 1419029356}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4285': {'gid': 4285, 'name': 'd', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.193:6837/3911527535', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3911527535}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 3911527535}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4288': {'gid': 4288, 'name': 'c', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.7:6838/2690556675', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6836', 'nonce': 2690556675}, {'type': 'v1', 'addr': '172.21.15.7:6838', 'nonce': 2690556675}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4291': {'gid': 4291, 'name': 'e', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.7:6839/1819203827', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6837', 'nonce': 1819203827}, {'type': 'v1', 'addr': '172.21.15.7:6839', 'nonce': 1819203827}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T18:23:35.331 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4282:up:creating 2023-11-28T18:23:36.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.480+0000 7f7f36438640 1 Processor -- start 2023-11-28T18:23:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f36438640 1 -- start start 2023-11-28T18:23:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f36438640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7f30159c90 0x7f7f3015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f36438640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7f3015def0 0x7f7f300ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f36438640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3015d240 con 0x7f7f30153ee0 2023-11-28T18:23:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f36438640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3015d520 con 0x7f7f30159c90 2023-11-28T18:23:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f36438640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f300cbd10 con 0x7f7f3015def0 2023-11-28T18:23:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f34c35640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7f30159c90 0x7f7f3015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f34c35640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7f30159c90 0x7f7f3015d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40056/0 (socket says 172.21.15.7:40056) 2023-11-28T18:23:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f34c35640 1 -- 172.21.15.7:0/1959451395 learned_addr learned my addr 172.21.15.7:0/1959451395 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f35c37640 1 --2- 172.21.15.7:0/1959451395 >> v2:172.21.15.193:3300/0 conn(0x7f7f3015def0 0x7f7f300ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f34c35640 1 -- 172.21.15.7:0/1959451395 >> v2:172.21.15.193:3300/0 conn(0x7f7f3015def0 msgr2=0x7f7f300ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f34c35640 1 --2- 172.21.15.7:0/1959451395 >> v2:172.21.15.193:3300/0 conn(0x7f7f3015def0 0x7f7f300ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f34c35640 1 -- 172.21.15.7:0/1959451395 >> v1:172.21.15.193:6789/0 conn(0x7f7f30153ee0 legacy=0x7f7f300cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.484+0000 7f7f34c35640 1 -- 172.21.15.7:0/1959451395 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f300c4bc0 con 0x7f7f30159c90 2023-11-28T18:23:36.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f34c35640 1 --2- 172.21.15.7:0/1959451395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f30159c90 0x7f7f3015d9b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7f3006b4e0 tx=0x7f7f2001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9417ef7c0e7a114b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:36.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f27fff640 1 -- 172.21.15.7:0/1959451395 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f200a89f0 con 0x7f7f30159c90 2023-11-28T18:23:36.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f27fff640 1 -- 172.21.15.7:0/1959451395 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f200a8b50 con 0x7f7f30159c90 2023-11-28T18:23:36.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f27fff640 1 -- 172.21.15.7:0/1959451395 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f200b0440 con 0x7f7f30159c90 2023-11-28T18:23:36.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 -- 172.21.15.7:0/1959451395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f30159c90 msgr2=0x7f7f3015d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:36.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 --2- 172.21.15.7:0/1959451395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f30159c90 0x7f7f3015d9b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7f3006b4e0 tx=0x7f7f2001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 -- 172.21.15.7:0/1959451395 shutdown_connections 2023-11-28T18:23:36.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 --2- 172.21.15.7:0/1959451395 >> v2:172.21.15.193:3300/0 conn(0x7f7f3015def0 0x7f7f300ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 --2- 172.21.15.7:0/1959451395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f30159c90 0x7f7f3015d9b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 -- 172.21.15.7:0/1959451395 >> 172.21.15.7:0/1959451395 conn(0x7f7f30071e20 msgr2=0x7f7f30072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 -- 172.21.15.7:0/1959451395 shutdown_connections 2023-11-28T18:23:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 -- 172.21.15.7:0/1959451395 wait complete. 2023-11-28T18:23:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.488+0000 7f7f36438640 1 Processor -- start 2023-11-28T18:23:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 -- start start 2023-11-28T18:23:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f30153ee0 0x7f7f300cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f30159c90 0x7f7f300ce240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f3015def0 0x7f7f30149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f301f05a0 con 0x7f7f3015def0 2023-11-28T18:23:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f300b54f0 con 0x7f7f30159c90 2023-11-28T18:23:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f301f0250 con 0x7f7f30153ee0 2023-11-28T18:23:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f3015def0 0x7f7f30149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f3015def0 0x7f7f30149e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40058/0 (socket says 172.21.15.7:40058) 2023-11-28T18:23:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 -- 172.21.15.7:0/2641187565 learned_addr learned my addr 172.21.15.7:0/2641187565 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f34c35640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f30159c90 0x7f7f300ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35436640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f30153ee0 0x7f7f300cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 -- 172.21.15.7:0/2641187565 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f30153ee0 msgr2=0x7f7f300cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f30153ee0 0x7f7f300cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 -- 172.21.15.7:0/2641187565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f30159c90 msgr2=0x7f7f300ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f30159c90 0x7f7f300ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 -- 172.21.15.7:0/2641187565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f200094d0 con 0x7f7f3015def0 2023-11-28T18:23:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f35c37640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f3015def0 0x7f7f30149e90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800efc0 tx=0x7f7f2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f25ffb640 1 -- 172.21.15.7:0/2641187565 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28009ca0 con 0x7f7f3015def0 2023-11-28T18:23:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f25ffb640 1 -- 172.21.15.7:0/2641187565 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2800f040 con 0x7f7f3015def0 2023-11-28T18:23:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f300cec60 con 0x7f7f3015def0 2023-11-28T18:23:36.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f25ffb640 1 -- 172.21.15.7:0/2641187565 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f280146c0 con 0x7f7f3015def0 2023-11-28T18:23:36.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3014faf0 con 0x7f7f3015def0 2023-11-28T18:23:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.492+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f301f06e0 con 0x7f7f3015def0 2023-11-28T18:23:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.496+0000 7f7f25ffb640 1 -- 172.21.15.7:0/2641187565 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7f28014820 con 0x7f7f3015def0 2023-11-28T18:23:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.496+0000 7f7f25ffb640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7f140668d0 0x7f7f14068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.496+0000 7f7f25ffb640 1 -- 172.21.15.7:0/2641187565 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f28085210 con 0x7f7f3015def0 2023-11-28T18:23:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.496+0000 7f7f35436640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7f140668d0 0x7f7f14068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.496+0000 7f7f35436640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7f140668d0 0x7f7f14068d90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c007fd0 tx=0x7f7f2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.500+0000 7f7f25ffb640 1 -- 172.21.15.7:0/2641187565 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7f301f06e0 con 0x7f7f3015def0 2023-11-28T18:23:36.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.696+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f301f06e0 con 0x7f7f3015def0 2023-11-28T18:23:36.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.696+0000 7f7f25ffb640 1 -- 172.21.15.7:0/2641187565 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f7f301f06e0 con 0x7f7f3015def0 2023-11-28T18:23:36.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:36.703 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:23:35.257171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:23:36.703 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2023-11-28T18:23:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7f140668d0 msgr2=0x7f7f14068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7f140668d0 0x7f7f14068d90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c007fd0 tx=0x7f7f2c009040 comp rx=0 tx=0).stop 2023-11-28T18:23:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f3015def0 msgr2=0x7f7f30149e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f3015def0 0x7f7f30149e90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800efc0 tx=0x7f7f2800c490 comp rx=0 tx=0).stop 2023-11-28T18:23:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 shutdown_connections 2023-11-28T18:23:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7f140668d0 0x7f7f14068d90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f3015def0 0x7f7f30149e90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f30159c90 0x7f7f300ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 --2- 172.21.15.7:0/2641187565 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7f30153ee0 0x7f7f300cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:36.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 >> 172.21.15.7:0/2641187565 conn(0x7f7f30071e20 msgr2=0x7f7f30151000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:36.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 shutdown_connections 2023-11-28T18:23:36.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:36.700+0000 7f7f36438640 1 -- 172.21.15.7:0/2641187565 wait complete. 2023-11-28T18:23:36.972 INFO:tasks.ceph.mds.e.smithi007.stderr:2023-11-28T18:23:36.968+0000 7f7e1dce5640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-11-28T18:23:37.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T18:23:08.041625+0000', 'modified': '2023-11-28T18:23:35.257171+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4291, 'mds_1': 4288, 'mds_2': 4285, 'mds_3': 4284, 'mds_4': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 4, 'incarnation': 15, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.193:6836/1372595381', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6833', 'nonce': 1372595381}, {'type': 'v1', 'addr': '172.21.15.193:6836', 'nonce': 1372595381}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4284': {'gid': 4284, 'name': 'a', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.7:6835/1419029356', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6834', 'nonce': 1419029356}, {'type': 'v1', 'addr': '172.21.15.7:6835', 'nonce': 1419029356}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4285': {'gid': 4285, 'name': 'd', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.193:6837/3911527535', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3911527535}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 3911527535}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4288': {'gid': 4288, 'name': 'c', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.7:6838/2690556675', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6836', 'nonce': 2690556675}, {'type': 'v1', 'addr': '172.21.15.7:6838', 'nonce': 2690556675}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4291': {'gid': 4291, 'name': 'e', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.7:6839/1819203827', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6837', 'nonce': 1819203827}, {'type': 'v1', 'addr': '172.21.15.7:6839', 'nonce': 1819203827}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T18:23:37.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-11-28T18:23:37.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-28T18:23:37.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d44c32640 1 Processor -- start 2023-11-28T18:23:37.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d44c32640 1 -- start start 2023-11-28T18:23:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d44c32640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9d400cb890 0x7f9d400ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d44c32640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d400c4b10 con 0x7f9d400cb420 2023-11-28T18:23:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d44c32640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4015a770 con 0x7f9d40154140 2023-11-28T18:23:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d44c32640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4015aa50 con 0x7f9d400cb890 2023-11-28T18:23:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d3ed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9d400cb890 0x7f9d400ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d3dd9b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9d400cb420 0x7f9d40072620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44826/0 (socket says 172.21.15.7:44826) 2023-11-28T18:23:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.476+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/425098022 learned_addr learned my addr 172.21.15.7:0/425098022 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3d59a640 1 -- 172.21.15.7:0/425098022 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282989320 0 0) 0x7f9d400c4b10 con 0x7f9d400cb420 2023-11-28T18:23:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3d59a640 1 -- 172.21.15.7:0/425098022 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d28003530 con 0x7f9d400cb420 2023-11-28T18:23:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3ed9d640 1 -- 172.21.15.7:0/425098022 >> v1:172.21.15.193:6789/0 conn(0x7f9d40154140 legacy=0x7f9d40159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3ed9d640 1 -- 172.21.15.7:0/425098022 >> v1:172.21.15.7:6789/0 conn(0x7f9d400cb420 legacy=0x7f9d40072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3ed9d640 1 -- 172.21.15.7:0/425098022 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4015ad30 con 0x7f9d400cb890 2023-11-28T18:23:37.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/425098022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb890 0x7f9d400ca500 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009d90 tx=0x7f9d3801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d882d286878f1cf4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:37.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3d59a640 1 -- 172.21.15.7:0/425098022 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d380a58f0 con 0x7f9d400cb890 2023-11-28T18:23:37.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3d59a640 1 -- 172.21.15.7:0/425098022 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d380a5a50 con 0x7f9d400cb890 2023-11-28T18:23:37.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d3d59a640 1 -- 172.21.15.7:0/425098022 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d380a5de0 con 0x7f9d400cb890 2023-11-28T18:23:37.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d44c32640 1 -- 172.21.15.7:0/425098022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb890 msgr2=0x7f9d400ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:37.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.480+0000 7f9d44c32640 1 --2- 172.21.15.7:0/425098022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb890 0x7f9d400ca500 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009d90 tx=0x7f9d3801c920 comp rx=0 tx=0).stop 2023-11-28T18:23:37.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 -- 172.21.15.7:0/425098022 shutdown_connections 2023-11-28T18:23:37.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 --2- 172.21.15.7:0/425098022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb890 0x7f9d400ca500 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 -- 172.21.15.7:0/425098022 >> 172.21.15.7:0/425098022 conn(0x7f9d400cfaa0 msgr2=0x7f9d400cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:37.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 -- 172.21.15.7:0/425098022 shutdown_connections 2023-11-28T18:23:37.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 -- 172.21.15.7:0/425098022 wait complete. 2023-11-28T18:23:37.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 Processor -- start 2023-11-28T18:23:37.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 -- start start 2023-11-28T18:23:37.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb420 0x7f9d4014c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d400cb890 0x7f9d401f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d40154140 0x7f9d400cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d40150c30 con 0x7f9d400cb420 2023-11-28T18:23:37.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4014c6a0 con 0x7f9d40154140 2023-11-28T18:23:37.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d44c32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d401513d0 con 0x7f9d400cb890 2023-11-28T18:23:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb420 0x7f9d4014c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb420 0x7f9d4014c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40074/0 (socket says 172.21.15.7:40074) 2023-11-28T18:23:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 -- 172.21.15.7:0/2790240134 learned_addr learned my addr 172.21.15.7:0/2790240134 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d40154140 0x7f9d400cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d400cb890 0x7f9d401f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 -- 172.21.15.7:0/2790240134 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d400cb890 msgr2=0x7f9d401f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d400cb890 0x7f9d401f71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 -- 172.21.15.7:0/2790240134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d40154140 msgr2=0x7f9d400cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d40154140 0x7f9d400cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 -- 172.21.15.7:0/2790240134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d38009a70 con 0x7f9d400cb420 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.484+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb420 0x7f9d4014c160 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000c3c0 tx=0x7f9d3000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.488+0000 7f9d2effd640 1 -- 172.21.15.7:0/2790240134 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d300052d0 con 0x7f9d400cb420 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.488+0000 7f9d2effd640 1 -- 172.21.15.7:0/2790240134 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d30005430 con 0x7f9d400cb420 2023-11-28T18:23:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.488+0000 7f9d44c32640 1 -- 172.21.15.7:0/2790240134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d401f76e0 con 0x7f9d400cb420 2023-11-28T18:23:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.488+0000 7f9d2effd640 1 -- 172.21.15.7:0/2790240134 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30003840 con 0x7f9d400cb420 2023-11-28T18:23:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.488+0000 7f9d44c32640 1 -- 172.21.15.7:0/2790240134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4014b500 con 0x7f9d400cb420 2023-11-28T18:23:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.488+0000 7f9d2effd640 1 -- 172.21.15.7:0/2790240134 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9d300039a0 con 0x7f9d400cb420 2023-11-28T18:23:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 Processor -- start 2023-11-28T18:23:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d2effd640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9d200668d0 0x7f9d20068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 -- start start 2023-11-28T18:23:37.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d2effd640 1 -- 172.21.15.7:0/2790240134 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d300838a0 con 0x7f9d400cb420 2023-11-28T18:23:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9d200668d0 0x7f9d20068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 0x561a7e40b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e445a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e445fc0 0x561a7e4463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a7e2cc110 con 0x561a7e40b960 2023-11-28T18:23:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a7e25c9a0 con 0x561a7e445fc0 2023-11-28T18:23:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a7e259ed0 con 0x561a7e40b040 2023-11-28T18:23:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9d200668d0 0x7f9d20068d90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d380af000 tx=0x7f9d380a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e445a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e445a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40078/0 (socket says 172.21.15.7:40078) 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/2231001491 learned_addr learned my addr 172.21.15.7:0/2231001491 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 0x561a7e40b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e445fc0 0x561a7e4463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/2231001491 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 msgr2=0x561a7e40b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 0x561a7e40b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/2231001491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e445fc0 msgr2=0x561a7e4463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e445fc0 0x561a7e4463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/2231001491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a7df4d890 con 0x561a7e40b960 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e445a80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001610 tx=0x7f9d34001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0964df75573c1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d1ffff640 1 -- 172.21.15.7:0/2231001491 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34001d20 con 0x561a7e40b960 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d1ffff640 1 -- 172.21.15.7:0/2231001491 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d34005ad0 con 0x561a7e40b960 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d1ffff640 1 -- 172.21.15.7:0/2231001491 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34005e40 con 0x561a7e40b960 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 -- 172.21.15.7:0/2231001491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 msgr2=0x561a7e445a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e445a80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001610 tx=0x7f9d34001ad0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 -- 172.21.15.7:0/2231001491 shutdown_connections 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e445fc0 0x561a7e4463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e445a80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2231001491 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 0x561a7e40b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.492+0000 7f9d47fab000 1 -- 172.21.15.7:0/2231001491 >> 172.21.15.7:0/2231001491 conn(0x561a7e442840 msgr2=0x561a7e444c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 -- 172.21.15.7:0/2231001491 shutdown_connections 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 -- 172.21.15.7:0/2231001491 wait complete. 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 -- start start 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 Processor -- start 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e40b040 0x561a7e440db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 0x561a7e4412f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e445fc0 0x561a7e4073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a7e2557c0 con 0x561a7e445fc0 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a7e20dbf0 con 0x561a7e40b040 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a7e213a90 con 0x561a7e40b960 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 10 client.0 _ll_register_callbacks cb 0x561a7e3f0e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e445fc0 0x561a7e4073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e445fc0 0x561a7e4073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40088/0 (socket says 172.21.15.7:40088) 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 -- 172.21.15.7:0/2390140665 learned_addr learned my addr 172.21.15.7:0/2390140665 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e40b040 0x561a7e440db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 0x561a7e4412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 -- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 msgr2=0x561a7e4412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 0x561a7e4412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 -- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e40b040 msgr2=0x561a7e440db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e40b040 0x561a7e440db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 -- 172.21.15.7:0/2390140665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3400f040 con 0x561a7e445fc0 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e445fc0 0x561a7e4073a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9d3801ce00 tx=0x7f9d38004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2c7e3c9d7625a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/2390140665 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d380b8d10 con 0x561a7e445fc0 2023-11-28T18:23:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/2390140665 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d380c5470 con 0x561a7e445fc0 2023-11-28T18:23:37.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 -- 172.21.15.7:0/2390140665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a7e217470 con 0x561a7e445fc0 2023-11-28T18:23:37.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/2390140665 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d380b8e70 con 0x561a7e445fc0 2023-11-28T18:23:37.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 10 client.4692 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:37.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d47fab000 1 -- 172.21.15.7:0/2390140665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x561a7e216190 con 0x561a7e445fc0 2023-11-28T18:23:37.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.496+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/2390140665 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561a7e217470 con 0x561a7e445fc0 2023-11-28T18:23:37.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/2390140665 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f9d380c5630 con 0x561a7e445fc0 2023-11-28T18:23:37.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d47fab000 10 client.4692 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:37.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d47fab000 10 client.4692 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-28T18:23:37.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d47fab000 20 client.4692 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:37.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e4080e0 0x561a7e4084c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:37.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d47fab000 4 client.4692 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T18:23:37.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d47fab000 1 -- 172.21.15.7:0/2390140665 --> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a7e21ab50 con 0x561a7e4080e0 2023-11-28T18:23:37.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e4080e0 0x561a7e4084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:37.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e4080e0 0x561a7e4084c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=396377a87cfc8e5f server_cookie=70e8a2492e4c0029 in_seq=0 out_seq=0 2023-11-28T18:23:37.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:37.500+0000 7f9d1d7fa640 10 client.4692 ms_handle_connect on v2:172.21.15.193:6833/1372595381 2023-11-28T18:23:41.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/2390140665 <== mds.4 v2:172.21.15.193:6833/1372595381 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561a7e21ab50 con 0x561a7e4080e0 2023-11-28T18:23:41.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d1d7fa640 10 client.4692 handle_command_reply: tid=0 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 client.4692 shutdown 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 -- 172.21.15.7:0/2390140665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e445fc0 msgr2=0x561a7e4073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e445fc0 0x561a7e4073a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9d3801ce00 tx=0x7f9d38004a50 comp rx=0 tx=0).stop 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 -- 172.21.15.7:0/2390140665 shutdown_connections 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e4080e0 0x561a7e4084c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e445fc0 0x561a7e4073a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 0x561a7e4412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 --2- 172.21.15.7:0/2390140665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e40b040 0x561a7e440db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d1d7fa640 0 client.4692 ms_handle_reset on v2:172.21.15.193:6833/1372595381 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 -- 172.21.15.7:0/2390140665 >> 172.21.15.7:0/2390140665 conn(0x561a7e442840 msgr2=0x561a7e445870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 -- 172.21.15.7:0/2390140665 shutdown_connections 2023-11-28T18:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 1 -- 172.21.15.7:0/2390140665 wait complete. 2023-11-28T18:23:41.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.259+0000 7f9d47fab000 20 client.4692 trim_cache size 0 max 16384 2023-11-28T18:23:41.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d47fab000 1 Processor -- start 2023-11-28T18:23:41.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d47fab000 1 -- start start 2023-11-28T18:23:41.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 0x561a7e406f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b040 0x561a7e4074b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 0x561a7e4a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d47fab000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a7df0e940 con 0x561a7e40b040 2023-11-28T18:23:41.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d47fab000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a7e296440 con 0x561a7e442840 2023-11-28T18:23:41.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d47fab000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a7e21f6c0 con 0x561a7e40b960 2023-11-28T18:23:41.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b040 0x561a7e4074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b040 0x561a7e4074b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40094/0 (socket says 172.21.15.7:40094) 2023-11-28T18:23:41.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/1626859174 learned_addr learned my addr 172.21.15.7:0/1626859174 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 0x561a7e4a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 0x561a7e406f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/1626859174 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 msgr2=0x561a7e4a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 0x561a7e4a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/1626859174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 msgr2=0x561a7e406f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 0x561a7e406f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/1626859174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a7e1fc490 con 0x561a7e40b040 2023-11-28T18:23:41.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b040 0x561a7e4074b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d34000f70 tx=0x7f9d3400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3871f1529f6df831 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d2cff9640 1 -- 172.21.15.7:0/1626859174 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34005ad0 con 0x561a7e40b040 2023-11-28T18:23:41.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d2cff9640 1 -- 172.21.15.7:0/1626859174 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d34005c50 con 0x561a7e40b040 2023-11-28T18:23:41.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.279+0000 7f9d2cff9640 1 -- 172.21.15.7:0/1626859174 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340070d0 con 0x561a7e40b040 2023-11-28T18:23:41.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- 172.21.15.7:0/1626859174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b040 msgr2=0x561a7e4074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b040 0x561a7e4074b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d34000f70 tx=0x7f9d3400a000 comp rx=0 tx=0).stop 2023-11-28T18:23:41.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- 172.21.15.7:0/1626859174 shutdown_connections 2023-11-28T18:23:41.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b960 0x561a7e4a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b040 0x561a7e4074b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 --2- 172.21.15.7:0/1626859174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 0x561a7e406f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- 172.21.15.7:0/1626859174 >> 172.21.15.7:0/1626859174 conn(0x561a7e458c20 msgr2=0x561a7e404640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- 172.21.15.7:0/1626859174 shutdown_connections 2023-11-28T18:23:41.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- 172.21.15.7:0/1626859174 wait complete. 2023-11-28T18:23:41.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- start start 2023-11-28T18:23:41.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 Processor -- start 2023-11-28T18:23:41.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 0x561a7e4a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e45b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 0x561a7e45bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a7e251130 con 0x561a7e40b960 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a7e2597c0 con 0x561a7e442840 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a7e4416b0 con 0x561a7e40b040 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d47fab000 10 client.0 _ll_register_callbacks cb 0x561a7e1d96e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d3dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e45b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d3dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e45b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40100/0 (socket says 172.21.15.7:40100) 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/3822869657 learned_addr learned my addr 172.21.15.7:0/3822869657 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d3ed9d640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 0x561a7e45bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.283+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 0x561a7e4a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 msgr2=0x561a7e4a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 0x561a7e4a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 msgr2=0x561a7e45bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 0x561a7e45bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3dd9b640 1 -- 172.21.15.7:0/3822869657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a7df4d890 con 0x561a7e40b960 2023-11-28T18:23:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3dd9b640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e45b9b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9d34013000 tx=0x7f9d34000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef3be406cf198 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/3822869657 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34005ad0 con 0x561a7e40b960 2023-11-28T18:23:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/3822869657 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d34007480 con 0x561a7e40b960 2023-11-28T18:23:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 1 -- 172.21.15.7:0/3822869657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a7e254110 con 0x561a7e40b960 2023-11-28T18:23:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/3822869657 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d34007680 con 0x561a7e40b960 2023-11-28T18:23:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 10 client.4698 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 1 -- 172.21.15.7:0/3822869657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x561a7df4dcc0 con 0x561a7e40b960 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/3822869657 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561a7e254110 con 0x561a7e40b960 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/3822869657 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f9d3400e740 con 0x561a7e40b960 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 10 client.4698 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 10 client.4698 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 20 client.4698 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 0x561a7e49f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 4 client.4698 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 1 -- 172.21.15.7:0/3822869657 --> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561a7e282650 con 0x561a7e445fc0 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 0x561a7e49f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 0x561a7e49f250 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b812f55ae90b474f server_cookie=eb854ac1c9cc6427 in_seq=0 out_seq=0 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 10 client.4698 ms_handle_connect on v2:172.21.15.193:6833/1372595381 2023-11-28T18:23:41.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3e59c640 1 -- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 msgr2=0x561a7e49f250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2023-11-28T18:23:41.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3e59c640 1 -- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 msgr2=0x561a7e49f250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T18:23:41.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 0x561a7e49f250 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T18:23:41.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 0x561a7e49f250 unknown :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2023-11-28T18:23:41.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 0x561a7e49f250 unknown :-1 s=BANNER_CONNECTING pgs=15 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d3e59c640 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 0x561a7e49f250 crc :-1 s=READY pgs=15 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b812f55ae90b474f server_cookie=eb854ac1c9cc6427 in_seq=0 out_seq=1 2023-11-28T18:23:41.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 10 client.4698 ms_handle_connect on v2:172.21.15.193:6833/1372595381 2023-11-28T18:23:41.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 1 -- 172.21.15.7:0/3822869657 <== mds.4 v2:172.21.15.193:6833/1372595381 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x561a7e282650 con 0x561a7e445fc0 2023-11-28T18:23:41.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d1d7fa640 10 client.4698 handle_command_reply: tid=0 2023-11-28T18:23:41.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.287+0000 7f9d47fab000 1 client.4698 shutdown 2023-11-28T18:23:41.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 -- 172.21.15.7:0/3822869657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 msgr2=0x561a7e45b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e45b9b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9d34013000 tx=0x7f9d34000ee0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 -- 172.21.15.7:0/3822869657 shutdown_connections 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:6833/1372595381,v1:172.21.15.193:6836/1372595381] conn(0x561a7e445fc0 0x561a7e49f250 crc :-1 s=READY pgs=15 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7e442840 0x561a7e45bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561a7e40b960 0x561a7e45b9b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 --2- 172.21.15.7:0/3822869657 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x561a7e40b040 0x561a7e4a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 -- 172.21.15.7:0/3822869657 >> 172.21.15.7:0/3822869657 conn(0x561a7e458c20 msgr2=0x561a7e4a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d1d7fa640 0 client.4698 ms_handle_reset on v2:172.21.15.193:6833/1372595381 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 -- 172.21.15.7:0/3822869657 shutdown_connections 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 1 -- 172.21.15.7:0/3822869657 wait complete. 2023-11-28T18:23:41.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d47fab000 20 client.4698 trim_cache size 0 max 16384 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "b4396e54-0cbb-426d-8760-05a802ba48a2", 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 4, 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4282, 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 11.057791183000001, 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 16, 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 20, 2023-11-28T18:23:41.299 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 20, 2023-11-28T18:23:41.300 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 40.330693066999999 2023-11-28T18:23:41.300 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-28T18:23:41.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 -- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9d200668d0 msgr2=0x7f9d20068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9d200668d0 0x7f9d20068d90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d380af000 tx=0x7f9d380a7070 comp rx=0 tx=0).stop 2023-11-28T18:23:41.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 -- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb420 msgr2=0x7f9d4014c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb420 0x7f9d4014c160 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000c3c0 tx=0x7f9d3000c880 comp rx=0 tx=0).stop 2023-11-28T18:23:41.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 -- 172.21.15.7:0/2790240134 shutdown_connections 2023-11-28T18:23:41.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9d200668d0 0x7f9d20068d90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d40154140 0x7f9d400cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9d400cb890 0x7f9d401f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 --2- 172.21.15.7:0/2790240134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d400cb420 0x7f9d4014c160 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 -- 172.21.15.7:0/2790240134 >> 172.21.15.7:0/2790240134 conn(0x7f9d400cfaa0 msgr2=0x7f9d400cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 -- 172.21.15.7:0/2790240134 shutdown_connections 2023-11-28T18:23:41.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.291+0000 7f9d44c32640 1 -- 172.21.15.7:0/2790240134 wait complete. 2023-11-28T18:23:41.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-28T18:23:41.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8996fe640 1 Processor -- start 2023-11-28T18:23:41.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8996fe640 1 -- start start 2023-11-28T18:23:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8996fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa8940c8430 0x7fa8940c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8996fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa8940c8810 0x7fa894156c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8996fe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8940c6a70 con 0x7fa8940c2620 2023-11-28T18:23:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8996fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8940c3470 con 0x7fa8940c8430 2023-11-28T18:23:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8996fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8940c7b30 con 0x7fa8940c8810 2023-11-28T18:23:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa892ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa8940c8430 0x7fa8940c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8937fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa8940c2620 0x7fa894150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44834/0 (socket says 172.21.15.7:44834) 2023-11-28T18:23:41.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8937fe640 1 -- 172.21.15.7:0/843548910 learned_addr learned my addr 172.21.15.7:0/843548910 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:41.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa88bfff640 1 --2- 172.21.15.7:0/843548910 >> v2:172.21.15.193:3300/0 conn(0x7fa8940c8810 0x7fa894156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa892ffd640 1 -- 172.21.15.7:0/843548910 >> v2:172.21.15.193:3300/0 conn(0x7fa8940c8810 msgr2=0x7fa894156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa892ffd640 1 --2- 172.21.15.7:0/843548910 >> v2:172.21.15.193:3300/0 conn(0x7fa8940c8810 0x7fa894156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa892ffd640 1 -- 172.21.15.7:0/843548910 >> v1:172.21.15.7:6789/0 conn(0x7fa8940c2620 legacy=0x7fa894150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa892ffd640 1 -- 172.21.15.7:0/843548910 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940c7e10 con 0x7fa8940c8430 2023-11-28T18:23:41.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.463+0000 7fa8927fc640 1 -- 172.21.15.7:0/843548910 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051706164 0 0) 0x7fa8940c6a70 con 0x7fa8940c2620 2023-11-28T18:23:41.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa892ffd640 1 --2- 172.21.15.7:0/843548910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8430 0x7fa8940c2f30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa8940c4920 tx=0x7fa88401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec93c217ff0c27be server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8927fc640 1 -- 172.21.15.7:0/843548910 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8840a8950 con 0x7fa8940c8430 2023-11-28T18:23:41.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8927fc640 1 -- 172.21.15.7:0/843548910 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8840a8ab0 con 0x7fa8940c8430 2023-11-28T18:23:41.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8927fc640 1 -- 172.21.15.7:0/843548910 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8840a8e40 con 0x7fa8940c8430 2023-11-28T18:23:41.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8996fe640 1 -- 172.21.15.7:0/843548910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8430 msgr2=0x7fa8940c2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8996fe640 1 --2- 172.21.15.7:0/843548910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8430 0x7fa8940c2f30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa8940c4920 tx=0x7fa88401ec50 comp rx=0 tx=0).stop 2023-11-28T18:23:41.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8996fe640 1 -- 172.21.15.7:0/843548910 shutdown_connections 2023-11-28T18:23:41.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8996fe640 1 --2- 172.21.15.7:0/843548910 >> v2:172.21.15.193:3300/0 conn(0x7fa8940c8810 0x7fa894156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8996fe640 1 --2- 172.21.15.7:0/843548910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8430 0x7fa8940c2f30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8996fe640 1 -- 172.21.15.7:0/843548910 >> 172.21.15.7:0/843548910 conn(0x7fa8940662e0 msgr2=0x7fa8940666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8996fe640 1 -- 172.21.15.7:0/843548910 shutdown_connections 2023-11-28T18:23:41.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.467+0000 7fa8996fe640 1 -- 172.21.15.7:0/843548910 wait complete. 2023-11-28T18:23:41.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 Processor -- start 2023-11-28T18:23:41.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 -- start start 2023-11-28T18:23:41.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8940c2620 0x7fa8941f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8940c8430 0x7fa8941ed970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8810 0x7fa89414c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8941363d0 con 0x7fa8940c8810 2023-11-28T18:23:41.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8941f1530 con 0x7fa8940c8430 2023-11-28T18:23:41.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa894137230 con 0x7fa8940c2620 2023-11-28T18:23:41.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8940c2620 0x7fa8941f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8937fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8810 0x7fa89414c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8940c2620 0x7fa8941f19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:60150/0 (socket says 172.21.15.7:60150) 2023-11-28T18:23:41.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa892ffd640 1 -- 172.21.15.7:0/2993994119 learned_addr learned my addr 172.21.15.7:0/2993994119 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa88bfff640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8940c8430 0x7fa8941ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8937fe640 1 -- 172.21.15.7:0/2993994119 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8940c2620 msgr2=0x7fa8941f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8937fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8940c2620 0x7fa8941f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8937fe640 1 -- 172.21.15.7:0/2993994119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8940c8430 msgr2=0x7fa8941ed970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8937fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8940c8430 0x7fa8941ed970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8937fe640 1 -- 172.21.15.7:0/2993994119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa884009420 con 0x7fa8940c8810 2023-11-28T18:23:41.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8937fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8810 0x7fa89414c540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa88c002240 tx=0x7fa88c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa88b7fe640 1 -- 172.21.15.7:0/2993994119 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00ce30 con 0x7fa8940c8810 2023-11-28T18:23:41.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa88b7fe640 1 -- 172.21.15.7:0/2993994119 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c00a5e0 con 0x7fa8940c8810 2023-11-28T18:23:41.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa88b7fe640 1 -- 172.21.15.7:0/2993994119 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00a990 con 0x7fa8940c8810 2023-11-28T18:23:41.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 -- 172.21.15.7:0/2993994119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa894136f70 con 0x7fa8940c8810 2023-11-28T18:23:41.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.471+0000 7fa8996fe640 1 -- 172.21.15.7:0/2993994119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa894137790 con 0x7fa8940c8810 2023-11-28T18:23:41.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.475+0000 7fa88b7fe640 1 -- 172.21.15.7:0/2993994119 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa88c00aaf0 con 0x7fa8940c8810 2023-11-28T18:23:41.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.475+0000 7fa88b7fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa874066900 0x7fa874068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.475+0000 7fa88b7fe640 1 -- 172.21.15.7:0/2993994119 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa88c0837c0 con 0x7fa8940c8810 2023-11-28T18:23:41.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.475+0000 7fa892ffd640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa874066900 0x7fa874068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.475+0000 7fa89ca5f000 1 Processor -- start 2023-11-28T18:23:41.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.475+0000 7fa89ca5f000 1 -- start start 2023-11-28T18:23:41.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.475+0000 7fa892ffd640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa874066900 0x7fa874068dc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa88401ec90 tx=0x7fa884008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595ae420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595e8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x5619595e93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x56195946f110 con 0x5619595ae040 2023-11-28T18:23:41.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619593ff9a0 con 0x5619595ae960 2023-11-28T18:23:41.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619593fced0 con 0x5619595e8fc0 2023-11-28T18:23:41.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40116/0 (socket says 172.21.15.7:40116) 2023-11-28T18:23:41.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 -- 172.21.15.7:0/3910344418 learned_addr learned my addr 172.21.15.7:0/3910344418 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa88bfff640 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595e8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa8937fe640 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x5619595e93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 -- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 msgr2=0x5619595e93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x5619595e93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 -- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 msgr2=0x5619595e8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595e8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 -- 172.21.15.7:0/3910344418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619590f0890 con 0x5619595ae040 2023-11-28T18:23:41.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa8937fe640 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x5619595e93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:41.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa892ffd640 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595ae420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa884000c10 tx=0x7fa884004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c0193c8e73f418 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa888ff9640 1 -- 172.21.15.7:0/3910344418 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884004030 con 0x5619595ae040 2023-11-28T18:23:41.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa888ff9640 1 -- 172.21.15.7:0/3910344418 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8840c1040 con 0x5619595ae040 2023-11-28T18:23:41.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa888ff9640 1 -- 172.21.15.7:0/3910344418 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8840043b0 con 0x5619595ae040 2023-11-28T18:23:41.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3910344418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 msgr2=0x5619595ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595ae420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa884000c10 tx=0x7fa884004740 comp rx=0 tx=0).stop 2023-11-28T18:23:41.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3910344418 shutdown_connections 2023-11-28T18:23:41.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x5619595e93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595e8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/3910344418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595ae420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3910344418 >> 172.21.15.7:0/3910344418 conn(0x5619595e5840 msgr2=0x5619595e7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3910344418 shutdown_connections 2023-11-28T18:23:41.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3910344418 wait complete. 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 -- start start 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.479+0000 7fa89ca5f000 1 Processor -- start 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595e4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595ae960 0x5619595e46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595e8fc0 0x5619595aa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619593f87c0 con 0x5619595ae040 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619593b0bf0 con 0x5619595e8fc0 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619593b6a90 con 0x5619595ae960 2023-11-28T18:23:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 10 client.0 _ll_register_callbacks cb 0x561959593e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595e4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595e4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40120/0 (socket says 172.21.15.7:40120) 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 -- 172.21.15.7:0/1169908606 learned_addr learned my addr 172.21.15.7:0/1169908606 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa8937fe640 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595e8fc0 0x5619595aa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa88bfff640 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595ae960 0x5619595e46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 -- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595ae960 msgr2=0x5619595e46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595ae960 0x5619595e46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 -- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595e8fc0 msgr2=0x5619595aa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595e8fc0 0x5619595aa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 -- 172.21.15.7:0/1169908606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8940c7e10 con 0x5619595ae040 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa88bfff640 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595ae960 0x5619595e46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa892ffd640 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595e4180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa8840b3040 tx=0x7fa88401d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c825661afe1dfb2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 1 -- 172.21.15.7:0/1169908606 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884003620 con 0x5619595ae040 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 1 -- 172.21.15.7:0/1169908606 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8840d1ae0 con 0x5619595ae040 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 -- 172.21.15.7:0/1169908606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619593ba470 con 0x5619595ae040 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 1 -- 172.21.15.7:0/1169908606 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8840d1ca0 con 0x5619595ae040 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 10 client.4713 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 -- 172.21.15.7:0/1169908606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5619593b9190 con 0x5619595ae040 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 1 -- 172.21.15.7:0/1169908606 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5619593ba470 con 0x5619595ae040 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 1 -- 172.21.15.7:0/1169908606 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fa8840c4070 con 0x5619595ae040 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 10 client.4713 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 10 client.4713 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 20 client.4713 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] conn(0x5619595ab170 0x5619595ab550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 4 client.4713 mds_command: new command op to 4284 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 -- 172.21.15.7:0/1169908606 --> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619593bdb50 con 0x5619595ab170 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa88bfff640 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] conn(0x5619595ab170 0x5619595ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 10 client.4713 ms_handle_connect on v2:172.21.15.7:6834/1419029356 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa88bfff640 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] conn(0x5619595ab170 0x5619595ab550 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ebaff437aad8c26f server_cookie=9fe3679715c0e7d8 in_seq=0 out_seq=0 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 1 -- 172.21.15.7:0/1169908606 <== mds.3 v2:172.21.15.7:6834/1419029356 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5619593bdb50 con 0x5619595ab170 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa871ffb640 10 client.4713 handle_command_reply: tid=0 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.483+0000 7fa89ca5f000 1 client.4713 shutdown 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 -- 172.21.15.7:0/1169908606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 msgr2=0x5619595e4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595e4180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa8840b3040 tx=0x7fa88401d490 comp rx=0 tx=0).stop 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 -- 172.21.15.7:0/1169908606 shutdown_connections 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595e8fc0 0x5619595aa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] conn(0x5619595ab170 0x5619595ab550 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595ae960 0x5619595e46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/1169908606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595e4180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa871ffb640 0 client.4713 ms_handle_reset on v2:172.21.15.7:6834/1419029356 2023-11-28T18:23:41.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 -- 172.21.15.7:0/1169908606 >> 172.21.15.7:0/1169908606 conn(0x5619595e5840 msgr2=0x5619595e8870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 -- 172.21.15.7:0/1169908606 shutdown_connections 2023-11-28T18:23:41.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 1 -- 172.21.15.7:0/1169908606 wait complete. 2023-11-28T18:23:41.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.487+0000 7fa89ca5f000 20 client.4713 trim_cache size 0 max 16384 2023-11-28T18:23:41.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.499+0000 7fa89ca5f000 1 Processor -- start 2023-11-28T18:23:41.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.499+0000 7fa89ca5f000 1 -- start start 2023-11-28T18:23:41.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x561959604e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619590b1940 con 0x5619595ae040 2023-11-28T18:23:41.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619593938a0 con 0x5619595ae960 2023-11-28T18:23:41.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619596009d0 con 0x5619595e8fc0 2023-11-28T18:23:41.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595a96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40128/0 (socket says 172.21.15.7:40128) 2023-11-28T18:23:41.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 -- 172.21.15.7:0/3477460995 learned_addr learned my addr 172.21.15.7:0/3477460995 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa88bfff640 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa8937fe640 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x561959604e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 -- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 msgr2=0x561959604e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x561959604e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 -- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 msgr2=0x5619595a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 -- 172.21.15.7:0/3477460995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56195939f490 con 0x5619595ae040 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa88bfff640 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa892ffd640 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595a96d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa88401c880 tx=0x7fa88400b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf58bc11180e557 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa8897fa640 1 -- 172.21.15.7:0/3477460995 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884004440 con 0x5619595ae040 2023-11-28T18:23:41.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa8897fa640 1 -- 172.21.15.7:0/3477460995 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8840c1040 con 0x5619595ae040 2023-11-28T18:23:41.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa8897fa640 1 -- 172.21.15.7:0/3477460995 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884003620 con 0x5619595ae040 2023-11-28T18:23:41.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3477460995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 msgr2=0x5619595a96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595a96d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa88401c880 tx=0x7fa88400b050 comp rx=0 tx=0).stop 2023-11-28T18:23:41.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3477460995 shutdown_connections 2023-11-28T18:23:41.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x561959604e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae960 0x5619595a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/3477460995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae040 0x5619595a96d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3477460995 >> 172.21.15.7:0/3477460995 conn(0x5619595e5840 msgr2=0x5619595a8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3477460995 shutdown_connections 2023-11-28T18:23:41.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- 172.21.15.7:0/3477460995 wait complete. 2023-11-28T18:23:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 -- start start 2023-11-28T18:23:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.503+0000 7fa89ca5f000 1 Processor -- start 2023-11-28T18:23:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae040 0x56195964cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae960 0x56195964d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x561959607140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619593b1d60 con 0x5619595ae960 2023-11-28T18:23:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619590f11b0 con 0x5619595ae040 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619593f87c0 con 0x5619595e8fc0 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 10 client.0 _ll_register_callbacks cb 0x56195937c6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa88bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae960 0x56195964d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae040 0x56195964cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae040 0x56195964cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:43780/0 (socket says 172.21.15.7:43780) 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa892ffd640 1 -- 172.21.15.7:0/554906719 learned_addr learned my addr 172.21.15.7:0/554906719 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa8937fe640 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x561959607140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa88bfff640 1 -- 172.21.15.7:0/554906719 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 msgr2=0x561959607140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa88bfff640 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x561959607140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa88bfff640 1 -- 172.21.15.7:0/554906719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae040 msgr2=0x56195964cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa88bfff640 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae040 0x56195964cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa88bfff640 1 -- 172.21.15.7:0/554906719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619593b9190 con 0x5619595ae960 2023-11-28T18:23:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa892ffd640 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae040 0x56195964cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa88bfff640 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae960 0x56195964d460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa89415b230 tx=0x7fa87c01ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07bde26a67f50f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 1 -- 172.21.15.7:0/554906719 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c01c920 con 0x5619595ae960 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- 172.21.15.7:0/554906719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619595a8590 con 0x5619595ae960 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 1 -- 172.21.15.7:0/554906719 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa87c004020 con 0x5619595ae960 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 1 -- 172.21.15.7:0/554906719 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa87c0041e0 con 0x5619595ae960 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 10 client.4719 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- 172.21.15.7:0/554906719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5619590f0cc0 con 0x5619595ae960 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 1 -- 172.21.15.7:0/554906719 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5619595a8590 con 0x5619595ae960 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 1 -- 172.21.15.7:0/554906719 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fa87c004340 con 0x5619595ae960 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 10 client.4719 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 10 client.4719 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T18:23:41.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 20 client.4719 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:41.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] conn(0x56195965a410 0x561959607d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 4 client.4719 mds_command: new command op to 4284 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T18:23:41.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- 172.21.15.7:0/554906719 --> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5619593f7110 con 0x56195965a410 2023-11-28T18:23:41.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa892ffd640 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] conn(0x56195965a410 0x561959607d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa892ffd640 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] conn(0x56195965a410 0x561959607d60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=20e7d2322392fd81 server_cookie=f9fbf317303d1dda in_seq=0 out_seq=0 2023-11-28T18:23:41.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 10 client.4719 ms_handle_connect on v2:172.21.15.7:6834/1419029356 2023-11-28T18:23:41.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 1 -- 172.21.15.7:0/554906719 <== mds.3 v2:172.21.15.7:6834/1419029356 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5619593f7110 con 0x56195965a410 2023-11-28T18:23:41.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 10 client.4719 handle_command_reply: tid=0 2023-11-28T18:23:41.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 client.4719 shutdown 2023-11-28T18:23:41.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- 172.21.15.7:0/554906719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae960 msgr2=0x56195964d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae960 0x56195964d460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa89415b230 tx=0x7fa87c01ced0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- 172.21.15.7:0/554906719 shutdown_connections 2023-11-28T18:23:41.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.7:6834/1419029356,v1:172.21.15.7:6835/1419029356] conn(0x56195965a410 0x561959607d60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x5619595e8fc0 0x561959607140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5619595ae960 0x56195964d460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 --2- 172.21.15.7:0/554906719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5619595ae040 0x56195964cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa871ffb640 0 client.4719 ms_handle_reset on v2:172.21.15.7:6834/1419029356 2023-11-28T18:23:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- 172.21.15.7:0/554906719 >> 172.21.15.7:0/554906719 conn(0x5619595e5840 msgr2=0x5619595a9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- 172.21.15.7:0/554906719 shutdown_connections 2023-11-28T18:23:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 1 -- 172.21.15.7:0/554906719 wait complete. 2023-11-28T18:23:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.507+0000 7fa89ca5f000 20 client.4719 trim_cache size 0 max 16384 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "b4396e54-0cbb-426d-8760-05a802ba48a2", 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 3, 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4284, 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 16.354072259999999, 2023-11-28T18:23:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 16, 2023-11-28T18:23:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 20, 2023-11-28T18:23:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 20, 2023-11-28T18:23:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 40.563777579000003 2023-11-28T18:23:41.526 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 -- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa874066900 msgr2=0x7fa874068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa874066900 0x7fa874068dc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa88401ec90 tx=0x7fa884008040 comp rx=0 tx=0).stop 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 -- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8810 msgr2=0x7fa89414c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8810 0x7fa89414c540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa88c002240 tx=0x7fa88c00cbe0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 -- 172.21.15.7:0/2993994119 shutdown_connections 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa874066900 0x7fa874068dc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8940c8810 0x7fa89414c540 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8940c8430 0x7fa8941ed970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 --2- 172.21.15.7:0/2993994119 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8940c2620 0x7fa8941f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 -- 172.21.15.7:0/2993994119 >> 172.21.15.7:0/2993994119 conn(0x7fa8940662e0 msgr2=0x7fa89414fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 -- 172.21.15.7:0/2993994119 shutdown_connections 2023-11-28T18:23:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.511+0000 7fa8996fe640 1 -- 172.21.15.7:0/2993994119 wait complete. 2023-11-28T18:23:41.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-28T18:23:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.671+0000 7f7321e17640 1 Processor -- start 2023-11-28T18:23:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.671+0000 7f7321e17640 1 -- start start 2023-11-28T18:23:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f7321e17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f731c0c2620 0x7f731c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f7321e17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c0c6a70 con 0x7f731c0c8430 2023-11-28T18:23:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f7321e17640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c157360 con 0x7f731c0c8810 2023-11-28T18:23:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f7321e17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731c0c7b30 con 0x7f731c0c2620 2023-11-28T18:23:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f731c0c8430 0x7f731c0c2f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44844/0 (socket says 172.21.15.7:44844) 2023-11-28T18:23:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731b7fe640 1 -- 172.21.15.7:0/1934069922 learned_addr learned my addr 172.21.15.7:0/1934069922 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731bfff640 1 --2- 172.21.15.7:0/1934069922 >> v2:172.21.15.193:3300/0 conn(0x7f731c0c2620 0x7f731c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731affd640 1 -- 172.21.15.7:0/1934069922 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152088388 0 0) 0x7f731c0c6a70 con 0x7f731c0c8430 2023-11-28T18:23:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731affd640 1 -- 172.21.15.7:0/1934069922 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7308003530 con 0x7f731c0c8430 2023-11-28T18:23:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731bfff640 1 -- 172.21.15.7:0/1934069922 >> v1:172.21.15.193:6789/0 conn(0x7f731c0c8810 legacy=0x7f731c156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731bfff640 1 -- 172.21.15.7:0/1934069922 >> v1:172.21.15.7:6789/0 conn(0x7f731c0c8430 legacy=0x7f731c0c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731affd640 1 -- 172.21.15.7:0/1934069922 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3719588062 0 0) 0x7f7308003530 con 0x7f731c0c8430 2023-11-28T18:23:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.675+0000 7f731bfff640 1 -- 172.21.15.7:0/1934069922 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0c7e10 con 0x7f731c0c2620 2023-11-28T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f731bfff640 1 --2- 172.21.15.7:0/1934069922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f731c0c2620 0x7f731c150ad0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7314009d90 tx=0x7f731401f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2accf4fc409c740b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f731affd640 1 -- 172.21.15.7:0/1934069922 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c001680 con 0x7f731c0c2620 2023-11-28T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f731affd640 1 -- 172.21.15.7:0/1934069922 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73140a8bf0 con 0x7f731c0c2620 2023-11-28T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f731affd640 1 -- 172.21.15.7:0/1934069922 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140b0440 con 0x7f731c0c2620 2023-11-28T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f7321e17640 1 -- 172.21.15.7:0/1934069922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f731c0c2620 msgr2=0x7f731c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f7321e17640 1 --2- 172.21.15.7:0/1934069922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f731c0c2620 0x7f731c150ad0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7314009d90 tx=0x7f731401f060 comp rx=0 tx=0).stop 2023-11-28T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f7321e17640 1 -- 172.21.15.7:0/1934069922 shutdown_connections 2023-11-28T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f7321e17640 1 --2- 172.21.15.7:0/1934069922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f731c0c2620 0x7f731c150ad0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f7321e17640 1 -- 172.21.15.7:0/1934069922 >> 172.21.15.7:0/1934069922 conn(0x7f731c0662e0 msgr2=0x7f731c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f7321e17640 1 -- 172.21.15.7:0/1934069922 shutdown_connections 2023-11-28T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f7321e17640 1 -- 172.21.15.7:0/1934069922 wait complete. 2023-11-28T18:23:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.679+0000 7f7321e17640 1 Processor -- start 2023-11-28T18:23:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 -- start start 2023-11-28T18:23:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f731c0c2620 0x7f731c153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f731c0c8430 0x7f731c1515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f731c0c8810 0x7f731c0c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c151ae0 con 0x7f731c0c8430 2023-11-28T18:23:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c151fc0 con 0x7f731c0c2620 2023-11-28T18:23:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c151ce0 con 0x7f731c0c8810 2023-11-28T18:23:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f731c0c8430 0x7f731c1515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f731c0c8430 0x7f731c1515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40152/0 (socket says 172.21.15.7:40152) 2023-11-28T18:23:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 -- 172.21.15.7:0/2205252795 learned_addr learned my addr 172.21.15.7:0/2205252795 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 -- 172.21.15.7:0/2205252795 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f731c0c8810 msgr2=0x7f731c0c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f731c0c8810 0x7f731c0c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 -- 172.21.15.7:0/2205252795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f731c0c2620 msgr2=0x7f731c153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f731c0c2620 0x7f731c153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 -- 172.21.15.7:0/2205252795 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314009a70 con 0x7f731c0c8430 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7313fff640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f731c0c8430 0x7f731c1515a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f730400c3c0 tx=0x7f730400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7318ff9640 1 -- 172.21.15.7:0/2205252795 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73040052d0 con 0x7f731c0c8430 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7318ff9640 1 -- 172.21.15.7:0/2205252795 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7304005430 con 0x7f731c0c8430 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 -- 172.21.15.7:0/2205252795 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c155cb0 con 0x7f731c0c8430 2023-11-28T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7318ff9640 1 -- 172.21.15.7:0/2205252795 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304003840 con 0x7f731c0c8430 2023-11-28T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.683+0000 7f7321e17640 1 -- 172.21.15.7:0/2205252795 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c156270 con 0x7f731c0c8430 2023-11-28T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f7318ff9640 1 -- 172.21.15.7:0/2205252795 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f73040039a0 con 0x7f731c0c8430 2023-11-28T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f7318ff9640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f72fc0668d0 0x7f72fc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f7318ff9640 1 -- 172.21.15.7:0/2205252795 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7304083ab0 con 0x7f731c0c8430 2023-11-28T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f72fc0668d0 0x7f72fc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f732517e000 1 Processor -- start 2023-11-28T18:23:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f732517e000 1 -- start start 2023-11-28T18:23:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f72fc0668d0 0x7f72fc068d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f730c001ec0 tx=0x7f730c001a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f732517e000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae31c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f732517e000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 0x55c2ae356a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f732517e000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f732517e000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae1dd110 con 0x55c2ae31c040 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f732517e000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae16d9a0 con 0x55c2ae31c960 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f732517e000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae16aed0 con 0x55c2ae356fc0 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae31c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae31c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40154/0 (socket says 172.21.15.7:40154) 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 -- 172.21.15.7:0/2791571365 learned_addr learned my addr 172.21.15.7:0/2791571365 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f7313fff640 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 0x55c2ae356a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731bfff640 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 -- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 msgr2=0x55c2ae3573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 -- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 msgr2=0x55c2ae356a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 0x55c2ae356a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 -- 172.21.15.7:0/2791571365 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2ade5e890 con 0x55c2ae31c040 2023-11-28T18:23:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731bfff640 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae31c420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f730c005c30 tx=0x7f730c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574cb6b3fdac4c2d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f73117fa640 1 -- 172.21.15.7:0/2791571365 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c01b070 con 0x55c2ae31c040 2023-11-28T18:23:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f73117fa640 1 -- 172.21.15.7:0/2791571365 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c007560 con 0x55c2ae31c040 2023-11-28T18:23:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.687+0000 7f73117fa640 1 -- 172.21.15.7:0/2791571365 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c0014a0 con 0x55c2ae31c040 2023-11-28T18:23:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- 172.21.15.7:0/2791571365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 msgr2=0x55c2ae31c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae31c420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f730c005c30 tx=0x7f730c00f000 comp rx=0 tx=0).stop 2023-11-28T18:23:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- 172.21.15.7:0/2791571365 shutdown_connections 2023-11-28T18:23:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 0x55c2ae356a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 --2- 172.21.15.7:0/2791571365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae31c420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- 172.21.15.7:0/2791571365 >> 172.21.15.7:0/2791571365 conn(0x55c2ae353840 msgr2=0x55c2ae355c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- 172.21.15.7:0/2791571365 shutdown_connections 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- 172.21.15.7:0/2791571365 wait complete. 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- start start 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 Processor -- start 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c040 0x55c2ae351db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c960 0x55c2ae3522f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3183a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae1667c0 con 0x55c2ae31c960 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae11ebf0 con 0x55c2ae31c040 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae124a90 con 0x55c2ae356fc0 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 10 client.0 _ll_register_callbacks cb 0x55c2ae301e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c960 0x55c2ae3522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c960 0x55c2ae3522f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40168/0 (socket says 172.21.15.7:40168) 2023-11-28T18:23:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 -- 172.21.15.7:0/2416349642 learned_addr learned my addr 172.21.15.7:0/2416349642 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c040 0x55c2ae351db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f731bfff640 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 -- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 msgr2=0x55c2ae3183a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3183a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 -- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c040 msgr2=0x55c2ae351db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c040 0x55c2ae351db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 -- 172.21.15.7:0/2416349642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c0058e0 con 0x55c2ae31c960 2023-11-28T18:23:41.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f7313fff640 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c960 0x55c2ae3522f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f730400c390 tx=0x7f73040549c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca69674ff2592d3d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f72fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:41.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2416349642 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304052ce0 con 0x55c2ae31c960 2023-11-28T18:23:41.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2416349642 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f730405c780 con 0x55c2ae31c960 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.691+0000 7f732517e000 1 -- 172.21.15.7:0/2416349642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2ae128470 con 0x55c2ae31c960 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2416349642 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f730405c940 con 0x55c2ae31c960 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 10 client.4731 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 -- 172.21.15.7:0/2416349642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55c2ae127190 con 0x55c2ae31c960 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2416349642 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7304070070 con 0x55c2ae31c960 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2416349642 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f7304055bb0 con 0x55c2ae31c960 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 10 client.4731 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 10 client.4731 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 20 client.4731 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] conn(0x55c2ae319140 0x55c2ae319520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 4 client.4731 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T18:23:41.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 -- 172.21.15.7:0/2416349642 --> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2ae12bb50 con 0x55c2ae319140 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] conn(0x55c2ae319140 0x55c2ae319520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f72fa7fc640 10 client.4731 ms_handle_connect on v2:172.21.15.193:6834/3911527535 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] conn(0x55c2ae319140 0x55c2ae319520 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=50cb9c188bbd020e server_cookie=34053c7cb7782bcb in_seq=0 out_seq=0 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2416349642 <== mds.2 v2:172.21.15.193:6834/3911527535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c2ae12bb50 con 0x55c2ae319140 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f72fa7fc640 10 client.4731 handle_command_reply: tid=0 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 client.4731 shutdown 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 -- 172.21.15.7:0/2416349642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c960 msgr2=0x55c2ae3522f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c960 0x55c2ae3522f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f730400c390 tx=0x7f73040549c0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 -- 172.21.15.7:0/2416349642 shutdown_connections 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] conn(0x55c2ae319140 0x55c2ae319520 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae356fc0 0x55c2ae3183a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c960 0x55c2ae3522f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 --2- 172.21.15.7:0/2416349642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c040 0x55c2ae351db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f72fa7fc640 0 client.4731 ms_handle_reset on v2:172.21.15.193:6834/3911527535 2023-11-28T18:23:41.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 -- 172.21.15.7:0/2416349642 >> 172.21.15.7:0/2416349642 conn(0x55c2ae353840 msgr2=0x55c2ae356870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 -- 172.21.15.7:0/2416349642 shutdown_connections 2023-11-28T18:23:41.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 1 -- 172.21.15.7:0/2416349642 wait complete. 2023-11-28T18:23:41.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.695+0000 7f732517e000 20 client.4731 trim_cache size 0 max 16384 2023-11-28T18:23:41.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f732517e000 1 Processor -- start 2023-11-28T18:23:41.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f732517e000 1 -- start start 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f732517e000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae317f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f732517e000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c960 0x55c2ae3184b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f732517e000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae356fc0 0x55c2ae3b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f732517e000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ade1f940 con 0x55c2ae31c040 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f732517e000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae11fd60 con 0x55c2ae356fc0 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f732517e000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae169320 con 0x55c2ae31c960 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f731b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae317f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f731b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae317f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40178/0 (socket says 172.21.15.7:40178) 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.711+0000 7f731b7fe640 1 -- 172.21.15.7:0/3151656997 learned_addr learned my addr 172.21.15.7:0/3151656997 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731bfff640 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae356fc0 0x55c2ae3b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f7313fff640 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c960 0x55c2ae3184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731b7fe640 1 -- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c960 msgr2=0x55c2ae3184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731b7fe640 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c960 0x55c2ae3184b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731b7fe640 1 -- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae356fc0 msgr2=0x55c2ae3b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731b7fe640 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae356fc0 0x55c2ae3b74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731b7fe640 1 -- 172.21.15.7:0/3151656997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2ae10d490 con 0x55c2ae31c040 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731bfff640 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae356fc0 0x55c2ae3b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f7313fff640 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c960 0x55c2ae3184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731b7fe640 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae317f70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f730c000c10 tx=0x7f730c007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25063770e265fad server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f7311ffb640 1 -- 172.21.15.7:0/3151656997 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c026040 con 0x55c2ae31c040 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f7311ffb640 1 -- 172.21.15.7:0/3151656997 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c00cc30 con 0x55c2ae31c040 2023-11-28T18:23:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f7311ffb640 1 -- 172.21.15.7:0/3151656997 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c02aa90 con 0x55c2ae31c040 2023-11-28T18:23:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- 172.21.15.7:0/3151656997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 msgr2=0x55c2ae317f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae317f70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f730c000c10 tx=0x7f730c007000 comp rx=0 tx=0).stop 2023-11-28T18:23:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- 172.21.15.7:0/3151656997 shutdown_connections 2023-11-28T18:23:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae356fc0 0x55c2ae3b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c960 0x55c2ae3184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 --2- 172.21.15.7:0/3151656997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae31c040 0x55c2ae317f70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- 172.21.15.7:0/3151656997 >> 172.21.15.7:0/3151656997 conn(0x55c2ae353840 msgr2=0x55c2ae314730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- 172.21.15.7:0/3151656997 shutdown_connections 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- 172.21.15.7:0/3151656997 wait complete. 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- start start 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 Processor -- start 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c040 0x55c2ae371b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 0x55c2ae37df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae356fc0 0x55c2ae37e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ade292a0 con 0x55c2ae356fc0 2023-11-28T18:23:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae1a7440 con 0x55c2ae31c960 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2ae162130 con 0x55c2ae31c040 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f732517e000 10 client.0 _ll_register_callbacks cb 0x55c2ae0ea6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae356fc0 0x55c2ae37e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae356fc0 0x55c2ae37e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40190/0 (socket says 172.21.15.7:40190) 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731bfff640 1 -- 172.21.15.7:0/2336336283 learned_addr learned my addr 172.21.15.7:0/2336336283 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f7313fff640 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 0x55c2ae37df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.715+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c040 0x55c2ae371b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f731bfff640 1 -- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c040 msgr2=0x55c2ae371b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f731bfff640 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c040 0x55c2ae371b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f731bfff640 1 -- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 msgr2=0x55c2ae37df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f731bfff640 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 0x55c2ae37df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f731bfff640 1 -- 172.21.15.7:0/2336336283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2ade5e890 con 0x55c2ae356fc0 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f731bfff640 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae356fc0 0x55c2ae37e470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7314009d90 tx=0x7f731401be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd896b021bbb4fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2336336283 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140c1070 con 0x55c2ae356fc0 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2336336283 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7314004180 con 0x55c2ae356fc0 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 -- 172.21.15.7:0/2336336283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2ae165110 con 0x55c2ae356fc0 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2336336283 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7314004340 con 0x55c2ae356fc0 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 10 client.4737 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 -- 172.21.15.7:0/2336336283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55c2ade5ecc0 con 0x55c2ae356fc0 2023-11-28T18:23:41.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2336336283 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c2ae165110 con 0x55c2ae356fc0 2023-11-28T18:23:41.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2336336283 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f73140044a0 con 0x55c2ae356fc0 2023-11-28T18:23:41.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 10 client.4737 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:41.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 10 client.4737 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-28T18:23:41.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 20 client.4737 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:41.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] conn(0x55c2ae3bd320 0x55c2ae3bd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 4 client.4737 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T18:23:41.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 -- 172.21.15.7:0/2336336283 --> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c2ae193650 con 0x55c2ae3bd320 2023-11-28T18:23:41.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] conn(0x55c2ae3bd320 0x55c2ae3bd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f731b7fe640 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] conn(0x55c2ae3bd320 0x55c2ae3bd700 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca438e56096086d2 server_cookie=827581e457718944 in_seq=0 out_seq=0 2023-11-28T18:23:41.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 10 client.4737 ms_handle_connect on v2:172.21.15.193:6834/3911527535 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 1 -- 172.21.15.7:0/2336336283 <== mds.2 v2:172.21.15.193:6834/3911527535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55c2ae193650 con 0x55c2ae3bd320 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 10 client.4737 handle_command_reply: tid=0 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 client.4737 shutdown 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 -- 172.21.15.7:0/2336336283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae356fc0 msgr2=0x55c2ae37e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae356fc0 0x55c2ae37e470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7314009d90 tx=0x7f731401be00 comp rx=0 tx=0).stop 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 -- 172.21.15.7:0/2336336283 shutdown_connections 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:6834/3911527535,v1:172.21.15.193:6837/3911527535] conn(0x55c2ae3bd320 0x55c2ae3bd700 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c2ae356fc0 0x55c2ae37e470 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2ae31c960 0x55c2ae37df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 --2- 172.21.15.7:0/2336336283 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x55c2ae31c040 0x55c2ae371b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f72fa7fc640 0 client.4737 ms_handle_reset on v2:172.21.15.193:6834/3911527535 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 -- 172.21.15.7:0/2336336283 >> 172.21.15.7:0/2336336283 conn(0x55c2ae353840 msgr2=0x55c2ae314fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 -- 172.21.15.7:0/2336336283 shutdown_connections 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 1 -- 172.21.15.7:0/2336336283 wait complete. 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.719+0000 7f732517e000 20 client.4737 trim_cache size 0 max 16384 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "b4396e54-0cbb-426d-8760-05a802ba48a2", 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 2, 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4285, 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 22.587659357, 2023-11-28T18:23:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 16, 2023-11-28T18:23:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 20, 2023-11-28T18:23:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 20, 2023-11-28T18:23:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 40.762381910000002 2023-11-28T18:23:41.732 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-28T18:23:41.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 -- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f72fc0668d0 msgr2=0x7f72fc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f72fc0668d0 0x7f72fc068d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f730c001ec0 tx=0x7f730c001a10 comp rx=0 tx=0).stop 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 -- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f731c0c8430 msgr2=0x7f731c1515a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f731c0c8430 0x7f731c1515a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f730400c3c0 tx=0x7f730400c880 comp rx=0 tx=0).stop 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 -- 172.21.15.7:0/2205252795 shutdown_connections 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f72fc0668d0 0x7f72fc068d90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f731c0c8810 0x7f731c0c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f731c0c8430 0x7f731c1515a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 --2- 172.21.15.7:0/2205252795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f731c0c2620 0x7f731c153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 -- 172.21.15.7:0/2205252795 >> 172.21.15.7:0/2205252795 conn(0x7f731c0662e0 msgr2=0x7f731c14e560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 -- 172.21.15.7:0/2205252795 shutdown_connections 2023-11-28T18:23:41.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.723+0000 7f7321e17640 1 -- 172.21.15.7:0/2205252795 wait complete. 2023-11-28T18:23:41.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T18:23:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f20555d0640 1 Processor -- start 2023-11-28T18:23:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f20555d0640 1 -- start start 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f20555d0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ca30 0x7f2050159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f20555d0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205015a620 con 0x7f20500cafe0 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f20555d0640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205015a900 con 0x7f205015ce10 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f20555d0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205015abe0 con 0x7f205015ca30 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f2054dcf640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f20500cafe0 0x7f20500cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44848/0 (socket says 172.21.15.7:44848) 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f2054dcf640 1 -- 172.21.15.7:0/3173993937 learned_addr learned my addr 172.21.15.7:0/3173993937 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204ffff640 1 --2- 172.21.15.7:0/3173993937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ca30 0x7f2050159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204effd640 1 -- 172.21.15.7:0/3173993937 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543354530 0 0) 0x7f205015a620 con 0x7f20500cafe0 2023-11-28T18:23:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204effd640 1 -- 172.21.15.7:0/3173993937 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2038003530 con 0x7f20500cafe0 2023-11-28T18:23:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204effd640 1 -- 172.21.15.7:0/3173993937 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453395759 0 0) 0x7f205015a900 con 0x7f205015ce10 2023-11-28T18:23:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204effd640 1 -- 172.21.15.7:0/3173993937 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205015a620 con 0x7f205015ce10 2023-11-28T18:23:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204effd640 1 -- 172.21.15.7:0/3173993937 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154608157 0 0) 0x7f2038003530 con 0x7f20500cafe0 2023-11-28T18:23:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204ffff640 1 -- 172.21.15.7:0/3173993937 >> v1:172.21.15.193:6789/0 conn(0x7f205015ce10 legacy=0x7f20500ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204ffff640 1 -- 172.21.15.7:0/3173993937 >> v1:172.21.15.7:6789/0 conn(0x7f20500cafe0 legacy=0x7f20500cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.887+0000 7f204ffff640 1 -- 172.21.15.7:0/3173993937 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500cbb10 con 0x7f205015ca30 2023-11-28T18:23:41.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f204ffff640 1 --2- 172.21.15.7:0/3173993937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ca30 0x7f2050159c90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20400097f0 tx=0x7f204001ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8799b3db597964f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f204effd640 1 -- 172.21.15.7:0/3173993937 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2048001a00 con 0x7f205015ca30 2023-11-28T18:23:41.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f204effd640 1 -- 172.21.15.7:0/3173993937 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20400a8650 con 0x7f205015ca30 2023-11-28T18:23:41.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f204effd640 1 -- 172.21.15.7:0/3173993937 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20400a89e0 con 0x7f205015ca30 2023-11-28T18:23:41.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f20555d0640 1 -- 172.21.15.7:0/3173993937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ca30 msgr2=0x7f2050159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f20555d0640 1 --2- 172.21.15.7:0/3173993937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ca30 0x7f2050159c90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20400097f0 tx=0x7f204001ebd0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f20555d0640 1 -- 172.21.15.7:0/3173993937 shutdown_connections 2023-11-28T18:23:41.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f20555d0640 1 --2- 172.21.15.7:0/3173993937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ca30 0x7f2050159c90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f20555d0640 1 -- 172.21.15.7:0/3173993937 >> 172.21.15.7:0/3173993937 conn(0x7f205006c060 msgr2=0x7f20500662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f20555d0640 1 -- 172.21.15.7:0/3173993937 shutdown_connections 2023-11-28T18:23:41.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.891+0000 7f20555d0640 1 -- 172.21.15.7:0/3173993937 wait complete. 2023-11-28T18:23:41.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 Processor -- start 2023-11-28T18:23:41.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 -- start start 2023-11-28T18:23:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20500cafe0 0x7f2050153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205015ca30 0x7f2050155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ce10 0x7f20501563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500c3a70 con 0x7f20500cafe0 2023-11-28T18:23:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205014e980 con 0x7f205015ca30 2023-11-28T18:23:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205014ec60 con 0x7f205015ce10 2023-11-28T18:23:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20500cafe0 0x7f2050153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20500cafe0 0x7f2050153010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40204/0 (socket says 172.21.15.7:40204) 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 -- 172.21.15.7:0/2984026688 learned_addr learned my addr 172.21.15.7:0/2984026688 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 -- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ce10 msgr2=0x7f20501563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f2054dcf640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ce10 0x7f20501563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204f7fe640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205015ca30 0x7f2050155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ce10 0x7f20501563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 -- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205015ca30 msgr2=0x7f2050155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205015ca30 0x7f2050155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 -- 172.21.15.7:0/2984026688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2040009420 con 0x7f20500cafe0 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f2054dcf640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ce10 0x7f20501563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204f7fe640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205015ca30 0x7f2050155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:41.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204ffff640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20500cafe0 0x7f2050153010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f20400097f0 tx=0x7f204001b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204cff9640 1 -- 172.21.15.7:0/2984026688 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20400bf070 con 0x7f20500cafe0 2023-11-28T18:23:41.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204cff9640 1 -- 172.21.15.7:0/2984026688 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040006e20 con 0x7f20500cafe0 2023-11-28T18:23:41.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 -- 172.21.15.7:0/2984026688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205014ef40 con 0x7f20500cafe0 2023-11-28T18:23:41.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f204cff9640 1 -- 172.21.15.7:0/2984026688 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20400b0870 con 0x7f20500cafe0 2023-11-28T18:23:41.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.895+0000 7f20555d0640 1 -- 172.21.15.7:0/2984026688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205014f500 con 0x7f20500cafe0 2023-11-28T18:23:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.899+0000 7f204cff9640 1 -- 172.21.15.7:0/2984026688 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f20400b7040 con 0x7f20500cafe0 2023-11-28T18:23:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.899+0000 7f204cff9640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f203c066900 0x7f203c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.899+0000 7f204cff9640 1 -- 172.21.15.7:0/2984026688 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2040121220 con 0x7f20500cafe0 2023-11-28T18:23:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.899+0000 7f204f7fe640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f203c066900 0x7f203c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.899+0000 7f2059ba3000 1 Processor -- start 2023-11-28T18:23:41.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.899+0000 7f2059ba3000 1 -- start start 2023-11-28T18:23:41.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.899+0000 7f204f7fe640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f203c066900 0x7f203c068dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f205015b740 tx=0x7f2044009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 0x559f9cf9e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e960 0x559f9cfd8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cfd8fc0 0x559f9cfd93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f9ce5f110 con 0x559f9cf9e960 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f9cdef9a0 con 0x559f9cf9e040 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x559f9cdeced0 con 0x559f9cfd8fc0 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e960 0x559f9cfd8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e960 0x559f9cfd8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40218/0 (socket says 172.21.15.7:40218) 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 -- 172.21.15.7:0/1482305110 learned_addr learned my addr 172.21.15.7:0/1482305110 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204ffff640 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 0x559f9cf9e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2054dcf640 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cfd8fc0 0x559f9cfd93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 -- 172.21.15.7:0/1482305110 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cfd8fc0 msgr2=0x559f9cfd93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cfd8fc0 0x559f9cfd93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 -- 172.21.15.7:0/1482305110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 msgr2=0x559f9cf9e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 0x559f9cf9e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 -- 172.21.15.7:0/1482305110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f9cae0890 con 0x559f9cf9e960 2023-11-28T18:23:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f204f7fe640 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e960 0x559f9cfd8a80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2044001e20 tx=0x7f2044010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92db7fa9c5c94753 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2035ffb640 1 -- 172.21.15.7:0/1482305110 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044014a20 con 0x559f9cf9e960 2023-11-28T18:23:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2035ffb640 1 -- 172.21.15.7:0/1482305110 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2044014b80 con 0x559f9cf9e960 2023-11-28T18:23:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2035ffb640 1 -- 172.21.15.7:0/1482305110 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204401f400 con 0x559f9cf9e960 2023-11-28T18:23:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- 172.21.15.7:0/1482305110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e960 msgr2=0x559f9cfd8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e960 0x559f9cfd8a80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2044001e20 tx=0x7f2044010000 comp rx=0 tx=0).stop 2023-11-28T18:23:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- 172.21.15.7:0/1482305110 shutdown_connections 2023-11-28T18:23:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cfd8fc0 0x559f9cfd93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e960 0x559f9cfd8a80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1482305110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 0x559f9cf9e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- 172.21.15.7:0/1482305110 >> 172.21.15.7:0/1482305110 conn(0x559f9cfd5840 msgr2=0x559f9cfd7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- 172.21.15.7:0/1482305110 shutdown_connections 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- 172.21.15.7:0/1482305110 wait complete. 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 -- start start 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.903+0000 7f2059ba3000 1 Processor -- start 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 0x559f9cfd3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 0x559f9cfd42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd8fc0 0x559f9cf9a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f9cde87c0 con 0x559f9cfd8fc0 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f9cda0bf0 con 0x559f9cf9e040 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x559f9cda6a90 con 0x559f9cf9e960 2023-11-28T18:23:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 10 client.0 _ll_register_callbacks cb 0x559f9cf83e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd8fc0 0x559f9cf9a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd8fc0 0x559f9cf9a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40228/0 (socket says 172.21.15.7:40228) 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 -- 172.21.15.7:0/1978338454 learned_addr learned my addr 172.21.15.7:0/1978338454 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f204f7fe640 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 0x559f9cfd42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f204ffff640 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 0x559f9cfd3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 -- 172.21.15.7:0/1978338454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 msgr2=0x559f9cfd42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 0x559f9cfd42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 -- 172.21.15.7:0/1978338454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 msgr2=0x559f9cfd3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 0x559f9cfd3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 -- 172.21.15.7:0/1978338454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205015a620 con 0x559f9cfd8fc0 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2054dcf640 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd8fc0 0x559f9cf9a340 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f20500cb490 tx=0x7f204800fe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6583c195c45341a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 1 -- 172.21.15.7:0/1978338454 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204800da60 con 0x559f9cfd8fc0 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 1 -- 172.21.15.7:0/1978338454 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f204800dbc0 con 0x559f9cfd8fc0 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 -- 172.21.15.7:0/1978338454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f9cdaa470 con 0x559f9cfd8fc0 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 1 -- 172.21.15.7:0/1978338454 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2048001dd0 con 0x559f9cfd8fc0 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 10 client.4749 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 -- 172.21.15.7:0/1978338454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x559f9cda9190 con 0x559f9cfd8fc0 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 1 -- 172.21.15.7:0/1978338454 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559f9cdaa470 con 0x559f9cfd8fc0 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 1 -- 172.21.15.7:0/1978338454 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f2048012070 con 0x559f9cfd8fc0 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 10 client.4749 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 10 client.4749 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 20 client.4749 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:41.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] conn(0x559f9cf9b0e0 0x559f9cf9b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 4 client.4749 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T18:23:41.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 -- 172.21.15.7:0/1978338454 --> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f9cdadb50 con 0x559f9cf9b0e0 2023-11-28T18:23:41.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f204ffff640 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] conn(0x559f9cf9b0e0 0x559f9cf9b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f204ffff640 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] conn(0x559f9cf9b0e0 0x559f9cf9b4c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f65bc6f82d5d82d server_cookie=1047bd5ceaa3a63c in_seq=0 out_seq=0 2023-11-28T18:23:41.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 10 client.4749 ms_handle_connect on v2:172.21.15.7:6836/2690556675 2023-11-28T18:23:41.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 1 -- 172.21.15.7:0/1978338454 <== mds.1 v2:172.21.15.7:6836/2690556675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559f9cdadb50 con 0x559f9cf9b0e0 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f202effd640 10 client.4749 handle_command_reply: tid=0 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 client.4749 shutdown 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 -- 172.21.15.7:0/1978338454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd8fc0 msgr2=0x559f9cf9a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.907+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd8fc0 0x559f9cf9a340 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f20500cb490 tx=0x7f204800fe80 comp rx=0 tx=0).stop 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 1 -- 172.21.15.7:0/1978338454 shutdown_connections 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] conn(0x559f9cf9b0e0 0x559f9cf9b4c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd8fc0 0x559f9cf9a340 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 0x559f9cfd42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 1 --2- 172.21.15.7:0/1978338454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e040 0x559f9cfd3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f202effd640 0 client.4749 ms_handle_reset on v2:172.21.15.7:6836/2690556675 2023-11-28T18:23:41.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 1 -- 172.21.15.7:0/1978338454 >> 172.21.15.7:0/1978338454 conn(0x559f9cfd5840 msgr2=0x559f9cfd8870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 1 -- 172.21.15.7:0/1978338454 shutdown_connections 2023-11-28T18:23:41.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 1 -- 172.21.15.7:0/1978338454 wait complete. 2023-11-28T18:23:41.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.911+0000 7f2059ba3000 20 client.4749 trim_cache size 0 max 16384 2023-11-28T18:23:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.923+0000 7f2059ba3000 1 Processor -- start 2023-11-28T18:23:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.923+0000 7f2059ba3000 1 -- start start 2023-11-28T18:23:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cfd5840 0x559f9d034ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e040 0x559f9d035410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 0x559f9d035950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f9caa1940 con 0x559f9cf9e040 2023-11-28T18:23:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f9cfd3880 con 0x559f9cfd5840 2023-11-28T18:23:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x559f9cdbdb70 con 0x559f9cf9e960 2023-11-28T18:23:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e040 0x559f9d035410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e040 0x559f9d035410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40238/0 (socket says 172.21.15.7:40238) 2023-11-28T18:23:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 -- 172.21.15.7:0/788168187 learned_addr learned my addr 172.21.15.7:0/788168187 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204ffff640 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cfd5840 0x559f9d034ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2054dcf640 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 0x559f9d035950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 -- 172.21.15.7:0/788168187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 msgr2=0x559f9d035950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 0x559f9d035950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 -- 172.21.15.7:0/788168187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cfd5840 msgr2=0x559f9d034ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cfd5840 0x559f9d034ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 -- 172.21.15.7:0/788168187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f9cd8f490 con 0x559f9cf9e040 2023-11-28T18:23:41.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f204f7fe640 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e040 0x559f9d035410 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f204401f000 tx=0x7f2044012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a56419cc083b1a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f20367fc640 1 -- 172.21.15.7:0/788168187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20440052d0 con 0x559f9cf9e040 2023-11-28T18:23:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f20367fc640 1 -- 172.21.15.7:0/788168187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2044018070 con 0x559f9cf9e040 2023-11-28T18:23:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f20367fc640 1 -- 172.21.15.7:0/788168187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044014820 con 0x559f9cf9e040 2023-11-28T18:23:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- 172.21.15.7:0/788168187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e040 msgr2=0x559f9d035410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e040 0x559f9d035410 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f204401f000 tx=0x7f2044012000 comp rx=0 tx=0).stop 2023-11-28T18:23:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- 172.21.15.7:0/788168187 shutdown_connections 2023-11-28T18:23:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e960 0x559f9d035950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cf9e040 0x559f9d035410 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 --2- 172.21.15.7:0/788168187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cfd5840 0x559f9d034ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- 172.21.15.7:0/788168187 >> 172.21.15.7:0/788168187 conn(0x559f9cfd8be0 msgr2=0x559f9cf96730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- 172.21.15.7:0/788168187 shutdown_connections 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- 172.21.15.7:0/788168187 wait complete. 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 -- start start 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.927+0000 7f2059ba3000 1 Processor -- start 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e040 0x559f9cff3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e960 0x559f9cffff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd5840 0x559f9d000470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f9cda1d60 con 0x559f9cfd5840 2023-11-28T18:23:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f9cae11b0 con 0x559f9cf9e960 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x559f9cd838a0 con 0x559f9cf9e040 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 10 client.0 _ll_register_callbacks cb 0x559f9cd6c6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd5840 0x559f9d000470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd5840 0x559f9d000470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40254/0 (socket says 172.21.15.7:40254) 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 -- 172.21.15.7:0/223303069 learned_addr learned my addr 172.21.15.7:0/223303069 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f204ffff640 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e040 0x559f9cff3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f204f7fe640 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e960 0x559f9cffff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 -- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e040 msgr2=0x559f9cff3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e040 0x559f9cff3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 -- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e960 msgr2=0x559f9cffff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e960 0x559f9cffff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 -- 172.21.15.7:0/223303069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f9cae0890 con 0x559f9cfd5840 2023-11-28T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f204f7fe640 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e960 0x559f9cffff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2054dcf640 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd5840 0x559f9d000470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2048001590 tx=0x7f2048013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ff5d6c36a2fe25 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 1 -- 172.21.15.7:0/223303069 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204801da70 con 0x559f9cfd5840 2023-11-28T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 1 -- 172.21.15.7:0/223303069 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20480011d0 con 0x559f9cfd5840 2023-11-28T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- 172.21.15.7:0/223303069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f9cde7110 con 0x559f9cfd5840 2023-11-28T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 1 -- 172.21.15.7:0/223303069 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f204800d520 con 0x559f9cfd5840 2023-11-28T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 10 client.4755 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- 172.21.15.7:0/223303069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x559f9cae0cc0 con 0x559f9cfd5840 2023-11-28T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 1 -- 172.21.15.7:0/223303069 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559f9cde7110 con 0x559f9cfd5840 2023-11-28T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 1 -- 172.21.15.7:0/223303069 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f204801dbd0 con 0x559f9cfd5840 2023-11-28T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 10 client.4755 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 10 client.4755 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 20 client.4755 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] conn(0x559f9cfd8fc0 0x559f9d03f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 4 client.4755 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T18:23:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- 172.21.15.7:0/223303069 --> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559f9ce15650 con 0x559f9cfd8fc0 2023-11-28T18:23:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f204ffff640 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] conn(0x559f9cfd8fc0 0x559f9d03f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f204ffff640 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] conn(0x559f9cfd8fc0 0x559f9d03f320 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7dfe7cbaa565888a server_cookie=aa5def5d50a683d3 in_seq=0 out_seq=0 2023-11-28T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 10 client.4755 ms_handle_connect on v2:172.21.15.7:6836/2690556675 2023-11-28T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 1 -- 172.21.15.7:0/223303069 <== mds.1 v2:172.21.15.7:6836/2690556675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559f9ce15650 con 0x559f9cfd8fc0 2023-11-28T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 10 client.4755 handle_command_reply: tid=0 2023-11-28T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 client.4755 shutdown 2023-11-28T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- 172.21.15.7:0/223303069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd5840 msgr2=0x559f9d000470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd5840 0x559f9d000470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2048001590 tx=0x7f2048013000 comp rx=0 tx=0).stop 2023-11-28T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- 172.21.15.7:0/223303069 shutdown_connections 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.7:6836/2690556675,v1:172.21.15.7:6838/2690556675] conn(0x559f9cfd8fc0 0x559f9d03f320 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559f9cfd5840 0x559f9d000470 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f9cf9e960 0x559f9cffff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 --2- 172.21.15.7:0/223303069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x559f9cf9e040 0x559f9cff3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f202effd640 0 client.4755 ms_handle_reset on v2:172.21.15.7:6836/2690556675 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- 172.21.15.7:0/223303069 >> 172.21.15.7:0/223303069 conn(0x559f9cfd8be0 msgr2=0x559f9cf96fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- 172.21.15.7:0/223303069 shutdown_connections 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.931+0000 7f2059ba3000 1 -- 172.21.15.7:0/223303069 wait complete. 2023-11-28T18:23:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f2059ba3000 20 client.4755 trim_cache size 0 max 16384 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "b4396e54-0cbb-426d-8760-05a802ba48a2", 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 1, 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4288, 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 28.235418326000001, 2023-11-28T18:23:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 16, 2023-11-28T18:23:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 20, 2023-11-28T18:23:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 20, 2023-11-28T18:23:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 40.970902508000002 2023-11-28T18:23:41.948 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-28T18:23:41.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 -- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f203c066900 msgr2=0x7f203c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f203c066900 0x7f203c068dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f205015b740 tx=0x7f2044009040 comp rx=0 tx=0).stop 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 -- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20500cafe0 msgr2=0x7f2050153010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20500cafe0 0x7f2050153010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f20400097f0 tx=0x7f204001b970 comp rx=0 tx=0).stop 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 -- 172.21.15.7:0/2984026688 shutdown_connections 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f203c066900 0x7f203c068dc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f205015ce10 0x7f20501563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205015ca30 0x7f2050155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 --2- 172.21.15.7:0/2984026688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20500cafe0 0x7f2050153010 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 -- 172.21.15.7:0/2984026688 >> 172.21.15.7:0/2984026688 conn(0x7f205006c060 msgr2=0x7f2050158610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 -- 172.21.15.7:0/2984026688 shutdown_connections 2023-11-28T18:23:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:41.935+0000 7f20555d0640 1 -- 172.21.15.7:0/2984026688 wait complete. 2023-11-28T18:23:41.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-11-28T18:23:42.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.095+0000 7fbf64b2a640 1 Processor -- start 2023-11-28T18:23:42.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.095+0000 7fbf64b2a640 1 -- start start 2023-11-28T18:23:42.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf64b2a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbf60151c40 0x7fbf600c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf64b2a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf601589d0 con 0x7fbf600c99f0 2023-11-28T18:23:42.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf64b2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf60157e00 con 0x7fbf600c50a0 2023-11-28T18:23:42.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf64b2a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf601580a0 con 0x7fbf60151c40 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5ed9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbf600c99f0 0x7fbf6015b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44858/0 (socket says 172.21.15.7:44858) 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2841206403 learned_addr learned my addr 172.21.15.7:0/2841206403 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/2841206403 >> v2:172.21.15.193:3300/0 conn(0x7fbf60151c40 0x7fbf600c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5dd9b640 1 -- 172.21.15.7:0/2841206403 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983807347 0 0) 0x7fbf601589d0 con 0x7fbf600c99f0 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5dd9b640 1 -- 172.21.15.7:0/2841206403 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf4c003530 con 0x7fbf600c99f0 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5dd9b640 1 -- 172.21.15.7:0/2841206403 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156927558 0 0) 0x7fbf60157e00 con 0x7fbf600c50a0 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5e59c640 1 -- 172.21.15.7:0/2841206403 >> v1:172.21.15.193:6789/0 conn(0x7fbf600c50a0 legacy=0x7fbf600cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5e59c640 1 -- 172.21.15.7:0/2841206403 >> v1:172.21.15.7:6789/0 conn(0x7fbf600c99f0 legacy=0x7fbf6015b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.099+0000 7fbf5e59c640 1 -- 172.21.15.7:0/2841206403 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60158380 con 0x7fbf60151c40 2023-11-28T18:23:42.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/2841206403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf60151c40 0x7fbf600c4b60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbf500097c0 tx=0x7fbf5001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e33c216c747fea16 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf5dd9b640 1 -- 172.21.15.7:0/2841206403 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50004230 con 0x7fbf60151c40 2023-11-28T18:23:42.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf5dd9b640 1 -- 172.21.15.7:0/2841206403 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf500a7050 con 0x7fbf60151c40 2023-11-28T18:23:42.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf5dd9b640 1 -- 172.21.15.7:0/2841206403 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50003620 con 0x7fbf60151c40 2023-11-28T18:23:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2841206403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf60151c40 msgr2=0x7fbf600c4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 --2- 172.21.15.7:0/2841206403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf60151c40 0x7fbf600c4b60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbf500097c0 tx=0x7fbf5001c480 comp rx=0 tx=0).stop 2023-11-28T18:23:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2841206403 shutdown_connections 2023-11-28T18:23:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 --2- 172.21.15.7:0/2841206403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf60151c40 0x7fbf600c4b60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2841206403 >> 172.21.15.7:0/2841206403 conn(0x7fbf6006ff10 msgr2=0x7fbf60070320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:42.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2841206403 shutdown_connections 2023-11-28T18:23:42.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2841206403 wait complete. 2023-11-28T18:23:42.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 Processor -- start 2023-11-28T18:23:42.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.103+0000 7fbf64b2a640 1 -- start start 2023-11-28T18:23:42.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf64b2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600c50a0 0x7fbf601f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf64b2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600c99f0 0x7fbf60150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf64b2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf60151c40 0x7fbf6014b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf64b2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf601f1420 con 0x7fbf60151c40 2023-11-28T18:23:42.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf64b2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6014d3a0 con 0x7fbf600c99f0 2023-11-28T18:23:42.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf64b2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6014c4c0 con 0x7fbf600c50a0 2023-11-28T18:23:42.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf60151c40 0x7fbf6014b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf60151c40 0x7fbf6014b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40260/0 (socket says 172.21.15.7:40260) 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2255278265 learned_addr learned my addr 172.21.15.7:0/2255278265 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf57fff640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600c99f0 0x7fbf60150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600c50a0 0x7fbf601f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600c50a0 msgr2=0x7fbf601f0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600c50a0 0x7fbf601f0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600c99f0 msgr2=0x7fbf60150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600c99f0 0x7fbf60150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2255278265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50009420 con 0x7fbf60151c40 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600c50a0 0x7fbf601f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf57fff640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600c99f0 0x7fbf60150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:42.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf60151c40 0x7fbf6014b1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800cb40 tx=0x7fbf5800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf56ffd640 1 -- 172.21.15.7:0/2255278265 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5800aca0 con 0x7fbf60151c40 2023-11-28T18:23:42.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2255278265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6014c7a0 con 0x7fbf60151c40 2023-11-28T18:23:42.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf56ffd640 1 -- 172.21.15.7:0/2255278265 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5800ae00 con 0x7fbf60151c40 2023-11-28T18:23:42.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf56ffd640 1 -- 172.21.15.7:0/2255278265 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58012620 con 0x7fbf60151c40 2023-11-28T18:23:42.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.107+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2255278265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf601535c0 con 0x7fbf60151c40 2023-11-28T18:23:42.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf56ffd640 1 -- 172.21.15.7:0/2255278265 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbf58007070 con 0x7fbf60151c40 2023-11-28T18:23:42.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf56ffd640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf400668d0 0x7fbf40068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf56ffd640 1 -- 172.21.15.7:0/2255278265 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf58084560 con 0x7fbf60151c40 2023-11-28T18:23:42.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf400668d0 0x7fbf40068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf67ea6000 1 Processor -- start 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf67ea6000 1 -- start start 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf400668d0 0x7fbf40068d90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf500ad000 tx=0x7fbf500a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b040 0x555ab8f0b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b960 0x555ab8f45a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ab8dcc110 con 0x555ab8f45fc0 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ab8d5c9a0 con 0x555ab8f0b960 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ab8d59ed0 con 0x555ab8f0b040 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f463a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40274/0 (socket says 172.21.15.7:40274) 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/3588627063 learned_addr learned my addr 172.21.15.7:0/3588627063 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:42.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b040 0x555ab8f0b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf57fff640 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b960 0x555ab8f45a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/3588627063 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b040 msgr2=0x555ab8f0b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b040 0x555ab8f0b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/3588627063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b960 msgr2=0x555ab8f45a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b960 0x555ab8f45a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.111+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/3588627063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ab8a4d890 con 0x555ab8f45fc0 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f463a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbf5805d190 tx=0x7fbf58055c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43133cc33ba8a4c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf3ffff640 1 -- 172.21.15.7:0/3588627063 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5805d6c0 con 0x555ab8f45fc0 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf3ffff640 1 -- 172.21.15.7:0/3588627063 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf58053760 con 0x555ab8f45fc0 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf3ffff640 1 -- 172.21.15.7:0/3588627063 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58059120 con 0x555ab8f45fc0 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3588627063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 msgr2=0x555ab8f463a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f463a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbf5805d190 tx=0x7fbf58055c00 comp rx=0 tx=0).stop 2023-11-28T18:23:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3588627063 shutdown_connections 2023-11-28T18:23:42.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f463a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b960 0x555ab8f45a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3588627063 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b040 0x555ab8f0b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3588627063 >> 172.21.15.7:0/3588627063 conn(0x555ab8f42840 msgr2=0x555ab8f44c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:42.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3588627063 shutdown_connections 2023-11-28T18:23:42.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3588627063 wait complete. 2023-11-28T18:23:42.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- start start 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 Processor -- start 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f41180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b960 0x555ab8f416c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f45fc0 0x555ab8f07310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ab8d557c0 con 0x555ab8f0b960 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ab8d0dbf0 con 0x555ab8f0b040 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ab8d13a90 con 0x555ab8f45fc0 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf67ea6000 10 client.0 _ll_register_callbacks cb 0x555ab8ef0e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf57fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b960 0x555ab8f416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf57fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b960 0x555ab8f416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40276/0 (socket says 172.21.15.7:40276) 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf57fff640 1 -- 172.21.15.7:0/3526809981 learned_addr learned my addr 172.21.15.7:0/3526809981 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f45fc0 0x555ab8f07310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.115+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f41180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf57fff640 1 -- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f45fc0 msgr2=0x555ab8f07310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf57fff640 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f45fc0 0x555ab8f07310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf57fff640 1 -- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 msgr2=0x555ab8f41180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf57fff640 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f41180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf57fff640 1 -- 172.21.15.7:0/3526809981 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf601535c0 con 0x555ab8f0b960 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f41180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf57fff640 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b960 0x555ab8f416c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbf480011f0 tx=0x7fbf48001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da1e629699631bb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/3526809981 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48014400 con 0x555ab8f0b960 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/3526809981 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf48014560 con 0x555ab8f0b960 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3526809981 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ab8d17470 con 0x555ab8f0b960 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/3526809981 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf48014720 con 0x555ab8f0b960 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 10 client.4767 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3526809981 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x555ab8d16190 con 0x555ab8f0b960 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/3526809981 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555ab8d17470 con 0x555ab8f0b960 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/3526809981 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fbf48014880 con 0x555ab8f0b960 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 10 client.4767 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 10 client.4767 resolve_mds: resolved name 'e' to daemon mds.e 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 20 client.4767 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] conn(0x555ab8f08050 0x555ab8f08430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 4 client.4767 mds_command: new command op to 4291 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3526809981 --> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ab8d1ab50 con 0x555ab8f08050 2023-11-28T18:23:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] conn(0x555ab8f08050 0x555ab8f08430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] conn(0x555ab8f08050 0x555ab8f08430 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f33423a97b55d719 server_cookie=b53087c5358416a3 in_seq=0 out_seq=0 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 10 client.4767 ms_handle_connect on v2:172.21.15.7:6837/1819203827 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/3526809981 <== mds.0 v2:172.21.15.7:6837/1819203827 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555ab8d1ab50 con 0x555ab8f08050 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 10 client.4767 handle_command_reply: tid=0 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 client.4767 shutdown 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3526809981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b960 msgr2=0x555ab8f416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b960 0x555ab8f416c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbf480011f0 tx=0x7fbf48001690 comp rx=0 tx=0).stop 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3526809981 shutdown_connections 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] conn(0x555ab8f08050 0x555ab8f08430 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f45fc0 0x555ab8f07310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b960 0x555ab8f416c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/3526809981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f41180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3526809981 >> 172.21.15.7:0/3526809981 conn(0x555ab8f42840 msgr2=0x555ab8f45870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:42.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf3d7fa640 0 client.4767 ms_handle_reset on v2:172.21.15.7:6837/1819203827 2023-11-28T18:23:42.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3526809981 shutdown_connections 2023-11-28T18:23:42.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 1 -- 172.21.15.7:0/3526809981 wait complete. 2023-11-28T18:23:42.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.119+0000 7fbf67ea6000 20 client.4767 trim_cache size 0 max 16384 2023-11-28T18:23:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.135+0000 7fbf67ea6000 1 Processor -- start 2023-11-28T18:23:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.135+0000 7fbf67ea6000 1 -- start start 2023-11-28T18:23:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f065e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 0x555ab8f06b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f61e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ab8a0e940 con 0x555ab8f45fc0 2023-11-28T18:23:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ab8a4e1b0 con 0x555ab8f0b040 2023-11-28T18:23:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ab8d5d0c0 con 0x555ab8f0b960 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f61e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f61e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40286/0 (socket says 172.21.15.7:40286) 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2139524431 learned_addr learned my addr 172.21.15.7:0/2139524431 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2139524431 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 msgr2=0x555ab8f06b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 0x555ab8f06b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2139524431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 msgr2=0x555ab8f065e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f065e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 -- 172.21.15.7:0/2139524431 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ab8cfc490 con 0x555ab8f45fc0 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f61e20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf58055580 tx=0x7fbf58084050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c06ea67042a6a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf54ff9640 1 -- 172.21.15.7:0/2139524431 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58073310 con 0x555ab8f45fc0 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf54ff9640 1 -- 172.21.15.7:0/2139524431 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5805d6c0 con 0x555ab8f45fc0 2023-11-28T18:23:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf54ff9640 1 -- 172.21.15.7:0/2139524431 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58051ad0 con 0x555ab8f45fc0 2023-11-28T18:23:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- 172.21.15.7:0/2139524431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 msgr2=0x555ab8f61e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f61e20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf58055580 tx=0x7fbf58084050 comp rx=0 tx=0).stop 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- 172.21.15.7:0/2139524431 shutdown_connections 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f45fc0 0x555ab8f61e20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 0x555ab8f06b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/2139524431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f0b040 0x555ab8f065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- 172.21.15.7:0/2139524431 >> 172.21.15.7:0/2139524431 conn(0x555ab8f42840 msgr2=0x555ab8f05ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- 172.21.15.7:0/2139524431 shutdown_connections 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- 172.21.15.7:0/2139524431 wait complete. 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 -- start start 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 Processor -- start 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b040 0x555ab8fa9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 0x555ab8faa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.139+0000 7fbf67ea6000 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f45fc0 0x555ab8f64140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ab8a4e1b0 con 0x555ab8f0b040 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ab8d4fac0 con 0x555ab8f45fc0 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ab8cf08a0 con 0x555ab8f0b960 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 10 client.0 _ll_register_callbacks cb 0x555ab8cd96e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b040 0x555ab8fa9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b040 0x555ab8fa9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40300/0 (socket says 172.21.15.7:40300) 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 -- 172.21.15.7:0/1494262733 learned_addr learned my addr 172.21.15.7:0/1494262733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf57fff640 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 0x555ab8faa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5ed9d640 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f45fc0 0x555ab8f64140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 -- 172.21.15.7:0/1494262733 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 msgr2=0x555ab8faa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 0x555ab8faa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 -- 172.21.15.7:0/1494262733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f45fc0 msgr2=0x555ab8f64140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f45fc0 0x555ab8f64140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 -- 172.21.15.7:0/1494262733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ab8a4d890 con 0x555ab8f0b040 2023-11-28T18:23:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf5e59c640 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b040 0x555ab8fa9d10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf50019090 tx=0x7fbf5001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567954a695650aed server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-11-28T18:23:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/1494262733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf500c9dc0 con 0x555ab8f0b040 2023-11-28T18:23:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/1494262733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf500c92e0 con 0x555ab8f0b040 2023-11-28T18:23:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- 172.21.15.7:0/1494262733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ab8f05590 con 0x555ab8f0b040 2023-11-28T18:23:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/1494262733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf500c94a0 con 0x555ab8f0b040 2023-11-28T18:23:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 10 client.4773 fetch_fsmap learned FSMap version 16 2023-11-28T18:23:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- 172.21.15.7:0/1494262733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x555ab8a4dcc0 con 0x555ab8f0b040 2023-11-28T18:23:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/1494262733 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555ab8f05590 con 0x555ab8f0b040 2023-11-28T18:23:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/1494262733 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fbf500c9600 con 0x555ab8f0b040 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 10 client.4773 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 10 client.4773 resolve_mds: resolved name 'e' to daemon mds.e 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 20 client.4773 populate_metadata read hostname 'smithi007' 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] conn(0x555ab8fb7200 0x555ab8f64d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 4 client.4773 mds_command: new command op to 4291 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- 172.21.15.7:0/1494262733 --> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555ab8d54110 con 0x555ab8fb7200 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf57fff640 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] conn(0x555ab8fb7200 0x555ab8f64d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf57fff640 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] conn(0x555ab8fb7200 0x555ab8f64d20 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b810008164e533ed server_cookie=1b5161b38a85760f in_seq=0 out_seq=0 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 10 client.4773 ms_handle_connect on v2:172.21.15.7:6837/1819203827 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 1 -- 172.21.15.7:0/1494262733 <== mds.0 v2:172.21.15.7:6837/1819203827 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x555ab8d54110 con 0x555ab8fb7200 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 10 client.4773 handle_command_reply: tid=0 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 client.4773 shutdown 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- 172.21.15.7:0/1494262733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b040 msgr2=0x555ab8fa9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b040 0x555ab8fa9d10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf50019090 tx=0x7fbf5001acc0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- 172.21.15.7:0/1494262733 shutdown_connections 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.7:6837/1819203827,v1:172.21.15.7:6839/1819203827] conn(0x555ab8fb7200 0x555ab8f64d20 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ab8f45fc0 0x555ab8f64140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x555ab8f0b960 0x555ab8faa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 --2- 172.21.15.7:0/1494262733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555ab8f0b040 0x555ab8fa9d10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- 172.21.15.7:0/1494262733 >> 172.21.15.7:0/1494262733 conn(0x555ab8f42840 msgr2=0x555ab8f05ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf3d7fa640 0 client.4773 ms_handle_reset on v2:172.21.15.7:6837/1819203827 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- 172.21.15.7:0/1494262733 shutdown_connections 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 1 -- 172.21.15.7:0/1494262733 wait complete. 2023-11-28T18:23:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.143+0000 7fbf67ea6000 20 client.4773 trim_cache size 0 max 16384 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "b4396e54-0cbb-426d-8760-05a802ba48a2", 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4291, 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 33.492469370999999, 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 16, 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 20, 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 20, 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 41.181189719000002 2023-11-28T18:23:42.155 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-28T18:23:42.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf400668d0 msgr2=0x7fbf40068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf400668d0 0x7fbf40068d90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf500ad000 tx=0x7fbf500a6000 comp rx=0 tx=0).stop 2023-11-28T18:23:42.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf60151c40 msgr2=0x7fbf6014b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf60151c40 0x7fbf6014b1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800cb40 tx=0x7fbf5800a060 comp rx=0 tx=0).stop 2023-11-28T18:23:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2255278265 shutdown_connections 2023-11-28T18:23:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf400668d0 0x7fbf40068d90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf60151c40 0x7fbf6014b1a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600c99f0 0x7fbf60150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 --2- 172.21.15.7:0/2255278265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600c50a0 0x7fbf601f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2255278265 >> 172.21.15.7:0/2255278265 conn(0x7fbf6006ff10 msgr2=0x7fbf60159530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2255278265 shutdown_connections 2023-11-28T18:23:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.147+0000 7fbf64b2a640 1 -- 172.21.15.7:0/2255278265 wait complete. 2023-11-28T18:23:42.170 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-11-28T18:23:42.195 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-11-28T18:23:42.196 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-11-28T18:23:42.196 INFO:tasks.ceph_fuse:client.0 config is {} 2023-11-28T18:23:42.196 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T18:23:42.196 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:42.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:42.196 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2023-11-28T18:23:42.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:42.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2023-11-28T18:23:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:Cannot find device "ceph-brx" 2023-11-28T18:23:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:23:42.278 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-11-28T18:23:42.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T18:23:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb8d52a640 1 Processor -- start 2023-11-28T18:23:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb8d52a640 1 -- start start 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb8d52a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88154000 0x7feb880c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb8d52a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7feb880c6ba0 0x7feb880c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb8d52a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8815e0f0 con 0x7feb880c3180 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb8d52a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8815d520 con 0x7feb880c6ba0 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb8d52a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb880c35d0 con 0x7feb88154000 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb877fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feb880c3180 0x7feb88159e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44860/0 (socket says 172.21.15.7:44860) 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb877fe640 1 -- 172.21.15.7:0/1187066603 learned_addr learned my addr 172.21.15.7:0/1187066603 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb7ffff640 1 --2- 172.21.15.7:0/1187066603 >> v2:172.21.15.193:3300/0 conn(0x7feb880c6ba0 0x7feb880c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.455+0000 7feb86ffd640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88154000 0x7feb880c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138370327 0 0) 0x7feb8815e0f0 con 0x7feb880c3180 2023-11-28T18:23:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb74003530 con 0x7feb880c3180 2023-11-28T18:23:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3657905865 0 0) 0x7feb74003530 con 0x7feb880c3180 2023-11-28T18:23:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88154000 msgr2=0x7feb880c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88154000 0x7feb880c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 >> v2:172.21.15.193:3300/0 conn(0x7feb880c6ba0 msgr2=0x7feb880c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 --2- 172.21.15.7:0/1187066603 >> v2:172.21.15.193:3300/0 conn(0x7feb880c6ba0 0x7feb880c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb880c7c70 con 0x7feb880c3180 2023-11-28T18:23:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7feb80001a00 con 0x7feb880c3180 2023-11-28T18:23:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 >> v1:172.21.15.7:6789/0 conn(0x7feb880c3180 legacy=0x7feb88159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88154000 0x7feb74003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74003f50 0x7feb740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb74004d10 0x7feb74005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8815d520 con 0x7feb74004d10 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb880c35d0 con 0x7feb74003f50 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7400ba10 con 0x7feb88154000 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb877fe640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb74004d10 0x7feb74005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb86ffd640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88154000 0x7feb74003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb7ffff640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74003f50 0x7feb740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb877fe640 1 -- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88154000 msgr2=0x7feb74003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb877fe640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88154000 0x7feb74003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb877fe640 1 -- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74003f50 msgr2=0x7feb740047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb877fe640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74003f50 0x7feb740047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb877fe640 1 -- 172.21.15.7:0/1187066603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb74003530 con 0x7feb74004d10 2023-11-28T18:23:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.459+0000 7feb7ffff640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74003f50 0x7feb740047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb877fe640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb74004d10 0x7feb74005190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7feb8000a810 tx=0x7feb8001d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31d167e28eec5ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800ad630 con 0x7feb74004d10 2023-11-28T18:23:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb800a8070 con 0x7feb74004d10 2023-11-28T18:23:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb867fc640 1 -- 172.21.15.7:0/1187066603 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800ada30 con 0x7feb74004d10 2023-11-28T18:23:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- 172.21.15.7:0/1187066603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb74004d10 msgr2=0x7feb74005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 --2- 172.21.15.7:0/1187066603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb74004d10 0x7feb74005190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7feb8000a810 tx=0x7feb8001d320 comp rx=0 tx=0).stop 2023-11-28T18:23:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb877fe640 1 -- 172.21.15.7:0/1187066603 reap_dead start 2023-11-28T18:23:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- 172.21.15.7:0/1187066603 shutdown_connections 2023-11-28T18:23:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- 172.21.15.7:0/1187066603 >> 172.21.15.7:0/1187066603 conn(0x7feb880662e0 msgr2=0x7feb880666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- 172.21.15.7:0/1187066603 shutdown_connections 2023-11-28T18:23:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- 172.21.15.7:0/1187066603 wait complete. 2023-11-28T18:23:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 Processor -- start 2023-11-28T18:23:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- start start 2023-11-28T18:23:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74004d10 0x7feb8815c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88158fb0 0x7feb88070640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb88143f00 0x7feb88144380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8815c850 con 0x7feb88143f00 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb881f33e0 con 0x7feb74004d10 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.463+0000 7feb8d52a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb881f36b0 con 0x7feb88158fb0 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb88143f00 0x7feb88144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb88143f00 0x7feb88144380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40306/0 (socket says 172.21.15.7:40306) 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 -- 172.21.15.7:0/962725695 learned_addr learned my addr 172.21.15.7:0/962725695 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb7ffff640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88158fb0 0x7feb88070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb86ffd640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74004d10 0x7feb8815c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 -- 172.21.15.7:0/962725695 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88158fb0 msgr2=0x7feb88070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88158fb0 0x7feb88070640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 -- 172.21.15.7:0/962725695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74004d10 msgr2=0x7feb8815c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74004d10 0x7feb8815c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 -- 172.21.15.7:0/962725695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb88151140 con 0x7feb88143f00 2023-11-28T18:23:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb877fe640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb88143f00 0x7feb88144380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb8001d150 tx=0x7feb74005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb7f7fe640 1 -- 172.21.15.7:0/962725695 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800019c0 con 0x7feb88143f00 2023-11-28T18:23:42.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb7f7fe640 1 -- 172.21.15.7:0/962725695 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb80001f00 con 0x7feb88143f00 2023-11-28T18:23:42.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb7f7fe640 1 -- 172.21.15.7:0/962725695 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800ad630 con 0x7feb88143f00 2023-11-28T18:23:42.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb88151420 con 0x7feb88143f00 2023-11-28T18:23:42.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb880cb3b0 con 0x7feb88143f00 2023-11-28T18:23:42.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb880cb3b0 con 0x7feb88143f00 2023-11-28T18:23:42.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.467+0000 7feb7f7fe640 1 -- 172.21.15.7:0/962725695 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7feb800aa030 con 0x7feb88143f00 2023-11-28T18:23:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.471+0000 7feb7f7fe640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb68066900 0x7feb68068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.471+0000 7feb7f7fe640 1 -- 172.21.15.7:0/962725695 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb8011eb80 con 0x7feb88143f00 2023-11-28T18:23:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.471+0000 7feb86ffd640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb68066900 0x7feb68068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.471+0000 7feb86ffd640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb68066900 0x7feb68068dc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feb78009be0 tx=0x7feb78009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:42.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.475+0000 7feb7f7fe640 1 -- 172.21.15.7:0/962725695 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feb800ebc00 con 0x7feb88143f00 2023-11-28T18:23:42.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feb880cb3b0 con 0x7feb88143f00 2023-11-28T18:23:42.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb7f7fe640 1 -- 172.21.15.7:0/962725695 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7feb800edb30 con 0x7feb88143f00 2023-11-28T18:23:42.669 INFO:teuthology.orchestra.run.smithi007.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T18:23:42.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb68066900 msgr2=0x7feb68068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb68066900 0x7feb68068dc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feb78009be0 tx=0x7feb78009190 comp rx=0 tx=0).stop 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb88143f00 msgr2=0x7feb88144380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb88143f00 0x7feb88144380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb8001d150 tx=0x7feb74005800 comp rx=0 tx=0).stop 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 shutdown_connections 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb68066900 0x7feb68068dc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb88143f00 0x7feb88144380 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb88158fb0 0x7feb88070640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 --2- 172.21.15.7:0/962725695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb74004d10 0x7feb8815c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 >> 172.21.15.7:0/962725695 conn(0x7feb880662e0 msgr2=0x7feb8815a510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 shutdown_connections 2023-11-28T18:23:42.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:42.663+0000 7feb8d52a640 1 -- 172.21.15.7:0/962725695 wait complete. 2023-11-28T18:23:42.690 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T18:23:42.691 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T18:23:42.691 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2023-11-28T18:23:42.691 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T18:23:42.691 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:42.691 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T18:23:42.691 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T18:23:42.691 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T18:23:42.691 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T18:23:42.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:42.691 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2023-11-28T18:23:42.700 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2023-11-28T18:23:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42375sec preferred_lft 42375sec 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 02:42:b7:9a:61:cf brd ff:ff:ff:ff:ff:ff 2023-11-28T18:23:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-11-28T18:23:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-11-28T18:23:42.704 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T18:23:42.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:42.704 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-11-28T18:23:42.704 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-11-28T18:23:42.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add name ceph-brx type bridge 2023-11-28T18:23:42.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr flush dev ceph-brx 2023-11-28T18:23:42.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set ceph-brx up 2023-11-28T18:23:42.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T18:23:42.704 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-11-28T18:23:42.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:42.820 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T18:23:42.840 INFO:teuthology.orchestra.run.smithi007.stdout:1 2023-11-28T18:23:42.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:42.846 DEBUG:teuthology.orchestra.run.smithi007:> route 2023-11-28T18:23:42.912 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2023-11-28T18:23:42.912 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T18:23:42.913 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-11-28T18:23:42.913 INFO:teuthology.orchestra.run.smithi007.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-11-28T18:23:42.913 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-11-28T18:23:42.913 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T18:23:42.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:42.913 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-11-28T18:23:42.913 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-11-28T18:23:42.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T18:23:42.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T18:23:42.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T18:23:42.914 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-11-28T18:23:43.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:43.009 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2023-11-28T18:23:43.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:43.019 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list-id 2023-11-28T18:23:43.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:43.072 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-11-28T18:23:43.072 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-11-28T18:23:43.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T18:23:43.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-28T18:23:43.072 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-11-28T18:23:43.155 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-28T18:23:43.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:43.155 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-11-28T18:23:43.155 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-11-28T18:23:43.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-28T18:23:43.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T18:23:43.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-28T18:23:43.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-28T18:23:43.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-28T18:23:43.156 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-11-28T18:23:43.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:43.320 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-11-28T18:23:43.320 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-11-28T18:23:43.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2023-11-28T18:23:43.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T18:23:43.320 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-11-28T18:23:43.364 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-28T18:23:43.365 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T18:23:43.365 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:43.420 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T18:23:43.421 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T18:23:43.421 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:43.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2023-11-28T18:23:43.537 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T18:23:43.593 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2023-11-28T18:23:43.593 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2023-11-28T18:23:43.593 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2023-11-28T18:23:43.593 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2023-11-28T18:23:43.594 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2023-11-28T18:23:43.594 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2023-11-28T18:23:43.594 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2023-11-28T18:23:43.594 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2023-11-28T18:23:43.594 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2023-11-28T18:23:43.594 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2023-11-28T18:23:43.595 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2023-11-28T18:23:43.595 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2023-11-28T18:23:43.595 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T18:23:43.595 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2023-11-28T18:23:43.595 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2023-11-28T18:23:43.595 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2023-11-28T18:23:43.595 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2023-11-28T18:23:43.596 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2023-11-28T18:23:43.596 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2023-11-28T18:23:43.596 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2023-11-28T18:23:43.596 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2023-11-28T18:23:43.596 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2023-11-28T18:23:43.596 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2023-11-28T18:23:43.596 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2023-11-28T18:23:43.597 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T18:23:43.597 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T18:23:43.597 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T18:23:43.597 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T18:23:43.597 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2023-11-28T18:23:43.597 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T18:23:43.597 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T18:23:43.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:43.598 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2023-11-28T18:23:43.651 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T18:23:43.651 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-11-28T18:23:43.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2023-11-28T18:23:43.721 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T18:23:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2023-11-28T18:23:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2023-11-28T18:23:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2023-11-28T18:23:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2023-11-28T18:23:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2023-11-28T18:23:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2023-11-28T18:23:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2023-11-28T18:23:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2023-11-28T18:23:43.778 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T18:23:43.779 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T18:23:43.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:43.780 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2023-11-28T18:23:43.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:2023-11-28T18:23:43.799+0000 7f94c5245580 -1 init, newargv = 0x55703a94bca0 newargc=15 2023-11-28T18:23:43.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:ceph-fuse[44657]: starting ceph client 2023-11-28T18:23:43.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:ceph-fuse[44657]: starting fuse 2023-11-28T18:23:44.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2023-11-28T18:23:44.941 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T18:23:44.998 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2023-11-28T18:23:44.998 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2023-11-28T18:23:44.998 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2023-11-28T18:23:44.998 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2023-11-28T18:23:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T18:23:45.000 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T18:23:45.001 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2023-11-28T18:23:45.001 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T18:23:45.001 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T18:23:45.001 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:45.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:45.002 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2023-11-28T18:23:45.056 INFO:teuthology.orchestra.run.smithi007.stdout:43 2023-11-28T18:23:45.057 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-11-28T18:23:45.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo stdin-killer -- python3 -c ' 2023-11-28T18:23:45.057 DEBUG:teuthology.orchestra.run.smithi007:> import glob 2023-11-28T18:23:45.057 DEBUG:teuthology.orchestra.run.smithi007:> import re 2023-11-28T18:23:45.057 DEBUG:teuthology.orchestra.run.smithi007:> import os 2023-11-28T18:23:45.057 DEBUG:teuthology.orchestra.run.smithi007:> import subprocess 2023-11-28T18:23:45.057 DEBUG:teuthology.orchestra.run.smithi007:> 2023-11-28T18:23:45.058 DEBUG:teuthology.orchestra.run.smithi007:> def _find_admin_socket(client_name): 2023-11-28T18:23:45.058 DEBUG:teuthology.orchestra.run.smithi007:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-28T18:23:45.058 DEBUG:teuthology.orchestra.run.smithi007:> files = glob.glob(asok_path) 2023-11-28T18:23:45.058 DEBUG:teuthology.orchestra.run.smithi007:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-28T18:23:45.058 DEBUG:teuthology.orchestra.run.smithi007:> 2023-11-28T18:23:45.058 DEBUG:teuthology.orchestra.run.smithi007:> # Given a non-glob path, it better be there 2023-11-28T18:23:45.058 DEBUG:teuthology.orchestra.run.smithi007:> if "*" not in asok_path: 2023-11-28T18:23:45.059 DEBUG:teuthology.orchestra.run.smithi007:> assert(len(files) == 1) 2023-11-28T18:23:45.059 DEBUG:teuthology.orchestra.run.smithi007:> return files[0] 2023-11-28T18:23:45.059 DEBUG:teuthology.orchestra.run.smithi007:> 2023-11-28T18:23:45.059 DEBUG:teuthology.orchestra.run.smithi007:> for f in files: 2023-11-28T18:23:45.059 DEBUG:teuthology.orchestra.run.smithi007:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T18:23:45.059 DEBUG:teuthology.orchestra.run.smithi007:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T18:23:45.059 DEBUG:teuthology.orchestra.run.smithi007:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T18:23:45.060 DEBUG:teuthology.orchestra.run.smithi007:> contents = proc_f.read() 2023-11-28T18:23:45.060 DEBUG:teuthology.orchestra.run.smithi007:> if mountpoint in contents: 2023-11-28T18:23:45.060 DEBUG:teuthology.orchestra.run.smithi007:> return f 2023-11-28T18:23:45.060 DEBUG:teuthology.orchestra.run.smithi007:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T18:23:45.060 DEBUG:teuthology.orchestra.run.smithi007:> 2023-11-28T18:23:45.060 DEBUG:teuthology.orchestra.run.smithi007:> print(_find_admin_socket("client.0")) 2023-11-28T18:23:45.060 DEBUG:teuthology.orchestra.run.smithi007:> ' 2023-11-28T18:23:45.195 INFO:teuthology.orchestra.run.smithi007.stdout:/var/run/ceph/ceph-client.0.44657.asok 2023-11-28T18:23:45.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T18:23:45.207 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44657.asok 2023-11-28T18:23:45.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:45.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44657.asok status 2023-11-28T18:23:45.387 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "metadata": { 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "entity_id": "0", 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": "smithi007", 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "pid": "44657", 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "root": "/" 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-11-28T18:23:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_count": 0, 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_pinned_count": 0, 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4785, 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "inst": { 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "name": { 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "client", 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "num": 4785 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2809650525 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-11-28T18:23:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2809650525 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "inst_str": "client.4785 192.168.0.1:0/2809650525", 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "addr_str": "192.168.0.1:0/2809650525", 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "inode_count": 1, 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_epoch": 16, 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch": 20, 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch_barrier": 0, 2023-11-28T18:23:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklisted": false, 2023-11-28T18:23:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs" 2023-11-28T18:23:45.391 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-28T18:23:45.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:45.404 DEBUG:teuthology.orchestra.run.smithi007:> stat --file-system '--printf=%T 2023-11-28T18:23:45.404 DEBUG:teuthology.orchestra.run.smithi007:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:45.414 INFO:teuthology.orchestra.run.smithi007.stdout:fuseblk 2023-11-28T18:23:45.414 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:45.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:23:45.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:46.215 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-11-28T18:23:46.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:46.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.375+0000 7f56e1e17640 1 Processor -- start 2023-11-28T18:23:46.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.375+0000 7f56e1e17640 1 -- start start 2023-11-28T18:23:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56e1e17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f56dc15ca30 0x7f56dc159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56e1e17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc15a620 con 0x7f56dc15ce10 2023-11-28T18:23:46.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56e1e17640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc15a900 con 0x7f56dc0cafe0 2023-11-28T18:23:46.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56e1e17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56dc15abe0 con 0x7f56dc15ca30 2023-11-28T18:23:46.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56dbfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f56dc15ce10 0x7f56dc0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57430/0 (socket says 172.21.15.7:57430) 2023-11-28T18:23:46.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56dbfff640 1 -- 172.21.15.7:0/3499758665 learned_addr learned my addr 172.21.15.7:0/3499758665 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56e0e15640 1 --2- 172.21.15.7:0/3499758665 >> v2:172.21.15.193:3300/0 conn(0x7f56dc15ca30 0x7f56dc159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721507367 0 0) 0x7f56dc15a620 con 0x7f56dc15ce10 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc003530 con 0x7f56dc15ce10 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687476713 0 0) 0x7f56dc15a900 con 0x7f56dc0cafe0 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56dc15a620 con 0x7f56dc0cafe0 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 532478627 0 0) 0x7f56cc003530 con 0x7f56dc15ce10 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 >> v2:172.21.15.193:3300/0 conn(0x7f56dc15ca30 msgr2=0x7f56dc159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 --2- 172.21.15.7:0/3499758665 >> v2:172.21.15.193:3300/0 conn(0x7f56dc15ca30 0x7f56dc159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 >> v1:172.21.15.193:6789/0 conn(0x7f56dc0cafe0 legacy=0x7f56dc0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc0cbb10 con 0x7f56dc15ce10 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f56c8001680 con 0x7f56dc15ce10 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 >> v1:172.21.15.7:6789/0 conn(0x7f56dc15ce10 legacy=0x7f56dc0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56cc003b10 0x7f56cc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56cc005190 0x7f56cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc15abe0 con 0x7f56cc0047d0 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc00c010 con 0x7f56cc003b10 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc00c1d0 con 0x7f56cc005190 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56dbfff640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56e0e15640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56cc003b10 0x7f56cc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56e1616640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56cc005190 0x7f56cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56dbfff640 1 -- 172.21.15.7:0/3499758665 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56cc005190 msgr2=0x7f56cc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56dbfff640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56cc005190 0x7f56cc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56dbfff640 1 -- 172.21.15.7:0/3499758665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56cc003b10 msgr2=0x7f56cc003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56dbfff640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56cc003b10 0x7f56cc003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56dbfff640 1 -- 172.21.15.7:0/3499758665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc003530 con 0x7f56cc0047d0 2023-11-28T18:23:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.379+0000 7f56e1616640 1 -- 172.21.15.7:0/3499758665 reap_dead start 2023-11-28T18:23:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56dbfff640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56cc004c50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f56c800c190 tx=0x7f56c801cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982875342fae264c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:46.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c80058e0 con 0x7f56cc0047d0 2023-11-28T18:23:46.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c8005a40 con 0x7f56cc0047d0 2023-11-28T18:23:46.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56db7fe640 1 -- 172.21.15.7:0/3499758665 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c8005e50 con 0x7f56cc0047d0 2023-11-28T18:23:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56e1e17640 1 -- 172.21.15.7:0/3499758665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 msgr2=0x7f56cc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56e1e17640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56cc004c50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f56c800c190 tx=0x7f56c801cba0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56e1e17640 1 -- 172.21.15.7:0/3499758665 shutdown_connections 2023-11-28T18:23:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56e1e17640 1 --2- 172.21.15.7:0/3499758665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56cc004c50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56e1e17640 1 -- 172.21.15.7:0/3499758665 >> 172.21.15.7:0/3499758665 conn(0x7f56dc06c060 msgr2=0x7f56dc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56e1e17640 1 -- 172.21.15.7:0/3499758665 shutdown_connections 2023-11-28T18:23:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.383+0000 7f56e1e17640 1 -- 172.21.15.7:0/3499758665 wait complete. 2023-11-28T18:23:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 Processor -- start 2023-11-28T18:23:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 -- start start 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56dc0c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56dc1efbd0 0x7f56dc1e6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56dc1f0010 0x7f56dc1e6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc1ef740 con 0x7f56cc0047d0 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc1f3710 con 0x7f56dc1f0010 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc151b90 con 0x7f56dc1efbd0 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56dc0c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56dc0c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47354/0 (socket says 172.21.15.7:47354) 2023-11-28T18:23:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 -- 172.21.15.7:0/2969856292 learned_addr learned my addr 172.21.15.7:0/2969856292 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:46.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 -- 172.21.15.7:0/2969856292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56dc1efbd0 msgr2=0x7f56dc1e6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56dc1efbd0 0x7f56dc1e6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 -- 172.21.15.7:0/2969856292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56dc1f0010 msgr2=0x7f56dc1e6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56dc1f0010 0x7f56dc1e6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 -- 172.21.15.7:0/2969856292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc151e50 con 0x7f56cc0047d0 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e0e15640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56dc0c3060 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f56d0011aa0 tx=0x7f56d0011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56d97fa640 1 -- 172.21.15.7:0/2969856292 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d000fb00 con 0x7f56cc0047d0 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56d97fa640 1 -- 172.21.15.7:0/2969856292 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d000fc60 con 0x7f56cc0047d0 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56d97fa640 1 -- 172.21.15.7:0/2969856292 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0014440 con 0x7f56cc0047d0 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc152130 con 0x7f56cc0047d0 2023-11-28T18:23:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.387+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc151920 con 0x7f56cc0047d0 2023-11-28T18:23:46.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.391+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc04e890 con 0x7f56cc0047d0 2023-11-28T18:23:46.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.391+0000 7f56d97fa640 1 -- 172.21.15.7:0/2969856292 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f56d00145a0 con 0x7f56cc0047d0 2023-11-28T18:23:46.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.391+0000 7f56d97fa640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f56c00668d0 0x7f56c0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.391+0000 7f56d97fa640 1 -- 172.21.15.7:0/2969856292 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56d0084c70 con 0x7f56cc0047d0 2023-11-28T18:23:46.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.391+0000 7f56dbfff640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f56c00668d0 0x7f56c0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.391+0000 7f56dbfff640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f56c00668d0 0x7f56c0068d90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f56c80aa000 tx=0x7f56dc0b5060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.395+0000 7f56d97fa640 1 -- 172.21.15.7:0/2969856292 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f56d0051ea0 con 0x7f56cc0047d0 2023-11-28T18:23:46.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.555+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56dc0cad10 con 0x7f56cc0047d0 2023-11-28T18:23:46.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56d97fa640 1 -- 172.21.15.7:0/2969856292 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f56d0022020 con 0x7f56cc0047d0 2023-11-28T18:23:46.564 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:46.564 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f56c00668d0 msgr2=0x7f56c0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f56c00668d0 0x7f56c0068d90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f56c80aa000 tx=0x7f56dc0b5060 comp rx=0 tx=0).stop 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 msgr2=0x7f56dc0c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56dc0c3060 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f56d0011aa0 tx=0x7f56d0011f60 comp rx=0 tx=0).stop 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 shutdown_connections 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f56c00668d0 0x7f56c0068d90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56dc1f0010 0x7f56dc1e6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56dc1efbd0 0x7f56dc1e6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 --2- 172.21.15.7:0/2969856292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56cc0047d0 0x7f56dc0c3060 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 >> 172.21.15.7:0/2969856292 conn(0x7f56dc06c060 msgr2=0x7f56dc0cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:46.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 shutdown_connections 2023-11-28T18:23:46.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.559+0000 7f56e1e17640 1 -- 172.21.15.7:0/2969856292 wait complete. 2023-11-28T18:23:46.585 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T18:23:05.706694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T18:23:07.048910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:23:46.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:23:46.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.723+0000 7fd871dd0640 1 Processor -- start 2023-11-28T18:23:46.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd871dd0640 1 -- start start 2023-11-28T18:23:46.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd871dd0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c151130 0x7fd86c157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd871dd0640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fd86c0c8840 0x7fd86c0c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd871dd0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c15a150 con 0x7fd86c0c6f20 2023-11-28T18:23:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd871dd0640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c15a3a0 con 0x7fd86c0c8840 2023-11-28T18:23:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd871dd0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c15a680 con 0x7fd86c151130 2023-11-28T18:23:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd8715cf640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd86c0c6f20 0x7fd86c0c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57438/0 (socket says 172.21.15.7:57438) 2023-11-28T18:23:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd8715cf640 1 -- 172.21.15.7:0/1293231333 learned_addr learned my addr 172.21.15.7:0/1293231333 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd870dce640 1 --2- 172.21.15.7:0/1293231333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c151130 0x7fd86c157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd86bfff640 1 --2- 172.21.15.7:0/1293231333 >> v2:172.21.15.193:3300/0 conn(0x7fd86c0c8840 0x7fd86c0c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd86b7fe640 1 -- 172.21.15.7:0/1293231333 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457792423 0 0) 0x7fd86c15a150 con 0x7fd86c0c6f20 2023-11-28T18:23:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd86b7fe640 1 -- 172.21.15.7:0/1293231333 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd85c003530 con 0x7fd86c0c6f20 2023-11-28T18:23:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd870dce640 1 -- 172.21.15.7:0/1293231333 >> v2:172.21.15.193:3300/0 conn(0x7fd86c0c8840 msgr2=0x7fd86c0c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd870dce640 1 --2- 172.21.15.7:0/1293231333 >> v2:172.21.15.193:3300/0 conn(0x7fd86c0c8840 0x7fd86c0c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd870dce640 1 -- 172.21.15.7:0/1293231333 >> v1:172.21.15.7:6789/0 conn(0x7fd86c0c6f20 legacy=0x7fd86c0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.727+0000 7fd870dce640 1 -- 172.21.15.7:0/1293231333 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c15a960 con 0x7fd86c151130 2023-11-28T18:23:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd870dce640 1 --2- 172.21.15.7:0/1293231333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c151130 0x7fd86c157660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd8600097f0 tx=0x7fd86001c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed5992957203c6e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd86b7fe640 1 -- 172.21.15.7:0/1293231333 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600ab600 con 0x7fd86c151130 2023-11-28T18:23:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd86b7fe640 1 -- 172.21.15.7:0/1293231333 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8600a7050 con 0x7fd86c151130 2023-11-28T18:23:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd86b7fe640 1 -- 172.21.15.7:0/1293231333 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600ab980 con 0x7fd86c151130 2023-11-28T18:23:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd871dd0640 1 -- 172.21.15.7:0/1293231333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c151130 msgr2=0x7fd86c157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd871dd0640 1 --2- 172.21.15.7:0/1293231333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c151130 0x7fd86c157660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd8600097f0 tx=0x7fd86001c3f0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd871dd0640 1 -- 172.21.15.7:0/1293231333 shutdown_connections 2023-11-28T18:23:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd871dd0640 1 --2- 172.21.15.7:0/1293231333 >> v2:172.21.15.193:3300/0 conn(0x7fd86c0c8840 0x7fd86c0c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd871dd0640 1 --2- 172.21.15.7:0/1293231333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c151130 0x7fd86c157660 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd871dd0640 1 -- 172.21.15.7:0/1293231333 >> 172.21.15.7:0/1293231333 conn(0x7fd86c0662e0 msgr2=0x7fd86c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd871dd0640 1 -- 172.21.15.7:0/1293231333 shutdown_connections 2023-11-28T18:23:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.731+0000 7fd871dd0640 1 -- 172.21.15.7:0/1293231333 wait complete. 2023-11-28T18:23:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 Processor -- start 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 -- start start 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c0c6f20 0x7fd86c0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c0c8840 0x7fd86c1f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd86c151130 0x7fd86c0cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0ca830 con 0x7fd86c0c6f20 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c1f1c70 con 0x7fd86c151130 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0cc230 con 0x7fd86c0c8840 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c0c6f20 0x7fd86c0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c0c6f20 0x7fd86c0ca2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47362/0 (socket says 172.21.15.7:47362) 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 -- 172.21.15.7:0/571554268 learned_addr learned my addr 172.21.15.7:0/571554268 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd86bfff640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c0c8840 0x7fd86c1f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd8715cf640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd86c151130 0x7fd86c0cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 -- 172.21.15.7:0/571554268 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c0c8840 msgr2=0x7fd86c1f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c0c8840 0x7fd86c1f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 -- 172.21.15.7:0/571554268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd86c151130 msgr2=0x7fd86c0cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd86c151130 0x7fd86c0cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 -- 172.21.15.7:0/571554268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860009420 con 0x7fd86c0c6f20 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd870dce640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c0c6f20 0x7fd86c0ca2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd86001a020 tx=0x7fd860019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd8697fa640 1 -- 172.21.15.7:0/571554268 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600abe40 con 0x7fd86c0c6f20 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd8697fa640 1 -- 172.21.15.7:0/571554268 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8600b5410 con 0x7fd86c0c6f20 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd8697fa640 1 -- 172.21.15.7:0/571554268 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600b57a0 con 0x7fd86c0c6f20 2023-11-28T18:23:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c0c94d0 con 0x7fd86c0c6f20 2023-11-28T18:23:46.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c0cb210 con 0x7fd86c0c6f20 2023-11-28T18:23:46.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.735+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c04e890 con 0x7fd86c0c6f20 2023-11-28T18:23:46.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.739+0000 7fd8697fa640 1 -- 172.21.15.7:0/571554268 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd8600b2050 con 0x7fd86c0c6f20 2023-11-28T18:23:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.739+0000 7fd8697fa640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8500668d0 0x7fd850068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.739+0000 7fd8697fa640 1 -- 172.21.15.7:0/571554268 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd86011e760 con 0x7fd86c0c6f20 2023-11-28T18:23:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.739+0000 7fd86bfff640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8500668d0 0x7fd850068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.739+0000 7fd86bfff640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8500668d0 0x7fd850068d90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd86c067120 tx=0x7fd858009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.743+0000 7fd8697fa640 1 -- 172.21.15.7:0/571554268 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd8600eb9b0 con 0x7fd86c0c6f20 2023-11-28T18:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd86c0c4bf0 con 0x7fd86c0c6f20 2023-11-28T18:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd8697fa640 1 -- 172.21.15.7:0/571554268 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8600b0020 con 0x7fd86c0c6f20 2023-11-28T18:23:46.908 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8500668d0 msgr2=0x7fd850068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8500668d0 0x7fd850068d90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd86c067120 tx=0x7fd858009190 comp rx=0 tx=0).stop 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c0c6f20 msgr2=0x7fd86c0ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c0c6f20 0x7fd86c0ca2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd86001a020 tx=0x7fd860019190 comp rx=0 tx=0).stop 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 shutdown_connections 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8500668d0 0x7fd850068d90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd86c151130 0x7fd86c0cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd86c0c8840 0x7fd86c1f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 --2- 172.21.15.7:0/571554268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c0c6f20 0x7fd86c0ca2f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 >> 172.21.15.7:0/571554268 conn(0x7fd86c0662e0 msgr2=0x7fd86c0c6630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:46.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 shutdown_connections 2023-11-28T18:23:46.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:46.903+0000 7fd871dd0640 1 -- 172.21.15.7:0/571554268 wait complete. 2023-11-28T18:23:46.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T18:23:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.067+0000 7ffbb3baa640 1 Processor -- start 2023-11-28T18:23:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb3baa640 1 -- start start 2023-11-28T18:23:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb3baa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbac154130 0x7ffbac159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb3baa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ffbac0c4c80 0x7ffbac15de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb3baa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac0ca530 0x7ffbac0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb3baa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac15a3f0 con 0x7ffbac154130 2023-11-28T18:23:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb3baa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac0caeb0 con 0x7ffbac0c4c80 2023-11-28T18:23:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb3baa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac15d9b0 con 0x7ffbac0ca530 2023-11-28T18:23:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1946640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbac154130 0x7ffbac159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1946640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbac154130 0x7ffbac159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47370/0 (socket says 172.21.15.7:47370) 2023-11-28T18:23:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1946640 1 -- 172.21.15.7:0/3757393149 learned_addr learned my addr 172.21.15.7:0/3757393149 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1145640 1 --2- 172.21.15.7:0/3757393149 >> v2:172.21.15.193:3300/0 conn(0x7ffbac0c4c80 0x7ffbac15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb2147640 1 --2- 172.21.15.7:0/3757393149 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac0ca530 0x7ffbac0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1946640 1 -- 172.21.15.7:0/3757393149 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac0ca530 msgr2=0x7ffbac0cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1946640 1 --2- 172.21.15.7:0/3757393149 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac0ca530 0x7ffbac0cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1946640 1 -- 172.21.15.7:0/3757393149 >> v2:172.21.15.193:3300/0 conn(0x7ffbac0c4c80 msgr2=0x7ffbac15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1946640 1 --2- 172.21.15.7:0/3757393149 >> v2:172.21.15.193:3300/0 conn(0x7ffbac0c4c80 0x7ffbac15de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb1946640 1 -- 172.21.15.7:0/3757393149 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac15e380 con 0x7ffbac154130 2023-11-28T18:23:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.071+0000 7ffbb2147640 1 --2- 172.21.15.7:0/3757393149 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac0ca530 0x7ffbac0cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:47.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb1946640 1 --2- 172.21.15.7:0/3757393149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac154130 0x7ffbac159eb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0c9e60 tx=0x7ffba801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded6a70841891706 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb0944640 1 -- 172.21.15.7:0/3757393149 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba80a89b0 con 0x7ffbac154130 2023-11-28T18:23:47.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb0944640 1 -- 172.21.15.7:0/3757393149 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba80a8b10 con 0x7ffbac154130 2023-11-28T18:23:47.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb0944640 1 -- 172.21.15.7:0/3757393149 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba80a8ea0 con 0x7ffbac154130 2023-11-28T18:23:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3757393149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac154130 msgr2=0x7ffbac159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3757393149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac154130 0x7ffbac159eb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0c9e60 tx=0x7ffba801ecf0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3757393149 shutdown_connections 2023-11-28T18:23:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3757393149 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac0ca530 0x7ffbac0cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3757393149 >> v2:172.21.15.193:3300/0 conn(0x7ffbac0c4c80 0x7ffbac15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3757393149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac154130 0x7ffbac159eb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3757393149 >> 172.21.15.7:0/3757393149 conn(0x7ffbac073ca0 msgr2=0x7ffbac071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3757393149 shutdown_connections 2023-11-28T18:23:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3757393149 wait complete. 2023-11-28T18:23:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 Processor -- start 2023-11-28T18:23:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.075+0000 7ffbb3baa640 1 -- start start 2023-11-28T18:23:47.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbac0c4c80 0x7ffbac1f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac0ca530 0x7ffbac15c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac154130 0x7ffbac154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac1f3080 con 0x7ffbac0ca530 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac1472a0 con 0x7ffbac0c4c80 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac147580 con 0x7ffbac154130 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac0ca530 0x7ffbac15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac0ca530 0x7ffbac15c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47378/0 (socket says 172.21.15.7:47378) 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 -- 172.21.15.7:0/3499272338 learned_addr learned my addr 172.21.15.7:0/3499272338 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1946640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbac0c4c80 0x7ffbac1f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb2147640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac154130 0x7ffbac154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 -- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac154130 msgr2=0x7ffbac154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac154130 0x7ffbac154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 -- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbac0c4c80 msgr2=0x7ffbac1f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbac0c4c80 0x7ffbac1f3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 -- 172.21.15.7:0/3499272338 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba80094d0 con 0x7ffbac0ca530 2023-11-28T18:23:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1946640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbac0c4c80 0x7ffbac1f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb1145640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac0ca530 0x7ffbac15c530 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00d800 tx=0x7ffb9c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffba27fc640 1 -- 172.21.15.7:0/3499272338 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c0096a0 con 0x7ffbac0ca530 2023-11-28T18:23:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffba27fc640 1 -- 172.21.15.7:0/3499272338 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9c009800 con 0x7ffbac0ca530 2023-11-28T18:23:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffba27fc640 1 -- 172.21.15.7:0/3499272338 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c009b70 con 0x7ffbac0ca530 2023-11-28T18:23:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac147860 con 0x7ffbac0ca530 2023-11-28T18:23:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac146e20 con 0x7ffbac0ca530 2023-11-28T18:23:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.079+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac1f3230 con 0x7ffbac0ca530 2023-11-28T18:23:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.083+0000 7ffba27fc640 1 -- 172.21.15.7:0/3499272338 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ffb9c009cd0 con 0x7ffbac0ca530 2023-11-28T18:23:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.083+0000 7ffba27fc640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb900668d0 0x7ffb90068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.083+0000 7ffba27fc640 1 -- 172.21.15.7:0/3499272338 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb9c084b90 con 0x7ffbac0ca530 2023-11-28T18:23:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.083+0000 7ffbb1946640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb900668d0 0x7ffb90068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.083+0000 7ffbb1946640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb900668d0 0x7ffb90068d90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffba80b2000 tx=0x7ffba8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.087+0000 7ffba27fc640 1 -- 172.21.15.7:0/3499272338 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffbac1f3230 con 0x7ffbac0ca530 2023-11-28T18:23:47.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffbac04e890 con 0x7ffbac0ca530 2023-11-28T18:23:47.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffba27fc640 1 -- 172.21.15.7:0/3499272338 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb9c051d50 con 0x7ffbac0ca530 2023-11-28T18:23:47.253 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb900668d0 msgr2=0x7ffb90068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb900668d0 0x7ffb90068d90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffba80b2000 tx=0x7ffba8009040 comp rx=0 tx=0).stop 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac0ca530 msgr2=0x7ffbac15c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac0ca530 0x7ffbac15c530 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00d800 tx=0x7ffb9c00dcc0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 shutdown_connections 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb900668d0 0x7ffb90068d90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbac154130 0x7ffbac154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbac0ca530 0x7ffbac15c530 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 --2- 172.21.15.7:0/3499272338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbac0c4c80 0x7ffbac1f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 >> 172.21.15.7:0/3499272338 conn(0x7ffbac073ca0 msgr2=0x7ffbac0c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 shutdown_connections 2023-11-28T18:23:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.247+0000 7ffbb3baa640 1 -- 172.21.15.7:0/3499272338 wait complete. 2023-11-28T18:23:47.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T18:23:47.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f04179f5640 1 Processor -- start 2023-11-28T18:23:47.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f04179f5640 1 -- start start 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f04179f5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04100c42b0 0x7f041015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f04179f5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04100c8530 con 0x7f041015a5c0 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f04179f5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04100c7680 con 0x7f041015a9a0 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f04179f5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04100c7920 con 0x7f04100c42b0 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0415791640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f041015a5c0 0x7f04100c6890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57442/0 (socket says 172.21.15.7:57442) 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0415791640 1 -- 172.21.15.7:0/1551880620 learned_addr learned my addr 172.21.15.7:0/1551880620 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0415f92640 1 --2- 172.21.15.7:0/1551880620 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04100c42b0 0x7f041015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0407fff640 1 -- 172.21.15.7:0/1551880620 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363509534 0 0) 0x7f04100c8530 con 0x7f041015a5c0 2023-11-28T18:23:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0407fff640 1 -- 172.21.15.7:0/1551880620 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03fc003530 con 0x7f041015a5c0 2023-11-28T18:23:47.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0415f92640 1 -- 172.21.15.7:0/1551880620 >> v1:172.21.15.193:6789/0 conn(0x7f041015a9a0 legacy=0x7f04100c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0407fff640 1 -- 172.21.15.7:0/1551880620 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451300858 0 0) 0x7f04100c7680 con 0x7f041015a9a0 2023-11-28T18:23:47.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0415f92640 1 -- 172.21.15.7:0/1551880620 >> v1:172.21.15.7:6789/0 conn(0x7f041015a5c0 legacy=0x7f04100c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.415+0000 7f0415f92640 1 -- 172.21.15.7:0/1551880620 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04100c7c00 con 0x7f04100c42b0 2023-11-28T18:23:47.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.419+0000 7f0415f92640 1 --2- 172.21.15.7:0/1551880620 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04100c42b0 0x7f041015adc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0408009dc0 tx=0x7f040801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df7ae7eb7e38024c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.419+0000 7f0407fff640 1 -- 172.21.15.7:0/1551880620 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04080af600 con 0x7f04100c42b0 2023-11-28T18:23:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.419+0000 7f0407fff640 1 -- 172.21.15.7:0/1551880620 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04080ab050 con 0x7f04100c42b0 2023-11-28T18:23:47.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.419+0000 7f0407fff640 1 -- 172.21.15.7:0/1551880620 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04080af980 con 0x7f04100c42b0 2023-11-28T18:23:47.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- 172.21.15.7:0/1551880620 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04100c42b0 msgr2=0x7f041015adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 --2- 172.21.15.7:0/1551880620 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04100c42b0 0x7f041015adc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0408009dc0 tx=0x7f040801f150 comp rx=0 tx=0).stop 2023-11-28T18:23:47.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- 172.21.15.7:0/1551880620 shutdown_connections 2023-11-28T18:23:47.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 --2- 172.21.15.7:0/1551880620 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04100c42b0 0x7f041015adc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- 172.21.15.7:0/1551880620 >> 172.21.15.7:0/1551880620 conn(0x7f0410073f00 msgr2=0x7f04100662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:47.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- 172.21.15.7:0/1551880620 shutdown_connections 2023-11-28T18:23:47.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- 172.21.15.7:0/1551880620 wait complete. 2023-11-28T18:23:47.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 Processor -- start 2023-11-28T18:23:47.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- start start 2023-11-28T18:23:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04100c42b0 0x7f0410157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f041015a5c0 0x7f04101e0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f041015a9a0 0x7f0410155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0410147800 con 0x7f04100c42b0 2023-11-28T18:23:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04100caca0 con 0x7f041015a9a0 2023-11-28T18:23:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f04179f5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04100caef0 con 0x7f041015a5c0 2023-11-28T18:23:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04100c42b0 0x7f0410157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04100c42b0 0x7f0410157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47388/0 (socket says 172.21.15.7:47388) 2023-11-28T18:23:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 -- 172.21.15.7:0/2802151385 learned_addr learned my addr 172.21.15.7:0/2802151385 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0414f90640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f041015a5c0 0x7f04101e0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415f92640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f041015a9a0 0x7f0410155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 -- 172.21.15.7:0/2802151385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f041015a5c0 msgr2=0x7f04101e0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f041015a5c0 0x7f04101e0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 -- 172.21.15.7:0/2802151385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f041015a9a0 msgr2=0x7f0410155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f041015a9a0 0x7f0410155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 -- 172.21.15.7:0/2802151385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0408009a70 con 0x7f04100c42b0 2023-11-28T18:23:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.423+0000 7f0415791640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04100c42b0 0x7f0410157660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f040c00c4d0 tx=0x7f040c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f0405ffb640 1 -- 172.21.15.7:0/2802151385 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c005cf0 con 0x7f04100c42b0 2023-11-28T18:23:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f0405ffb640 1 -- 172.21.15.7:0/2802151385 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c005e50 con 0x7f04100c42b0 2023-11-28T18:23:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f0405ffb640 1 -- 172.21.15.7:0/2802151385 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c007270 con 0x7f04100c42b0 2023-11-28T18:23:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04100cbd70 con 0x7f04100c42b0 2023-11-28T18:23:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0410154c60 con 0x7f04100c42b0 2023-11-28T18:23:47.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04100cbbd0 con 0x7f04100c42b0 2023-11-28T18:23:47.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f0405ffb640 1 -- 172.21.15.7:0/2802151385 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f040c0073d0 con 0x7f04100c42b0 2023-11-28T18:23:47.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f0405ffb640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03f40668d0 0x7f03f4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f0405ffb640 1 -- 172.21.15.7:0/2802151385 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f040c083900 con 0x7f04100c42b0 2023-11-28T18:23:47.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.427+0000 7f0414f90640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03f40668d0 0x7f03f4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.431+0000 7f0414f90640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03f40668d0 0x7f03f4068d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f041015a330 tx=0x7f0400009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.431+0000 7f0405ffb640 1 -- 172.21.15.7:0/2802151385 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f04100cbbd0 con 0x7f04100c42b0 2023-11-28T18:23:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.591+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f04100cbbd0 con 0x7f04100c42b0 2023-11-28T18:23:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.591+0000 7f0405ffb640 1 -- 172.21.15.7:0/2802151385 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f04100cbbd0 con 0x7f04100c42b0 2023-11-28T18:23:47.598 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-11-28T18:23:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03f40668d0 msgr2=0x7f03f4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03f40668d0 0x7f03f4068d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f041015a330 tx=0x7f0400009040 comp rx=0 tx=0).stop 2023-11-28T18:23:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04100c42b0 msgr2=0x7f0410157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04100c42b0 0x7f0410157660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f040c00c4d0 tx=0x7f040c00c990 comp rx=0 tx=0).stop 2023-11-28T18:23:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 shutdown_connections 2023-11-28T18:23:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03f40668d0 0x7f03f4068d90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f041015a9a0 0x7f0410155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f041015a5c0 0x7f04101e0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 --2- 172.21.15.7:0/2802151385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04100c42b0 0x7f0410157660 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 >> 172.21.15.7:0/2802151385 conn(0x7f0410073f00 msgr2=0x7f0410071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:47.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 shutdown_connections 2023-11-28T18:23:47.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.595+0000 7f04179f5640 1 -- 172.21.15.7:0/2802151385 wait complete. 2023-11-28T18:23:47.619 INFO:tasks.mds_thrash:mds thrasher using random seed: 1701195827 2023-11-28T18:23:47.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:47.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f749629a640 1 Processor -- start 2023-11-28T18:23:47.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f749629a640 1 -- start start 2023-11-28T18:23:47.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f749629a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f749015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f749629a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7490158000 con 0x7f749006e560 2023-11-28T18:23:47.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f749629a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7490071e60 con 0x7f7490157b90 2023-11-28T18:23:47.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f749629a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7490070930 con 0x7f7490151cd0 2023-11-28T18:23:47.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f748f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f749006e560 0x7f74900c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57456/0 (socket says 172.21.15.7:57456) 2023-11-28T18:23:47.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f748f7fe640 1 -- 172.21.15.7:0/1023977173 learned_addr learned my addr 172.21.15.7:0/1023977173 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:47.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f748ffff640 1 --2- 172.21.15.7:0/1023977173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f749015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f748effd640 1 -- 172.21.15.7:0/1023977173 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757680887 0 0) 0x7f7490158000 con 0x7f749006e560 2023-11-28T18:23:47.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f748effd640 1 -- 172.21.15.7:0/1023977173 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478003530 con 0x7f749006e560 2023-11-28T18:23:47.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f748effd640 1 -- 172.21.15.7:0/1023977173 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371373069 0 0) 0x7f7490071e60 con 0x7f7490157b90 2023-11-28T18:23:47.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.759+0000 7f748effd640 1 -- 172.21.15.7:0/1023977173 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7490158000 con 0x7f7490157b90 2023-11-28T18:23:47.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f748ffff640 1 -- 172.21.15.7:0/1023977173 >> v1:172.21.15.193:6789/0 conn(0x7f7490157b90 legacy=0x7f7490162510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f748effd640 1 -- 172.21.15.7:0/1023977173 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3605028845 0 0) 0x7f7478003530 con 0x7f749006e560 2023-11-28T18:23:47.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f748ffff640 1 -- 172.21.15.7:0/1023977173 >> v1:172.21.15.7:6789/0 conn(0x7f749006e560 legacy=0x7f74900c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f748ffff640 1 -- 172.21.15.7:0/1023977173 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7490070bf0 con 0x7f7490151cd0 2023-11-28T18:23:47.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f748ffff640 1 --2- 172.21.15.7:0/1023977173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f749015bd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74800097f0 tx=0x7f748001c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b226d6195ecbb3cd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f748effd640 1 -- 172.21.15.7:0/1023977173 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484001680 con 0x7f7490151cd0 2023-11-28T18:23:47.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f748effd640 1 -- 172.21.15.7:0/1023977173 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74800a7050 con 0x7f7490151cd0 2023-11-28T18:23:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f748effd640 1 -- 172.21.15.7:0/1023977173 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800ab620 con 0x7f7490151cd0 2023-11-28T18:23:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f749629a640 1 -- 172.21.15.7:0/1023977173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 msgr2=0x7f749015bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f749629a640 1 --2- 172.21.15.7:0/1023977173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f749015bd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74800097f0 tx=0x7f748001c350 comp rx=0 tx=0).stop 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f749629a640 1 -- 172.21.15.7:0/1023977173 shutdown_connections 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f749629a640 1 --2- 172.21.15.7:0/1023977173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f749015bd00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.763+0000 7f749629a640 1 -- 172.21.15.7:0/1023977173 >> 172.21.15.7:0/1023977173 conn(0x7f74900662e0 msgr2=0x7f74900666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 -- 172.21.15.7:0/1023977173 shutdown_connections 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 -- 172.21.15.7:0/1023977173 wait complete. 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 Processor -- start 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 -- start start 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f749006e560 0x7f74901f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f74900c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7490157b90 0x7f74900c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7490163990 con 0x7f7490157b90 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7490147720 con 0x7f749006e560 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f749629a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74900c9030 con 0x7f7490151cd0 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f7494837640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7490157b90 0x7f74900c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f748ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f749006e560 0x7f74901f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f7494837640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7490157b90 0x7f74900c7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47396/0 (socket says 172.21.15.7:47396) 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f7494837640 1 -- 172.21.15.7:0/4229295010 learned_addr learned my addr 172.21.15.7:0/4229295010 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f748f7fe640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f74900c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f7494837640 1 -- 172.21.15.7:0/4229295010 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 msgr2=0x7f74900c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f7494837640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f74900c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f7494837640 1 -- 172.21.15.7:0/4229295010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f749006e560 msgr2=0x7f74901f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f7494837640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f749006e560 0x7f74901f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.767+0000 7f7494837640 1 -- 172.21.15.7:0/4229295010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7480009420 con 0x7f7490157b90 2023-11-28T18:23:47.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f7494837640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7490157b90 0x7f74900c7d00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f748800c730 tx=0x7f748800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f748cff9640 1 -- 172.21.15.7:0/4229295010 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488003b80 con 0x7f7490157b90 2023-11-28T18:23:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f748cff9640 1 -- 172.21.15.7:0/4229295010 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7488003ce0 con 0x7f7490157b90 2023-11-28T18:23:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f748cff9640 1 -- 172.21.15.7:0/4229295010 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748800aa30 con 0x7f7490157b90 2023-11-28T18:23:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74900c8a70 con 0x7f7490157b90 2023-11-28T18:23:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74901f2ac0 con 0x7f7490157b90 2023-11-28T18:23:47.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74901f2920 con 0x7f7490157b90 2023-11-28T18:23:47.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f748cff9640 1 -- 172.21.15.7:0/4229295010 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f748800ab90 con 0x7f7490157b90 2023-11-28T18:23:47.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f748cff9640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f747c0668d0 0x7f747c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:47.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.771+0000 7f748cff9640 1 -- 172.21.15.7:0/4229295010 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7488084060 con 0x7f7490157b90 2023-11-28T18:23:47.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.775+0000 7f748ffff640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f747c0668d0 0x7f747c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:47.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.775+0000 7f748ffff640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f747c0668d0 0x7f747c068d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f74800ad000 tx=0x7f74800a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:47.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.775+0000 7f748cff9640 1 -- 172.21.15.7:0/4229295010 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f74901f2920 con 0x7f7490157b90 2023-11-28T18:23:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74901f2920 con 0x7f7490157b90 2023-11-28T18:23:47.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f748cff9640 1 -- 172.21.15.7:0/4229295010 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f74901f2920 con 0x7f7490157b90 2023-11-28T18:23:47.940 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:47.941 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:09.100591+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:47.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f747c0668d0 msgr2=0x7f747c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f747c0668d0 0x7f747c068d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f74800ad000 tx=0x7f74800a6000 comp rx=0 tx=0).stop 2023-11-28T18:23:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7490157b90 msgr2=0x7f74900c7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7490157b90 0x7f74900c7d00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f748800c730 tx=0x7f748800cbf0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 shutdown_connections 2023-11-28T18:23:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f747c0668d0 0x7f747c068d90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7490157b90 0x7f74900c7d00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7490151cd0 0x7f74900c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 --2- 172.21.15.7:0/4229295010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f749006e560 0x7f74901f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 >> 172.21.15.7:0/4229295010 conn(0x7f74900662e0 msgr2=0x7f74900c62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 shutdown_connections 2023-11-28T18:23:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:47.935+0000 7f749629a640 1 -- 172.21.15.7:0/4229295010 wait complete. 2023-11-28T18:23:47.962 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T18:23:05.706694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T18:23:07.048910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:23:47.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:23:48.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1e73e640 1 Processor -- start 2023-11-28T18:23:48.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1e73e640 1 -- start start 2023-11-28T18:23:48.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1e73e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0b18154130 0x7f0b18159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1e73e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b180c4c80 0x7f0b1815de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1e73e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180ca530 0x7f0b180cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1e73e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b1815a3f0 con 0x7f0b180c4c80 2023-11-28T18:23:48.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1e73e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b180caeb0 con 0x7f0b18154130 2023-11-28T18:23:48.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1e73e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1815d9b0 con 0x7f0b180ca530 2023-11-28T18:23:48.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b177fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b180c4c80 0x7f0b1815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b177fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b180c4c80 0x7f0b1815de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47404/0 (socket says 172.21.15.7:47404) 2023-11-28T18:23:48.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b177fe640 1 -- 172.21.15.7:0/4176311603 learned_addr learned my addr 172.21.15.7:0/4176311603 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:48.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b17fff640 1 --2- 172.21.15.7:0/4176311603 >> v2:172.21.15.193:3300/0 conn(0x7f0b18154130 0x7f0b18159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b1ccdb640 1 --2- 172.21.15.7:0/4176311603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180ca530 0x7f0b180cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b177fe640 1 -- 172.21.15.7:0/4176311603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180ca530 msgr2=0x7f0b180cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:48.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b177fe640 1 --2- 172.21.15.7:0/4176311603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180ca530 0x7f0b180cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b177fe640 1 -- 172.21.15.7:0/4176311603 >> v2:172.21.15.193:3300/0 conn(0x7f0b18154130 msgr2=0x7f0b18159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:48.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b177fe640 1 --2- 172.21.15.7:0/4176311603 >> v2:172.21.15.193:3300/0 conn(0x7f0b18154130 0x7f0b18159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.103+0000 7f0b177fe640 1 -- 172.21.15.7:0/4176311603 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1815e380 con 0x7f0b180c4c80 2023-11-28T18:23:48.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b177fe640 1 --2- 172.21.15.7:0/4176311603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180c4c80 0x7f0b1815de40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0b180cf700 tx=0x7f0b0c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586c5c3c82870049 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b16ffd640 1 -- 172.21.15.7:0/4176311603 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0a89b0 con 0x7f0b180c4c80 2023-11-28T18:23:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b16ffd640 1 -- 172.21.15.7:0/4176311603 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c0a8b10 con 0x7f0b180c4c80 2023-11-28T18:23:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b16ffd640 1 -- 172.21.15.7:0/4176311603 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0a8ea0 con 0x7f0b180c4c80 2023-11-28T18:23:48.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 -- 172.21.15.7:0/4176311603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180c4c80 msgr2=0x7f0b1815de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/4176311603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180c4c80 0x7f0b1815de40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0b180cf700 tx=0x7f0b0c01ecf0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 -- 172.21.15.7:0/4176311603 shutdown_connections 2023-11-28T18:23:48.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/4176311603 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180ca530 0x7f0b180cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/4176311603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180c4c80 0x7f0b1815de40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/4176311603 >> v2:172.21.15.193:3300/0 conn(0x7f0b18154130 0x7f0b18159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 -- 172.21.15.7:0/4176311603 >> 172.21.15.7:0/4176311603 conn(0x7f0b18073ca0 msgr2=0x7f0b18071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:48.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 -- 172.21.15.7:0/4176311603 shutdown_connections 2023-11-28T18:23:48.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.107+0000 7f0b1e73e640 1 -- 172.21.15.7:0/4176311603 wait complete. 2023-11-28T18:23:48.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1e73e640 1 Processor -- start 2023-11-28T18:23:48.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1e73e640 1 -- start start 2023-11-28T18:23:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1e73e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180c4c80 0x7f0b18159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1e73e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180ca530 0x7f0b18158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1e73e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b18154130 0x7f0b18149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1e73e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b181f4720 con 0x7f0b180ca530 2023-11-28T18:23:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1e73e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1815a1b0 con 0x7f0b18154130 2023-11-28T18:23:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1e73e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b181f3b10 con 0x7f0b180c4c80 2023-11-28T18:23:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180ca530 0x7f0b18158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180ca530 0x7f0b18158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47416/0 (socket says 172.21.15.7:47416) 2023-11-28T18:23:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 -- 172.21.15.7:0/750007404 learned_addr learned my addr 172.21.15.7:0/750007404 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b1ccdb640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b18154130 0x7f0b18149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b17fff640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180c4c80 0x7f0b18159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 -- 172.21.15.7:0/750007404 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180c4c80 msgr2=0x7f0b18159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180c4c80 0x7f0b18159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 -- 172.21.15.7:0/750007404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b18154130 msgr2=0x7f0b18149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b18154130 0x7f0b18149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 -- 172.21.15.7:0/750007404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0094d0 con 0x7f0b180ca530 2023-11-28T18:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.111+0000 7f0b177fe640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180ca530 0x7f0b18158df0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c01d3a0 tx=0x7f0b0c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b14ff9640 1 -- 172.21.15.7:0/750007404 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0b0d40 con 0x7f0b180ca530 2023-11-28T18:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b14ff9640 1 -- 172.21.15.7:0/750007404 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c0bb460 con 0x7f0b180ca530 2023-11-28T18:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b181f3df0 con 0x7f0b180ca530 2023-11-28T18:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b14ff9640 1 -- 172.21.15.7:0/750007404 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0b0ea0 con 0x7f0b180ca530 2023-11-28T18:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b181f43b0 con 0x7f0b180ca530 2023-11-28T18:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1804e890 con 0x7f0b180ca530 2023-11-28T18:23:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b14ff9640 1 -- 172.21.15.7:0/750007404 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0b0c0b83f0 con 0x7f0b180ca530 2023-11-28T18:23:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b14ff9640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b040668d0 0x7f0b04068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b14ff9640 1 -- 172.21.15.7:0/750007404 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b0c121de0 con 0x7f0b180ca530 2023-11-28T18:23:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.115+0000 7f0b17fff640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b040668d0 0x7f0b04068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.119+0000 7f0b17fff640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b040668d0 0x7f0b04068d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009ac0 tx=0x7f0b08009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.119+0000 7f0b14ff9640 1 -- 172.21.15.7:0/750007404 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0b0c0ef030 con 0x7f0b180ca530 2023-11-28T18:23:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.279+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b180709f0 con 0x7f0b180ca530 2023-11-28T18:23:48.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b14ff9640 1 -- 172.21.15.7:0/750007404 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b0c0c00a0 con 0x7f0b180ca530 2023-11-28T18:23:48.288 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:23:48.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b040668d0 msgr2=0x7f0b04068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b040668d0 0x7f0b04068d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009ac0 tx=0x7f0b08009190 comp rx=0 tx=0).stop 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180ca530 msgr2=0x7f0b18158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180ca530 0x7f0b18158df0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c01d3a0 tx=0x7f0b0c01ba90 comp rx=0 tx=0).stop 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 shutdown_connections 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b040668d0 0x7f0b04068d90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b18154130 0x7f0b18149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b180ca530 0x7f0b18158df0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 --2- 172.21.15.7:0/750007404 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b180c4c80 0x7f0b18159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 >> 172.21.15.7:0/750007404 conn(0x7f0b18073ca0 msgr2=0x7f0b180cb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 shutdown_connections 2023-11-28T18:23:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.283+0000 7f0b1e73e640 1 -- 172.21.15.7:0/750007404 wait complete. 2023-11-28T18:23:48.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T18:23:48.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.455+0000 7f70ed9d0640 1 Processor -- start 2023-11-28T18:23:48.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.455+0000 7f70ed9d0640 1 -- start start 2023-11-28T18:23:48.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ed9d0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70e80cb3a0 0x7f70e80c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ed9d0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e80cbf50 0x7f70e80cb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ed9d0640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e8158430 con 0x7f70e81588c0 2023-11-28T18:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ed9d0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e8157bf0 con 0x7f70e80cb3a0 2023-11-28T18:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ed9d0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e80cbcc0 con 0x7f70e80cbf50 2023-11-28T18:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ec9ce640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70e80cb3a0 0x7f70e80c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ec9ce640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70e80cb3a0 0x7f70e80c5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47424/0 (socket says 172.21.15.7:47424) 2023-11-28T18:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ec9ce640 1 -- 172.21.15.7:0/1922961730 learned_addr learned my addr 172.21.15.7:0/1922961730 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:48.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ed1cf640 1 --2- 172.21.15.7:0/1922961730 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e80cbf50 0x7f70e80cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ec9ce640 1 -- 172.21.15.7:0/1922961730 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e80cbf50 msgr2=0x7f70e80cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:48.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ec9ce640 1 --2- 172.21.15.7:0/1922961730 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e80cbf50 0x7f70e80cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ec9ce640 1 -- 172.21.15.7:0/1922961730 >> v1:172.21.15.193:6789/0 conn(0x7f70e81588c0 legacy=0x7f70e815b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.459+0000 7f70ec9ce640 1 -- 172.21.15.7:0/1922961730 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e80c2c50 con 0x7f70e80cb3a0 2023-11-28T18:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ec9ce640 1 --2- 172.21.15.7:0/1922961730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cb3a0 0x7f70e80c5d10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009870 tx=0x7f70dc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8210e0dd3bf467d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70e77fe640 1 -- 172.21.15.7:0/1922961730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0aa670 con 0x7f70e80cb3a0 2023-11-28T18:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70e77fe640 1 -- 172.21.15.7:0/1922961730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc0aa7d0 con 0x7f70e80cb3a0 2023-11-28T18:23:48.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70e77fe640 1 -- 172.21.15.7:0/1922961730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0aab60 con 0x7f70e80cb3a0 2023-11-28T18:23:48.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- 172.21.15.7:0/1922961730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cb3a0 msgr2=0x7f70e80c5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/1922961730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cb3a0 0x7f70e80c5d10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009870 tx=0x7f70dc01c510 comp rx=0 tx=0).stop 2023-11-28T18:23:48.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- 172.21.15.7:0/1922961730 shutdown_connections 2023-11-28T18:23:48.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/1922961730 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e80cbf50 0x7f70e80cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/1922961730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cb3a0 0x7f70e80c5d10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- 172.21.15.7:0/1922961730 >> 172.21.15.7:0/1922961730 conn(0x7f70e80662e0 msgr2=0x7f70e80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:48.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- 172.21.15.7:0/1922961730 shutdown_connections 2023-11-28T18:23:48.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- 172.21.15.7:0/1922961730 wait complete. 2023-11-28T18:23:48.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 Processor -- start 2023-11-28T18:23:48.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- start start 2023-11-28T18:23:48.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e80cb3a0 0x7f70e8142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cbf50 0x7f70e8141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e81588c0 0x7f70e81f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8145b30 con 0x7f70e80cbf50 2023-11-28T18:23:48.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8143070 con 0x7f70e80cb3a0 2023-11-28T18:23:48.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.463+0000 7f70ed9d0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e81f3290 con 0x7f70e81588c0 2023-11-28T18:23:48.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cbf50 0x7f70e8141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cbf50 0x7f70e8141990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47430/0 (socket says 172.21.15.7:47430) 2023-11-28T18:23:48.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70ec9ce640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e80cb3a0 0x7f70e8142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 -- 172.21.15.7:0/2740470969 learned_addr learned my addr 172.21.15.7:0/2740470969 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:48.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70ed1cf640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e81588c0 0x7f70e81f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 -- 172.21.15.7:0/2740470969 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e81588c0 msgr2=0x7f70e81f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e81588c0 0x7f70e81f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 -- 172.21.15.7:0/2740470969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e80cb3a0 msgr2=0x7f70e8142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e80cb3a0 0x7f70e8142b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 -- 172.21.15.7:0/2740470969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc0094d0 con 0x7f70e80cbf50 2023-11-28T18:23:48.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e7fff640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cbf50 0x7f70e8141990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f70d400c3c0 tx=0x7f70d400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e57fa640 1 -- 172.21.15.7:0/2740470969 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d40052d0 con 0x7f70e80cbf50 2023-11-28T18:23:48.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e57fa640 1 -- 172.21.15.7:0/2740470969 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d4005430 con 0x7f70e80cbf50 2023-11-28T18:23:48.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e81432b0 con 0x7f70e80cbf50 2023-11-28T18:23:48.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e57fa640 1 -- 172.21.15.7:0/2740470969 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4003840 con 0x7f70e80cbf50 2023-11-28T18:23:48.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e81f24d0 con 0x7f70e80cbf50 2023-11-28T18:23:48.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e804e890 con 0x7f70e80cbf50 2023-11-28T18:23:48.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.467+0000 7f70e57fa640 1 -- 172.21.15.7:0/2740470969 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f70d40039a0 con 0x7f70e80cbf50 2023-11-28T18:23:48.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.471+0000 7f70e57fa640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70cc0668d0 0x7f70cc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.471+0000 7f70e57fa640 1 -- 172.21.15.7:0/2740470969 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f70d4083a80 con 0x7f70e80cbf50 2023-11-28T18:23:48.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.471+0000 7f70ec9ce640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70cc0668d0 0x7f70cc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.471+0000 7f70ec9ce640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70cc0668d0 0x7f70cc068d90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0ae000 tx=0x7f70dc0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.475+0000 7f70e57fa640 1 -- 172.21.15.7:0/2740470969 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f70d4050bc0 con 0x7f70e80cbf50 2023-11-28T18:23:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f70e8072630 con 0x7f70e80cbf50 2023-11-28T18:23:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70e57fa640 1 -- 172.21.15.7:0/2740470969 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f70d40170a0 con 0x7f70e80cbf50 2023-11-28T18:23:48.636 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-11-28T18:23:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70cc0668d0 msgr2=0x7f70cc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70cc0668d0 0x7f70cc068d90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0ae000 tx=0x7f70dc0b5040 comp rx=0 tx=0).stop 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cbf50 msgr2=0x7f70e8141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cbf50 0x7f70e8141990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f70d400c3c0 tx=0x7f70d400c880 comp rx=0 tx=0).stop 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 shutdown_connections 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70cc0668d0 0x7f70cc068d90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f70e81588c0 0x7f70e81f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e80cbf50 0x7f70e8141990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 --2- 172.21.15.7:0/2740470969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e80cb3a0 0x7f70e8142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 >> 172.21.15.7:0/2740470969 conn(0x7f70e80662e0 msgr2=0x7f70e8151410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 shutdown_connections 2023-11-28T18:23:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.631+0000 7f70ed9d0640 1 -- 172.21.15.7:0/2740470969 wait complete. 2023-11-28T18:23:48.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T18:23:48.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef1e780640 1 Processor -- start 2023-11-28T18:23:48.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef1e780640 1 -- start start 2023-11-28T18:23:48.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef1e780640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fef1815a9a0 0x7fef1815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef1e780640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef180c8530 con 0x7fef1815a5c0 2023-11-28T18:23:48.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef1e780640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef180c7680 con 0x7fef180c3ba0 2023-11-28T18:23:48.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef1e780640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef180c7920 con 0x7fef1815a9a0 2023-11-28T18:23:48.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef17fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fef1815a5c0 0x7fef180c6890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57472/0 (socket says 172.21.15.7:57472) 2023-11-28T18:23:48.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef17fff640 1 -- 172.21.15.7:0/446515064 learned_addr learned my addr 172.21.15.7:0/446515064 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:48.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef177fe640 1 --2- 172.21.15.7:0/446515064 >> v2:172.21.15.193:3300/0 conn(0x7fef1815a9a0 0x7fef1815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef16ffd640 1 -- 172.21.15.7:0/446515064 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491210577 0 0) 0x7fef180c8530 con 0x7fef1815a5c0 2023-11-28T18:23:48.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef16ffd640 1 -- 172.21.15.7:0/446515064 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef00003530 con 0x7fef1815a5c0 2023-11-28T18:23:48.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef16ffd640 1 -- 172.21.15.7:0/446515064 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746257248 0 0) 0x7fef180c7680 con 0x7fef180c3ba0 2023-11-28T18:23:48.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef16ffd640 1 -- 172.21.15.7:0/446515064 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef180c8530 con 0x7fef180c3ba0 2023-11-28T18:23:48.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef16ffd640 1 -- 172.21.15.7:0/446515064 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2070494385 0 0) 0x7fef00003530 con 0x7fef1815a5c0 2023-11-28T18:23:48.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef177fe640 1 -- 172.21.15.7:0/446515064 >> v1:172.21.15.193:6789/0 conn(0x7fef180c3ba0 legacy=0x7fef180c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef177fe640 1 -- 172.21.15.7:0/446515064 >> v1:172.21.15.7:6789/0 conn(0x7fef1815a5c0 legacy=0x7fef180c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.799+0000 7fef177fe640 1 -- 172.21.15.7:0/446515064 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef180c7c00 con 0x7fef1815a9a0 2023-11-28T18:23:48.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef177fe640 1 --2- 172.21.15.7:0/446515064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef1815a9a0 0x7fef1815adc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0097c0 tx=0x7fef0c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d21542439b1c20e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef16ffd640 1 -- 172.21.15.7:0/446515064 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef08001680 con 0x7fef1815a9a0 2023-11-28T18:23:48.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef16ffd640 1 -- 172.21.15.7:0/446515064 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c0a8610 con 0x7fef1815a9a0 2023-11-28T18:23:48.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef16ffd640 1 -- 172.21.15.7:0/446515064 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c0a89a0 con 0x7fef1815a9a0 2023-11-28T18:23:48.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef1e780640 1 -- 172.21.15.7:0/446515064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef1815a9a0 msgr2=0x7fef1815adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef1e780640 1 --2- 172.21.15.7:0/446515064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef1815a9a0 0x7fef1815adc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0097c0 tx=0x7fef0c01eb90 comp rx=0 tx=0).stop 2023-11-28T18:23:48.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef1e780640 1 -- 172.21.15.7:0/446515064 shutdown_connections 2023-11-28T18:23:48.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef1e780640 1 --2- 172.21.15.7:0/446515064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef1815a9a0 0x7fef1815adc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef1e780640 1 -- 172.21.15.7:0/446515064 >> 172.21.15.7:0/446515064 conn(0x7fef18073f00 msgr2=0x7fef180662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:48.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef1e780640 1 -- 172.21.15.7:0/446515064 shutdown_connections 2023-11-28T18:23:48.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.803+0000 7fef1e780640 1 -- 172.21.15.7:0/446515064 wait complete. 2023-11-28T18:23:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 Processor -- start 2023-11-28T18:23:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 -- start start 2023-11-28T18:23:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef180c3ba0 0x7fef181f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1815a5c0 0x7fef181f0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef1815a9a0 0x7fef180c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef181f3d40 con 0x7fef1815a5c0 2023-11-28T18:23:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef181f0860 con 0x7fef180c3ba0 2023-11-28T18:23:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef180c96b0 con 0x7fef1815a9a0 2023-11-28T18:23:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1815a5c0 0x7fef181f0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1815a5c0 0x7fef181f0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47440/0 (socket says 172.21.15.7:47440) 2023-11-28T18:23:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 -- 172.21.15.7:0/1093968183 learned_addr learned my addr 172.21.15.7:0/1093968183 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef17fff640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef180c3ba0 0x7fef181f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1cd1d640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef1815a9a0 0x7fef180c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 -- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef1815a9a0 msgr2=0x7fef180c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef1815a9a0 0x7fef180c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 -- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef180c3ba0 msgr2=0x7fef181f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef180c3ba0 0x7fef181f3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 -- 172.21.15.7:0/1093968183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c009420 con 0x7fef1815a5c0 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef17fff640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef180c3ba0 0x7fef181f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1cd1d640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef1815a9a0 0x7fef180c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef177fe640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1815a5c0 0x7fef181f0320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0b3040 tx=0x7fef0c01b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef14ff9640 1 -- 172.21.15.7:0/1093968183 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c004230 con 0x7fef1815a5c0 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef14ff9640 1 -- 172.21.15.7:0/1093968183 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c004390 con 0x7fef1815a5c0 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef18159f20 con 0x7fef1815a5c0 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef14ff9640 1 -- 172.21.15.7:0/1093968183 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c0b0890 con 0x7fef1815a5c0 2023-11-28T18:23:48.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.807+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef180cbfd0 con 0x7fef1815a5c0 2023-11-28T18:23:48.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.811+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1804e890 con 0x7fef1815a5c0 2023-11-28T18:23:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.811+0000 7fef14ff9640 1 -- 172.21.15.7:0/1093968183 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fef0c0b7040 con 0x7fef1815a5c0 2023-11-28T18:23:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.811+0000 7fef14ff9640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef040668d0 0x7fef04068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.811+0000 7fef14ff9640 1 -- 172.21.15.7:0/1093968183 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fef0c120e50 con 0x7fef1815a5c0 2023-11-28T18:23:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.811+0000 7fef17fff640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef040668d0 0x7fef04068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.811+0000 7fef17fff640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef040668d0 0x7fef04068d90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fef08000fc0 tx=0x7fef08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:48.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.815+0000 7fef14ff9640 1 -- 172.21.15.7:0/1093968183 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fef0c0ee000 con 0x7fef1815a5c0 2023-11-28T18:23:48.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.971+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef180c6c10 con 0x7fef1815a5c0 2023-11-28T18:23:48.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.971+0000 7fef14ff9640 1 -- 172.21.15.7:0/1093968183 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef0c006e20 con 0x7fef1815a5c0 2023-11-28T18:23:48.979 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-11-28T18:23:48.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef040668d0 msgr2=0x7fef04068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef040668d0 0x7fef04068d90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fef08000fc0 tx=0x7fef08009040 comp rx=0 tx=0).stop 2023-11-28T18:23:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1815a5c0 msgr2=0x7fef181f0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1815a5c0 0x7fef181f0320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0b3040 tx=0x7fef0c01b930 comp rx=0 tx=0).stop 2023-11-28T18:23:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 shutdown_connections 2023-11-28T18:23:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef040668d0 0x7fef04068d90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef1815a9a0 0x7fef180c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1815a5c0 0x7fef181f0320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 --2- 172.21.15.7:0/1093968183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef180c3ba0 0x7fef181f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 >> 172.21.15.7:0/1093968183 conn(0x7fef18073f00 msgr2=0x7fef1814e720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 shutdown_connections 2023-11-28T18:23:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:48.975+0000 7fef1e780640 1 -- 172.21.15.7:0/1093968183 wait complete. 2023-11-28T18:23:49.000 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-11-28T18:23:49.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.135+0000 7f5fe3b8d640 1 Processor -- start 2023-11-28T18:23:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.135+0000 7f5fe3b8d640 1 -- start start 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe3b8d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5fdc1542e0 0x7f5fdc15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe3b8d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 0x7f5fdc0728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe3b8d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc0ca920 con 0x7f5fdc0cbb70 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe3b8d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc071d50 con 0x7f5fdc1542e0 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe3b8d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc072030 con 0x7f5fdc072450 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe1128640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5fdc0cbb70 0x7f5fdc0cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57476/0 (socket says 172.21.15.7:57476) 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe1128640 1 -- 172.21.15.7:0/875120570 learned_addr learned my addr 172.21.15.7:0/875120570 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe1929640 1 --2- 172.21.15.7:0/875120570 >> v2:172.21.15.7:3300/0 conn(0x7f5fdc1542e0 0x7f5fdc15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe212a640 1 --2- 172.21.15.7:0/875120570 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 0x7f5fdc0728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe0927640 1 -- 172.21.15.7:0/875120570 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378441264 0 0) 0x7f5fdc0ca920 con 0x7f5fdc0cbb70 2023-11-28T18:23:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe0927640 1 -- 172.21.15.7:0/875120570 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc8003530 con 0x7f5fdc0cbb70 2023-11-28T18:23:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe1929640 1 -- 172.21.15.7:0/875120570 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 msgr2=0x7f5fdc0728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe1929640 1 --2- 172.21.15.7:0/875120570 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 0x7f5fdc0728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe1929640 1 -- 172.21.15.7:0/875120570 >> v1:172.21.15.7:6789/0 conn(0x7f5fdc0cbb70 legacy=0x7f5fdc0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.139+0000 7f5fe1929640 1 -- 172.21.15.7:0/875120570 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc15e550 con 0x7f5fdc1542e0 2023-11-28T18:23:49.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe1929640 1 --2- 172.21.15.7:0/875120570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc15e010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80097f0 tx=0x7f5fd801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5b7b308c7c7167 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe0927640 1 -- 172.21.15.7:0/875120570 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80ad640 con 0x7f5fdc1542e0 2023-11-28T18:23:49.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe0927640 1 -- 172.21.15.7:0/875120570 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd80ad7a0 con 0x7f5fdc1542e0 2023-11-28T18:23:49.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe0927640 1 -- 172.21.15.7:0/875120570 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80adb30 con 0x7f5fdc1542e0 2023-11-28T18:23:49.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/875120570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 msgr2=0x7f5fdc15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/875120570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc15e010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80097f0 tx=0x7f5fd801c480 comp rx=0 tx=0).stop 2023-11-28T18:23:49.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/875120570 shutdown_connections 2023-11-28T18:23:49.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/875120570 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 0x7f5fdc0728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/875120570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc15e010 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/875120570 >> 172.21.15.7:0/875120570 conn(0x7f5fdc0717e0 msgr2=0x7f5fdc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:49.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/875120570 shutdown_connections 2023-11-28T18:23:49.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/875120570 wait complete. 2023-11-28T18:23:49.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 Processor -- start 2023-11-28T18:23:49.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.143+0000 7f5fe3b8d640 1 -- start start 2023-11-28T18:23:49.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 0x7f5fdc15e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc0cbb70 0x7f5fdc15f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc1f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc151580 con 0x7f5fdc1542e0 2023-11-28T18:23:49.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc1f3630 con 0x7f5fdc0cbb70 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc160710 con 0x7f5fdc072450 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc1f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc1f30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47462/0 (socket says 172.21.15.7:47462) 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 -- 172.21.15.7:0/330262541 learned_addr learned my addr 172.21.15.7:0/330262541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe1929640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 0x7f5fdc15e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe1128640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc0cbb70 0x7f5fdc15f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 -- 172.21.15.7:0/330262541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 msgr2=0x7f5fdc15e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 0x7f5fdc15e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 -- 172.21.15.7:0/330262541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc0cbb70 msgr2=0x7f5fdc15f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc0cbb70 0x7f5fdc15f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 -- 172.21.15.7:0/330262541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd8009420 con 0x7f5fdc1542e0 2023-11-28T18:23:49.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe212a640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc1f30f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009dd0 tx=0x7f5fd400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fd27fc640 1 -- 172.21.15.7:0/330262541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400ced0 con 0x7f5fdc1542e0 2023-11-28T18:23:49.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fd27fc640 1 -- 172.21.15.7:0/330262541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd400f040 con 0x7f5fdc1542e0 2023-11-28T18:23:49.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fd27fc640 1 -- 172.21.15.7:0/330262541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd40146a0 con 0x7f5fdc1542e0 2023-11-28T18:23:49.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc1f6360 con 0x7f5fdc1542e0 2023-11-28T18:23:49.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc15f330 con 0x7f5fdc1542e0 2023-11-28T18:23:49.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.147+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc06c670 con 0x7f5fdc1542e0 2023-11-28T18:23:49.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.151+0000 7f5fd27fc640 1 -- 172.21.15.7:0/330262541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5fd4014800 con 0x7f5fdc1542e0 2023-11-28T18:23:49.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.151+0000 7f5fd27fc640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5fc00668d0 0x7f5fc0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.151+0000 7f5fd27fc640 1 -- 172.21.15.7:0/330262541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5fd4085360 con 0x7f5fdc1542e0 2023-11-28T18:23:49.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.151+0000 7f5fe1929640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5fc00668d0 0x7f5fc0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.151+0000 7f5fe1929640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5fc00668d0 0x7f5fc0068d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80af000 tx=0x7f5fd80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.155+0000 7f5fd27fc640 1 -- 172.21.15.7:0/330262541 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5fd4052520 con 0x7f5fdc1542e0 2023-11-28T18:23:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.351+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fdc1f64f0 con 0x7f5fdc1542e0 2023-11-28T18:23:49.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.351+0000 7f5fd27fc640 1 -- 172.21.15.7:0/330262541 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f5fd4055b40 con 0x7f5fdc1542e0 2023-11-28T18:23:49.357 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:49.357 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:23:35.257171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:23:49.357 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2023-11-28T18:23:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.351+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5fc00668d0 msgr2=0x7f5fc0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.351+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5fc00668d0 0x7f5fc0068d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80af000 tx=0x7f5fd80a7070 comp rx=0 tx=0).stop 2023-11-28T18:23:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.351+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 msgr2=0x7f5fdc1f30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.351+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc1f30f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009dd0 tx=0x7f5fd400c490 comp rx=0 tx=0).stop 2023-11-28T18:23:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.355+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 shutdown_connections 2023-11-28T18:23:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.355+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5fc00668d0 0x7f5fc0068d90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.355+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fdc1542e0 0x7f5fdc1f30f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.355+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc0cbb70 0x7f5fdc15f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.355+0000 7f5fe3b8d640 1 --2- 172.21.15.7:0/330262541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5fdc072450 0x7f5fdc15e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.355+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 >> 172.21.15.7:0/330262541 conn(0x7f5fdc0717e0 msgr2=0x7f5fdc152c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.355+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 shutdown_connections 2023-11-28T18:23:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.355+0000 7f5fe3b8d640 1 -- 172.21.15.7:0/330262541 wait complete. 2023-11-28T18:23:49.378 INFO:tasks.mds_thrash:Ready to start thrashing 2023-11-28T18:23:49.378 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-11-28T18:23:49.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T18:23:49.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.515+0000 7f9c4e99d640 1 Processor -- start 2023-11-28T18:23:49.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c4e99d640 1 -- start start 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c4e99d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c481540a0 0x7f9c48072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c4e99d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c480c8ec0 0x7f9c480c9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c4e99d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480c8750 con 0x7f9c4815d350 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c4e99d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480c9890 con 0x7f9c481540a0 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c4e99d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480c8a30 con 0x7f9c480c8ec0 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c47fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c481540a0 0x7f9c48072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c47fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c481540a0 0x7f9c48072ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47478/0 (socket says 172.21.15.7:47478) 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c47fff640 1 -- 172.21.15.7:0/139766905 learned_addr learned my addr 172.21.15.7:0/139766905 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c4cf3a640 1 --2- 172.21.15.7:0/139766905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c480c8ec0 0x7f9c480c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c47fff640 1 -- 172.21.15.7:0/139766905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c480c8ec0 msgr2=0x7f9c480c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:49.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c47fff640 1 --2- 172.21.15.7:0/139766905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c480c8ec0 0x7f9c480c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c47fff640 1 -- 172.21.15.7:0/139766905 >> v1:172.21.15.193:6789/0 conn(0x7f9c4815d350 legacy=0x7f9c4806a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.519+0000 7f9c47fff640 1 -- 172.21.15.7:0/139766905 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4815d840 con 0x7f9c481540a0 2023-11-28T18:23:49.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c47fff640 1 --2- 172.21.15.7:0/139766905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c48072ab0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9c38009870 tx=0x7f9c3801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee4b981b90df694 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c46ffd640 1 -- 172.21.15.7:0/139766905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c380a7070 con 0x7f9c481540a0 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c46ffd640 1 -- 172.21.15.7:0/139766905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c380ab430 con 0x7f9c481540a0 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c46ffd640 1 -- 172.21.15.7:0/139766905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c380ab7c0 con 0x7f9c481540a0 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 -- 172.21.15.7:0/139766905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 msgr2=0x7f9c48072ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/139766905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c48072ab0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9c38009870 tx=0x7f9c3801c540 comp rx=0 tx=0).stop 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 -- 172.21.15.7:0/139766905 shutdown_connections 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/139766905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c480c8ec0 0x7f9c480c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/139766905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c48072ab0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 -- 172.21.15.7:0/139766905 >> 172.21.15.7:0/139766905 conn(0x7f9c48073cd0 msgr2=0x7f9c480662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 -- 172.21.15.7:0/139766905 shutdown_connections 2023-11-28T18:23:49.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 -- 172.21.15.7:0/139766905 wait complete. 2023-11-28T18:23:49.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 Processor -- start 2023-11-28T18:23:49.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.523+0000 7f9c4e99d640 1 -- start start 2023-11-28T18:23:49.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c480c8ec0 0x7f9c481488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c4814f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c4815d350 0x7f9c48144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480cbbb0 con 0x7f9c481540a0 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480cb670 con 0x7f9c480c8ec0 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480cb910 con 0x7f9c4815d350 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c4814f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c4814f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47490/0 (socket says 172.21.15.7:47490) 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 -- 172.21.15.7:0/1136281450 learned_addr learned my addr 172.21.15.7:0/1136281450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4cf3a640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c4815d350 0x7f9c48144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c47fff640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c480c8ec0 0x7f9c481488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 -- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c4815d350 msgr2=0x7f9c48144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c4815d350 0x7f9c48144130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 -- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c480c8ec0 msgr2=0x7f9c481488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c480c8ec0 0x7f9c481488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 -- 172.21.15.7:0/1136281450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c380094d0 con 0x7f9c481540a0 2023-11-28T18:23:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4cf3a640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c4815d350 0x7f9c48144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c47fff640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c480c8ec0 0x7f9c481488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c477fe640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c4814f500 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00c3c0 tx=0x7f9c3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c44ff9640 1 -- 172.21.15.7:0/1136281450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c0052d0 con 0x7f9c481540a0 2023-11-28T18:23:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c44ff9640 1 -- 172.21.15.7:0/1136281450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3c005430 con 0x7f9c481540a0 2023-11-28T18:23:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c44ff9640 1 -- 172.21.15.7:0/1136281450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c003840 con 0x7f9c481540a0 2023-11-28T18:23:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c480cc9c0 con 0x7f9c481540a0 2023-11-28T18:23:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c480ccf80 con 0x7f9c481540a0 2023-11-28T18:23:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4804e890 con 0x7f9c481540a0 2023-11-28T18:23:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.527+0000 7f9c44ff9640 1 -- 172.21.15.7:0/1136281450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9c3c0039a0 con 0x7f9c481540a0 2023-11-28T18:23:49.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.531+0000 7f9c44ff9640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9c340668d0 0x7f9c34068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.531+0000 7f9c44ff9640 1 -- 172.21.15.7:0/1136281450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9c3c083aa0 con 0x7f9c481540a0 2023-11-28T18:23:49.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.531+0000 7f9c47fff640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9c340668d0 0x7f9c34068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.531+0000 7f9c47fff640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9c340668d0 0x7f9c34068d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9c380af000 tx=0x7f9c38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.535+0000 7f9c44ff9640 1 -- 172.21.15.7:0/1136281450 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9c3c050bc0 con 0x7f9c481540a0 2023-11-28T18:23:49.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.691+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 --> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c480ce0c0 con 0x7f9c340668d0 2023-11-28T18:23:49.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.699+0000 7f9c44ff9640 1 -- 172.21.15.7:0/1136281450 <== mgr.4102 v2:172.21.15.7:6832/264069339 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264664 (secure 0 0 0) 0x7f9c380bb090 con 0x7f9c340668d0 2023-11-28T18:23:49.722 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-28T18:23:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9c340668d0 msgr2=0x7f9c34068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9c340668d0 0x7f9c34068d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9c380af000 tx=0x7f9c38008040 comp rx=0 tx=0).stop 2023-11-28T18:23:49.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 msgr2=0x7f9c4814f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c4814f500 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00c3c0 tx=0x7f9c3c00c880 comp rx=0 tx=0).stop 2023-11-28T18:23:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 shutdown_connections 2023-11-28T18:23:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9c340668d0 0x7f9c34068d90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c4815d350 0x7f9c48144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c481540a0 0x7f9c4814f500 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 --2- 172.21.15.7:0/1136281450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c480c8ec0 0x7f9c481488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 >> 172.21.15.7:0/1136281450 conn(0x7f9c48073cd0 msgr2=0x7f9c4815ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:49.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 shutdown_connections 2023-11-28T18:23:49.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.703+0000 7f9c4e99d640 1 -- 172.21.15.7:0/1136281450 wait complete. 2023-11-28T18:23:49.738 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-11-28T18:23:49.470780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600162,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":371,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235784,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738936,"statfs":{"total":773094113280,"available":772852670464,"internally_reserved":0,"allocated":3088384,"data_stored":1765844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1760,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001731"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001273+0000","last_change":"2023-11-28T18:23:08.134633+0000","last_active":"2023-11-28T18:23:11.001273+0000","last_peered":"2023-11-28T18:23:11.001273+0000","last_clean":"2023-11-28T18:23:11.001273+0000","last_became_active":"2023-11-28T18:23:08.134320+0000","last_became_peered":"2023-11-28T18:23:08.134320+0000","last_unstale":"2023-11-28T18:23:11.001273+0000","last_undegraded":"2023-11-28T18:23:11.001273+0000","last_fullsized":"2023-11-28T18:23:11.001273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:54.995718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105857+0000","last_change":"2023-11-28T18:23:08.122460+0000","last_active":"2023-11-28T18:23:09.105857+0000","last_peered":"2023-11-28T18:23:09.105857+0000","last_clean":"2023-11-28T18:23:09.105857+0000","last_became_active":"2023-11-28T18:23:08.122153+0000","last_became_peered":"2023-11-28T18:23:08.122153+0000","last_unstale":"2023-11-28T18:23:09.105857+0000","last_undegraded":"2023-11-28T18:23:09.105857+0000","last_fullsized":"2023-11-28T18:23:09.105857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:17:31.074978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914207+0000","last_change":"2023-11-28T18:23:08.110187+0000","last_active":"2023-11-28T18:23:10.914207+0000","last_peered":"2023-11-28T18:23:10.914207+0000","last_clean":"2023-11-28T18:23:10.914207+0000","last_became_active":"2023-11-28T18:23:08.109915+0000","last_became_peered":"2023-11-28T18:23:08.109915+0000","last_unstale":"2023-11-28T18:23:10.914207+0000","last_undegraded":"2023-11-28T18:23:10.914207+0000","last_fullsized":"2023-11-28T18:23:10.914207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:32.098408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113335+0000","last_change":"2023-11-28T18:23:08.122000+0000","last_active":"2023-11-28T18:23:09.113335+0000","last_peered":"2023-11-28T18:23:09.113335+0000","last_clean":"2023-11-28T18:23:09.113335+0000","last_became_active":"2023-11-28T18:23:08.121652+0000","last_became_peered":"2023-11-28T18:23:08.121652+0000","last_unstale":"2023-11-28T18:23:09.113335+0000","last_undegraded":"2023-11-28T18:23:09.113335+0000","last_fullsized":"2023-11-28T18:23:09.113335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:15.540606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000810+0000","last_change":"2023-11-28T18:23:08.126587+0000","last_active":"2023-11-28T18:23:11.000810+0000","last_peered":"2023-11-28T18:23:11.000810+0000","last_clean":"2023-11-28T18:23:11.000810+0000","last_became_active":"2023-11-28T18:23:08.126313+0000","last_became_peered":"2023-11-28T18:23:08.126313+0000","last_unstale":"2023-11-28T18:23:11.000810+0000","last_undegraded":"2023-11-28T18:23:11.000810+0000","last_fullsized":"2023-11-28T18:23:11.000810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.956197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113040+0000","last_change":"2023-11-28T18:23:08.122878+0000","last_active":"2023-11-28T18:23:09.113040+0000","last_peered":"2023-11-28T18:23:09.113040+0000","last_clean":"2023-11-28T18:23:09.113040+0000","last_became_active":"2023-11-28T18:23:08.122609+0000","last_became_peered":"2023-11-28T18:23:08.122609+0000","last_unstale":"2023-11-28T18:23:09.113040+0000","last_undegraded":"2023-11-28T18:23:09.113040+0000","last_fullsized":"2023-11-28T18:23:09.113040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:00.610074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000604+0000","last_change":"2023-11-28T18:23:08.133485+0000","last_active":"2023-11-28T18:23:11.000604+0000","last_peered":"2023-11-28T18:23:11.000604+0000","last_clean":"2023-11-28T18:23:11.000604+0000","last_became_active":"2023-11-28T18:23:08.133211+0000","last_became_peered":"2023-11-28T18:23:08.133211+0000","last_unstale":"2023-11-28T18:23:11.000604+0000","last_undegraded":"2023-11-28T18:23:11.000604+0000","last_fullsized":"2023-11-28T18:23:11.000604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:04:33.507685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565355+0000","last_change":"2023-11-28T18:23:08.117903+0000","last_active":"2023-11-28T18:23:09.565355+0000","last_peered":"2023-11-28T18:23:09.565355+0000","last_clean":"2023-11-28T18:23:09.565355+0000","last_became_active":"2023-11-28T18:23:08.117482+0000","last_became_peered":"2023-11-28T18:23:08.117482+0000","last_unstale":"2023-11-28T18:23:09.565355+0000","last_undegraded":"2023-11-28T18:23:09.565355+0000","last_fullsized":"2023-11-28T18:23:09.565355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:45.293402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001351+0000","last_change":"2023-11-28T18:23:08.135606+0000","last_active":"2023-11-28T18:23:11.001351+0000","last_peered":"2023-11-28T18:23:11.001351+0000","last_clean":"2023-11-28T18:23:11.001351+0000","last_became_active":"2023-11-28T18:23:08.135328+0000","last_became_peered":"2023-11-28T18:23:08.135328+0000","last_unstale":"2023-11-28T18:23:11.001351+0000","last_undegraded":"2023-11-28T18:23:11.001351+0000","last_fullsized":"2023-11-28T18:23:11.001351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:25.011869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110046+0000","last_change":"2023-11-28T18:23:08.123993+0000","last_active":"2023-11-28T18:23:09.110046+0000","last_peered":"2023-11-28T18:23:09.110046+0000","last_clean":"2023-11-28T18:23:09.110046+0000","last_became_active":"2023-11-28T18:23:08.123583+0000","last_became_peered":"2023-11-28T18:23:08.123583+0000","last_unstale":"2023-11-28T18:23:09.110046+0000","last_undegraded":"2023-11-28T18:23:09.110046+0000","last_fullsized":"2023-11-28T18:23:09.110046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:16:33.089307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:08.120983+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:08.120486+0000","last_became_peered":"2023-11-28T18:23:08.120486+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:46:12.881629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637634+0000","last_change":"2023-11-28T18:23:08.130169+0000","last_active":"2023-11-28T18:23:09.637634+0000","last_peered":"2023-11-28T18:23:09.637634+0000","last_clean":"2023-11-28T18:23:09.637634+0000","last_became_active":"2023-11-28T18:23:08.129771+0000","last_became_peered":"2023-11-28T18:23:08.129771+0000","last_unstale":"2023-11-28T18:23:09.637634+0000","last_undegraded":"2023-11-28T18:23:09.637634+0000","last_fullsized":"2023-11-28T18:23:09.637634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:45.753322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565505+0000","last_change":"2023-11-28T18:23:08.123268+0000","last_active":"2023-11-28T18:23:09.565505+0000","last_peered":"2023-11-28T18:23:09.565505+0000","last_clean":"2023-11-28T18:23:09.565505+0000","last_became_active":"2023-11-28T18:23:08.122978+0000","last_became_peered":"2023-11-28T18:23:08.122978+0000","last_unstale":"2023-11-28T18:23:09.565505+0000","last_undegraded":"2023-11-28T18:23:09.565505+0000","last_fullsized":"2023-11-28T18:23:09.565505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:15:01.320918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913943+0000","last_change":"2023-11-28T18:23:08.118361+0000","last_active":"2023-11-28T18:23:10.913943+0000","last_peered":"2023-11-28T18:23:10.913943+0000","last_clean":"2023-11-28T18:23:10.913943+0000","last_became_active":"2023-11-28T18:23:08.117973+0000","last_became_peered":"2023-11-28T18:23:08.117973+0000","last_unstale":"2023-11-28T18:23:10.913943+0000","last_undegraded":"2023-11-28T18:23:10.913943+0000","last_fullsized":"2023-11-28T18:23:10.913943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:57.485937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105530+0000","last_change":"2023-11-28T18:23:08.139744+0000","last_active":"2023-11-28T18:23:09.105530+0000","last_peered":"2023-11-28T18:23:09.105530+0000","last_clean":"2023-11-28T18:23:09.105530+0000","last_became_active":"2023-11-28T18:23:08.139464+0000","last_became_peered":"2023-11-28T18:23:08.139464+0000","last_unstale":"2023-11-28T18:23:09.105530+0000","last_undegraded":"2023-11-28T18:23:09.105530+0000","last_fullsized":"2023-11-28T18:23:09.105530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:19:34.780979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113036+0000","last_change":"2023-11-28T18:23:08.119103+0000","last_active":"2023-11-28T18:23:09.113036+0000","last_peered":"2023-11-28T18:23:09.113036+0000","last_clean":"2023-11-28T18:23:09.113036+0000","last_became_active":"2023-11-28T18:23:08.116080+0000","last_became_peered":"2023-11-28T18:23:08.116080+0000","last_unstale":"2023-11-28T18:23:09.113036+0000","last_undegraded":"2023-11-28T18:23:09.113036+0000","last_fullsized":"2023-11-28T18:23:09.113036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:44.279413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113688+0000","last_change":"2023-11-28T18:23:08.126817+0000","last_active":"2023-11-28T18:23:09.113688+0000","last_peered":"2023-11-28T18:23:09.113688+0000","last_clean":"2023-11-28T18:23:09.113688+0000","last_became_active":"2023-11-28T18:23:08.126540+0000","last_became_peered":"2023-11-28T18:23:08.126540+0000","last_unstale":"2023-11-28T18:23:09.113688+0000","last_undegraded":"2023-11-28T18:23:09.113688+0000","last_fullsized":"2023-11-28T18:23:09.113688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:08.807847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271940+0000","last_change":"2023-11-28T18:23:08.121775+0000","last_active":"2023-11-28T18:23:10.271940+0000","last_peered":"2023-11-28T18:23:10.271940+0000","last_clean":"2023-11-28T18:23:10.271940+0000","last_became_active":"2023-11-28T18:23:08.121466+0000","last_became_peered":"2023-11-28T18:23:08.121466+0000","last_unstale":"2023-11-28T18:23:10.271940+0000","last_undegraded":"2023-11-28T18:23:10.271940+0000","last_fullsized":"2023-11-28T18:23:10.271940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:32:51.612113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001115+0000","last_change":"2023-11-28T18:23:08.138282+0000","last_active":"2023-11-28T18:23:11.001115+0000","last_peered":"2023-11-28T18:23:11.001115+0000","last_clean":"2023-11-28T18:23:11.001115+0000","last_became_active":"2023-11-28T18:23:08.137171+0000","last_became_peered":"2023-11-28T18:23:08.137171+0000","last_unstale":"2023-11-28T18:23:11.001115+0000","last_undegraded":"2023-11-28T18:23:11.001115+0000","last_fullsized":"2023-11-28T18:23:11.001115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:39:07.113697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110124+0000","last_change":"2023-11-28T18:23:08.133521+0000","last_active":"2023-11-28T18:23:09.110124+0000","last_peered":"2023-11-28T18:23:09.110124+0000","last_clean":"2023-11-28T18:23:09.110124+0000","last_became_active":"2023-11-28T18:23:08.133248+0000","last_became_peered":"2023-11-28T18:23:08.133248+0000","last_unstale":"2023-11-28T18:23:09.110124+0000","last_undegraded":"2023-11-28T18:23:09.110124+0000","last_fullsized":"2023-11-28T18:23:09.110124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:17:34.585808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914195+0000","last_change":"2023-11-28T18:23:08.129150+0000","last_active":"2023-11-28T18:23:10.914195+0000","last_peered":"2023-11-28T18:23:10.914195+0000","last_clean":"2023-11-28T18:23:10.914195+0000","last_became_active":"2023-11-28T18:23:08.128622+0000","last_became_peered":"2023-11-28T18:23:08.128622+0000","last_unstale":"2023-11-28T18:23:10.914195+0000","last_undegraded":"2023-11-28T18:23:10.914195+0000","last_fullsized":"2023-11-28T18:23:10.914195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:33.197161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113274+0000","last_change":"2023-11-28T18:23:08.131333+0000","last_active":"2023-11-28T18:23:09.113274+0000","last_peered":"2023-11-28T18:23:09.113274+0000","last_clean":"2023-11-28T18:23:09.113274+0000","last_became_active":"2023-11-28T18:23:08.130917+0000","last_became_peered":"2023-11-28T18:23:08.130917+0000","last_unstale":"2023-11-28T18:23:09.113274+0000","last_undegraded":"2023-11-28T18:23:09.113274+0000","last_fullsized":"2023-11-28T18:23:09.113274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:04:48.114417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565507+0000","last_change":"2023-11-28T18:23:08.132355+0000","last_active":"2023-11-28T18:23:09.565507+0000","last_peered":"2023-11-28T18:23:09.565507+0000","last_clean":"2023-11-28T18:23:09.565507+0000","last_became_active":"2023-11-28T18:23:08.132016+0000","last_became_peered":"2023-11-28T18:23:08.132016+0000","last_unstale":"2023-11-28T18:23:09.565507+0000","last_undegraded":"2023-11-28T18:23:09.565507+0000","last_fullsized":"2023-11-28T18:23:09.565507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:11.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113066+0000","last_change":"2023-11-28T18:23:08.119935+0000","last_active":"2023-11-28T18:23:09.113066+0000","last_peered":"2023-11-28T18:23:09.113066+0000","last_clean":"2023-11-28T18:23:09.113066+0000","last_became_active":"2023-11-28T18:23:08.118372+0000","last_became_peered":"2023-11-28T18:23:08.118372+0000","last_unstale":"2023-11-28T18:23:09.113066+0000","last_undegraded":"2023-11-28T18:23:09.113066+0000","last_fullsized":"2023-11-28T18:23:09.113066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:52.339024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.132272+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.131985+0000","last_became_peered":"2023-11-28T18:23:08.131985+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:44.594691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110084+0000","last_change":"2023-11-28T18:23:08.127816+0000","last_active":"2023-11-28T18:23:09.110084+0000","last_peered":"2023-11-28T18:23:09.110084+0000","last_clean":"2023-11-28T18:23:09.110084+0000","last_became_active":"2023-11-28T18:23:08.127380+0000","last_became_peered":"2023-11-28T18:23:08.127380+0000","last_unstale":"2023-11-28T18:23:09.110084+0000","last_undegraded":"2023-11-28T18:23:09.110084+0000","last_fullsized":"2023-11-28T18:23:09.110084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:31.242361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105706+0000","last_change":"2023-11-28T18:23:08.128188+0000","last_active":"2023-11-28T18:23:09.105706+0000","last_peered":"2023-11-28T18:23:09.105706+0000","last_clean":"2023-11-28T18:23:09.105706+0000","last_became_active":"2023-11-28T18:23:08.127860+0000","last_became_peered":"2023-11-28T18:23:08.127860+0000","last_unstale":"2023-11-28T18:23:09.105706+0000","last_undegraded":"2023-11-28T18:23:09.105706+0000","last_fullsized":"2023-11-28T18:23:09.105706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:12.095768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565712+0000","last_change":"2023-11-28T18:23:08.132381+0000","last_active":"2023-11-28T18:23:09.565712+0000","last_peered":"2023-11-28T18:23:09.565712+0000","last_clean":"2023-11-28T18:23:09.565712+0000","last_became_active":"2023-11-28T18:23:08.132072+0000","last_became_peered":"2023-11-28T18:23:08.132072+0000","last_unstale":"2023-11-28T18:23:09.565712+0000","last_undegraded":"2023-11-28T18:23:09.565712+0000","last_fullsized":"2023-11-28T18:23:09.565712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:16:38.666702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914263+0000","last_change":"2023-11-28T18:23:08.115379+0000","last_active":"2023-11-28T18:23:10.914263+0000","last_peered":"2023-11-28T18:23:10.914263+0000","last_clean":"2023-11-28T18:23:10.914263+0000","last_became_active":"2023-11-28T18:23:08.115236+0000","last_became_peered":"2023-11-28T18:23:08.115236+0000","last_unstale":"2023-11-28T18:23:10.914263+0000","last_undegraded":"2023-11-28T18:23:10.914263+0000","last_fullsized":"2023-11-28T18:23:10.914263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:26:52.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113293+0000","last_change":"2023-11-28T18:23:08.121402+0000","last_active":"2023-11-28T18:23:09.113293+0000","last_peered":"2023-11-28T18:23:09.113293+0000","last_clean":"2023-11-28T18:23:09.113293+0000","last_became_active":"2023-11-28T18:23:08.121129+0000","last_became_peered":"2023-11-28T18:23:08.121129+0000","last_unstale":"2023-11-28T18:23:09.113293+0000","last_undegraded":"2023-11-28T18:23:09.113293+0000","last_fullsized":"2023-11-28T18:23:09.113293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:02.927891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271829+0000","last_change":"2023-11-28T18:23:08.121594+0000","last_active":"2023-11-28T18:23:10.271829+0000","last_peered":"2023-11-28T18:23:10.271829+0000","last_clean":"2023-11-28T18:23:10.271829+0000","last_became_active":"2023-11-28T18:23:08.119009+0000","last_became_peered":"2023-11-28T18:23:08.119009+0000","last_unstale":"2023-11-28T18:23:10.271829+0000","last_undegraded":"2023-11-28T18:23:10.271829+0000","last_fullsized":"2023-11-28T18:23:10.271829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:38.801632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109676+0000","last_change":"2023-11-28T18:23:08.115403+0000","last_active":"2023-11-28T18:23:09.109676+0000","last_peered":"2023-11-28T18:23:09.109676+0000","last_clean":"2023-11-28T18:23:09.109676+0000","last_became_active":"2023-11-28T18:23:08.114955+0000","last_became_peered":"2023-11-28T18:23:08.114955+0000","last_unstale":"2023-11-28T18:23:09.109676+0000","last_undegraded":"2023-11-28T18:23:09.109676+0000","last_fullsized":"2023-11-28T18:23:09.109676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:41.242830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637404+0000","last_change":"2023-11-28T18:23:08.129745+0000","last_active":"2023-11-28T18:23:09.637404+0000","last_peered":"2023-11-28T18:23:09.637404+0000","last_clean":"2023-11-28T18:23:09.637404+0000","last_became_active":"2023-11-28T18:23:08.129388+0000","last_became_peered":"2023-11-28T18:23:08.129388+0000","last_unstale":"2023-11-28T18:23:09.637404+0000","last_undegraded":"2023-11-28T18:23:09.637404+0000","last_fullsized":"2023-11-28T18:23:09.637404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:50.570821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.743781+0000","last_change":"2023-11-28T18:23:07.664773+0000","last_active":"2023-11-28T18:23:18.743781+0000","last_peered":"2023-11-28T18:23:18.743781+0000","last_clean":"2023-11-28T18:23:18.743781+0000","last_became_active":"2023-11-28T18:23:07.664478+0000","last_became_peered":"2023-11-28T18:23:07.664478+0000","last_unstale":"2023-11-28T18:23:18.743781+0000","last_undegraded":"2023-11-28T18:23:18.743781+0000","last_fullsized":"2023-11-28T18:23:18.743781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:28.322454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914337+0000","last_change":"2023-11-28T18:23:08.129441+0000","last_active":"2023-11-28T18:23:10.914337+0000","last_peered":"2023-11-28T18:23:10.914337+0000","last_clean":"2023-11-28T18:23:10.914337+0000","last_became_active":"2023-11-28T18:23:08.129143+0000","last_became_peered":"2023-11-28T18:23:08.129143+0000","last_unstale":"2023-11-28T18:23:10.914337+0000","last_undegraded":"2023-11-28T18:23:10.914337+0000","last_fullsized":"2023-11-28T18:23:10.914337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:36.540819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.173128+0000","last_change":"2023-11-28T18:23:07.668192+0000","last_active":"2023-11-28T18:23:30.173128+0000","last_peered":"2023-11-28T18:23:30.173128+0000","last_clean":"2023-11-28T18:23:30.173128+0000","last_became_active":"2023-11-28T18:23:07.667780+0000","last_became_peered":"2023-11-28T18:23:07.667780+0000","last_unstale":"2023-11-28T18:23:30.173128+0000","last_undegraded":"2023-11-28T18:23:30.173128+0000","last_fullsized":"2023-11-28T18:23:30.173128+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:24.148981+0000","last_change":"2023-11-28T18:23:07.665608+0000","last_active":"2023-11-28T18:23:24.148981+0000","last_peered":"2023-11-28T18:23:24.148981+0000","last_clean":"2023-11-28T18:23:24.148981+0000","last_became_active":"2023-11-28T18:23:07.665361+0000","last_became_peered":"2023-11-28T18:23:07.665361+0000","last_unstale":"2023-11-28T18:23:24.148981+0000","last_undegraded":"2023-11-28T18:23:24.148981+0000","last_fullsized":"2023-11-28T18:23:24.148981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:17.279597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.118091+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.117524+0000","last_became_peered":"2023-11-28T18:23:08.117524+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:04.054226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171679+0000","last_change":"2023-11-28T18:23:07.667847+0000","last_active":"2023-11-28T18:23:30.171679+0000","last_peered":"2023-11-28T18:23:30.171679+0000","last_clean":"2023-11-28T18:23:30.171679+0000","last_became_active":"2023-11-28T18:23:07.667466+0000","last_became_peered":"2023-11-28T18:23:07.667466+0000","last_unstale":"2023-11-28T18:23:30.171679+0000","last_undegraded":"2023-11-28T18:23:30.171679+0000","last_fullsized":"2023-11-28T18:23:30.171679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:06.807146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.127497+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.127216+0000","last_became_peered":"2023-11-28T18:23:08.127216+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:03:14.999484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172207+0000","last_change":"2023-11-28T18:23:07.681698+0000","last_active":"2023-11-28T18:23:30.172207+0000","last_peered":"2023-11-28T18:23:30.172207+0000","last_clean":"2023-11-28T18:23:30.172207+0000","last_became_active":"2023-11-28T18:23:07.681365+0000","last_became_peered":"2023-11-28T18:23:07.681365+0000","last_unstale":"2023-11-28T18:23:30.172207+0000","last_undegraded":"2023-11-28T18:23:30.172207+0000","last_fullsized":"2023-11-28T18:23:30.172207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:21.985029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113361+0000","last_change":"2023-11-28T18:23:08.132797+0000","last_active":"2023-11-28T18:23:09.113361+0000","last_peered":"2023-11-28T18:23:09.113361+0000","last_clean":"2023-11-28T18:23:09.113361+0000","last_became_active":"2023-11-28T18:23:08.132525+0000","last_became_peered":"2023-11-28T18:23:08.132525+0000","last_unstale":"2023-11-28T18:23:09.113361+0000","last_undegraded":"2023-11-28T18:23:09.113361+0000","last_fullsized":"2023-11-28T18:23:09.113361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:12:10.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.248118+0000","last_change":"2023-11-28T18:23:07.672337+0000","last_active":"2023-11-28T18:23:35.248118+0000","last_peered":"2023-11-28T18:23:35.248118+0000","last_clean":"2023-11-28T18:23:35.248118+0000","last_became_active":"2023-11-28T18:23:07.671922+0000","last_became_peered":"2023-11-28T18:23:07.671922+0000","last_unstale":"2023-11-28T18:23:35.248118+0000","last_undegraded":"2023-11-28T18:23:35.248118+0000","last_fullsized":"2023-11-28T18:23:35.248118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:22.936770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914223+0000","last_change":"2023-11-28T18:23:08.127919+0000","last_active":"2023-11-28T18:23:10.914223+0000","last_peered":"2023-11-28T18:23:10.914223+0000","last_clean":"2023-11-28T18:23:10.914223+0000","last_became_active":"2023-11-28T18:23:08.127602+0000","last_became_peered":"2023-11-28T18:23:08.127602+0000","last_unstale":"2023-11-28T18:23:10.914223+0000","last_undegraded":"2023-11-28T18:23:10.914223+0000","last_fullsized":"2023-11-28T18:23:10.914223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:32:45.967018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.734841+0000","last_change":"2023-11-28T18:23:07.679219+0000","last_active":"2023-11-28T18:23:18.734841+0000","last_peered":"2023-11-28T18:23:18.734841+0000","last_clean":"2023-11-28T18:23:18.734841+0000","last_became_active":"2023-11-28T18:23:07.678920+0000","last_became_peered":"2023-11-28T18:23:07.678920+0000","last_unstale":"2023-11-28T18:23:18.734841+0000","last_undegraded":"2023-11-28T18:23:18.734841+0000","last_fullsized":"2023-11-28T18:23:18.734841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:02.332690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637820+0000","last_change":"2023-11-28T18:23:08.132429+0000","last_active":"2023-11-28T18:23:09.637820+0000","last_peered":"2023-11-28T18:23:09.637820+0000","last_clean":"2023-11-28T18:23:09.637820+0000","last_became_active":"2023-11-28T18:23:08.132106+0000","last_became_peered":"2023-11-28T18:23:08.132106+0000","last_unstale":"2023-11-28T18:23:09.637820+0000","last_undegraded":"2023-11-28T18:23:09.637820+0000","last_fullsized":"2023-11-28T18:23:09.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:52.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'12","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:43.822203+0000","last_change":"2023-11-28T18:23:07.667479+0000","last_active":"2023-11-28T18:23:43.822203+0000","last_peered":"2023-11-28T18:23:43.822203+0000","last_clean":"2023-11-28T18:23:43.822203+0000","last_became_active":"2023-11-28T18:23:07.665310+0000","last_became_peered":"2023-11-28T18:23:07.665310+0000","last_unstale":"2023-11-28T18:23:43.822203+0000","last_undegraded":"2023-11-28T18:23:43.822203+0000","last_fullsized":"2023-11-28T18:23:43.822203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:03.174425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2052,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000928+0000","last_change":"2023-11-28T18:23:08.136812+0000","last_active":"2023-11-28T18:23:11.000928+0000","last_peered":"2023-11-28T18:23:11.000928+0000","last_clean":"2023-11-28T18:23:11.000928+0000","last_became_active":"2023-11-28T18:23:08.136525+0000","last_became_peered":"2023-11-28T18:23:08.136525+0000","last_unstale":"2023-11-28T18:23:11.000928+0000","last_undegraded":"2023-11-28T18:23:11.000928+0000","last_fullsized":"2023-11-28T18:23:11.000928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:07.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.249898+0000","last_change":"2023-11-28T18:23:07.660519+0000","last_active":"2023-11-28T18:23:35.249898+0000","last_peered":"2023-11-28T18:23:35.249898+0000","last_clean":"2023-11-28T18:23:35.249898+0000","last_became_active":"2023-11-28T18:23:07.660205+0000","last_became_peered":"2023-11-28T18:23:07.660205+0000","last_unstale":"2023-11-28T18:23:35.249898+0000","last_undegraded":"2023-11-28T18:23:35.249898+0000","last_fullsized":"2023-11-28T18:23:35.249898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:04:54.125737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914264+0000","last_change":"2023-11-28T18:23:08.127921+0000","last_active":"2023-11-28T18:23:10.914264+0000","last_peered":"2023-11-28T18:23:10.914264+0000","last_clean":"2023-11-28T18:23:10.914264+0000","last_became_active":"2023-11-28T18:23:08.127678+0000","last_became_peered":"2023-11-28T18:23:08.127678+0000","last_unstale":"2023-11-28T18:23:10.914264+0000","last_undegraded":"2023-11-28T18:23:10.914264+0000","last_fullsized":"2023-11-28T18:23:10.914264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:38:47.175911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171696+0000","last_change":"2023-11-28T18:23:07.671589+0000","last_active":"2023-11-28T18:23:30.171696+0000","last_peered":"2023-11-28T18:23:30.171696+0000","last_clean":"2023-11-28T18:23:30.171696+0000","last_became_active":"2023-11-28T18:23:07.671291+0000","last_became_peered":"2023-11-28T18:23:07.671291+0000","last_unstale":"2023-11-28T18:23:30.171696+0000","last_undegraded":"2023-11-28T18:23:30.171696+0000","last_fullsized":"2023-11-28T18:23:30.171696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:23.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000866+0000","last_change":"2023-11-28T18:23:08.141316+0000","last_active":"2023-11-28T18:23:11.000866+0000","last_peered":"2023-11-28T18:23:11.000866+0000","last_clean":"2023-11-28T18:23:11.000866+0000","last_became_active":"2023-11-28T18:23:08.141011+0000","last_became_peered":"2023-11-28T18:23:08.141011+0000","last_unstale":"2023-11-28T18:23:11.000866+0000","last_undegraded":"2023-11-28T18:23:11.000866+0000","last_fullsized":"2023-11-28T18:23:11.000866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:13:25.849340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172815+0000","last_change":"2023-11-28T18:23:07.671192+0000","last_active":"2023-11-28T18:23:30.172815+0000","last_peered":"2023-11-28T18:23:30.172815+0000","last_clean":"2023-11-28T18:23:30.172815+0000","last_became_active":"2023-11-28T18:23:07.670812+0000","last_became_peered":"2023-11-28T18:23:07.670812+0000","last_unstale":"2023-11-28T18:23:30.172815+0000","last_undegraded":"2023-11-28T18:23:30.172815+0000","last_fullsized":"2023-11-28T18:23:30.172815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:28.806061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109506+0000","last_change":"2023-11-28T18:23:08.115229+0000","last_active":"2023-11-28T18:23:09.109506+0000","last_peered":"2023-11-28T18:23:09.109506+0000","last_clean":"2023-11-28T18:23:09.109506+0000","last_became_active":"2023-11-28T18:23:08.114541+0000","last_became_peered":"2023-11-28T18:23:08.114541+0000","last_unstale":"2023-11-28T18:23:09.109506+0000","last_undegraded":"2023-11-28T18:23:09.109506+0000","last_fullsized":"2023-11-28T18:23:09.109506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:38.816731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":29,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171599+0000","last_change":"2023-11-28T18:23:07.673554+0000","last_active":"2023-11-28T18:23:30.171599+0000","last_peered":"2023-11-28T18:23:30.171599+0000","last_clean":"2023-11-28T18:23:30.171599+0000","last_became_active":"2023-11-28T18:23:07.672923+0000","last_became_peered":"2023-11-28T18:23:07.672923+0000","last_unstale":"2023-11-28T18:23:30.171599+0000","last_undegraded":"2023-11-28T18:23:30.171599+0000","last_fullsized":"2023-11-28T18:23:30.171599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:49.805572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000699+0000","last_change":"2023-11-28T18:23:08.141169+0000","last_active":"2023-11-28T18:23:11.000699+0000","last_peered":"2023-11-28T18:23:11.000699+0000","last_clean":"2023-11-28T18:23:11.000699+0000","last_became_active":"2023-11-28T18:23:08.140801+0000","last_became_peered":"2023-11-28T18:23:08.140801+0000","last_unstale":"2023-11-28T18:23:11.000699+0000","last_undegraded":"2023-11-28T18:23:11.000699+0000","last_fullsized":"2023-11-28T18:23:11.000699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:16.669717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.667707+0000","last_change":"2023-11-28T18:23:07.680094+0000","last_active":"2023-11-28T18:23:13.667707+0000","last_peered":"2023-11-28T18:23:13.667707+0000","last_clean":"2023-11-28T18:23:13.667707+0000","last_became_active":"2023-11-28T18:23:07.679801+0000","last_became_peered":"2023-11-28T18:23:07.679801+0000","last_unstale":"2023-11-28T18:23:13.667707+0000","last_undegraded":"2023-11-28T18:23:13.667707+0000","last_fullsized":"2023-11-28T18:23:13.667707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:02.916155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001208+0000","last_change":"2023-11-28T18:23:08.137825+0000","last_active":"2023-11-28T18:23:11.001208+0000","last_peered":"2023-11-28T18:23:11.001208+0000","last_clean":"2023-11-28T18:23:11.001208+0000","last_became_active":"2023-11-28T18:23:08.137499+0000","last_became_peered":"2023-11-28T18:23:08.137499+0000","last_unstale":"2023-11-28T18:23:11.001208+0000","last_undegraded":"2023-11-28T18:23:11.001208+0000","last_fullsized":"2023-11-28T18:23:11.001208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:42.807830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.244409+0000","last_change":"2023-11-28T18:23:07.670359+0000","last_active":"2023-11-28T18:23:35.244409+0000","last_peered":"2023-11-28T18:23:35.244409+0000","last_clean":"2023-11-28T18:23:35.244409+0000","last_became_active":"2023-11-28T18:23:07.669687+0000","last_became_peered":"2023-11-28T18:23:07.669687+0000","last_unstale":"2023-11-28T18:23:35.244409+0000","last_undegraded":"2023-11-28T18:23:35.244409+0000","last_fullsized":"2023-11-28T18:23:35.244409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:45.450623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001353+0000","last_change":"2023-11-28T18:23:08.141300+0000","last_active":"2023-11-28T18:23:11.001353+0000","last_peered":"2023-11-28T18:23:11.001353+0000","last_clean":"2023-11-28T18:23:11.001353+0000","last_became_active":"2023-11-28T18:23:08.141037+0000","last_became_peered":"2023-11-28T18:23:08.141037+0000","last_unstale":"2023-11-28T18:23:11.001353+0000","last_undegraded":"2023-11-28T18:23:11.001353+0000","last_fullsized":"2023-11-28T18:23:11.001353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:11.093903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172901+0000","last_change":"2023-11-28T18:23:07.673086+0000","last_active":"2023-11-28T18:23:30.172901+0000","last_peered":"2023-11-28T18:23:30.172901+0000","last_clean":"2023-11-28T18:23:30.172901+0000","last_became_active":"2023-11-28T18:23:07.672773+0000","last_became_peered":"2023-11-28T18:23:07.672773+0000","last_unstale":"2023-11-28T18:23:30.172901+0000","last_undegraded":"2023-11-28T18:23:30.172901+0000","last_fullsized":"2023-11-28T18:23:30.172901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:34.962428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113586+0000","last_change":"2023-11-28T18:23:08.121987+0000","last_active":"2023-11-28T18:23:09.113586+0000","last_peered":"2023-11-28T18:23:09.113586+0000","last_clean":"2023-11-28T18:23:09.113586+0000","last_became_active":"2023-11-28T18:23:08.121279+0000","last_became_peered":"2023-11-28T18:23:08.121279+0000","last_unstale":"2023-11-28T18:23:09.113586+0000","last_undegraded":"2023-11-28T18:23:09.113586+0000","last_fullsized":"2023-11-28T18:23:09.113586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:47.973973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.174488+0000","last_change":"2023-11-28T18:23:07.681312+0000","last_active":"2023-11-28T18:23:30.174488+0000","last_peered":"2023-11-28T18:23:30.174488+0000","last_clean":"2023-11-28T18:23:30.174488+0000","last_became_active":"2023-11-28T18:23:07.680806+0000","last_became_peered":"2023-11-28T18:23:07.680806+0000","last_unstale":"2023-11-28T18:23:30.174488+0000","last_undegraded":"2023-11-28T18:23:30.174488+0000","last_fullsized":"2023-11-28T18:23:30.174488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:58:41.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001014+0000","last_change":"2023-11-28T18:23:08.137874+0000","last_active":"2023-11-28T18:23:11.001014+0000","last_peered":"2023-11-28T18:23:11.001014+0000","last_clean":"2023-11-28T18:23:11.001014+0000","last_became_active":"2023-11-28T18:23:08.137104+0000","last_became_peered":"2023-11-28T18:23:08.137104+0000","last_unstale":"2023-11-28T18:23:11.001014+0000","last_undegraded":"2023-11-28T18:23:11.001014+0000","last_fullsized":"2023-11-28T18:23:11.001014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:39:11.857836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117053+0000","last_change":"2023-11-28T18:22:56.774057+0000","last_active":"2023-11-28T18:23:08.117053+0000","last_peered":"2023-11-28T18:23:08.117053+0000","last_clean":"2023-11-28T18:23:08.117053+0000","last_became_active":"2023-11-28T18:22:56.772763+0000","last_became_peered":"2023-11-28T18:22:56.772763+0000","last_unstale":"2023-11-28T18:23:08.117053+0000","last_undegraded":"2023-11-28T18:23:08.117053+0000","last_fullsized":"2023-11-28T18:23:08.117053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_clean_scrub_stamp":"2023-11-28T18:22:55.591288+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:05.470408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172368+0000","last_change":"2023-11-28T18:23:07.674072+0000","last_active":"2023-11-28T18:23:30.172368+0000","last_peered":"2023-11-28T18:23:30.172368+0000","last_clean":"2023-11-28T18:23:30.172368+0000","last_became_active":"2023-11-28T18:23:07.673220+0000","last_became_peered":"2023-11-28T18:23:07.673220+0000","last_unstale":"2023-11-28T18:23:30.172368+0000","last_undegraded":"2023-11-28T18:23:30.172368+0000","last_fullsized":"2023-11-28T18:23:30.172368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:58.524112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565598+0000","last_change":"2023-11-28T18:23:08.130989+0000","last_active":"2023-11-28T18:23:09.565598+0000","last_peered":"2023-11-28T18:23:09.565598+0000","last_clean":"2023-11-28T18:23:09.565598+0000","last_became_active":"2023-11-28T18:23:08.130646+0000","last_became_peered":"2023-11-28T18:23:08.130646+0000","last_unstale":"2023-11-28T18:23:09.565598+0000","last_undegraded":"2023-11-28T18:23:09.565598+0000","last_fullsized":"2023-11-28T18:23:09.565598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:50.371045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:24.150194+0000","last_change":"2023-11-28T18:23:07.686046+0000","last_active":"2023-11-28T18:23:24.150194+0000","last_peered":"2023-11-28T18:23:24.150194+0000","last_clean":"2023-11-28T18:23:24.150194+0000","last_became_active":"2023-11-28T18:23:07.685739+0000","last_became_peered":"2023-11-28T18:23:07.685739+0000","last_unstale":"2023-11-28T18:23:24.150194+0000","last_undegraded":"2023-11-28T18:23:24.150194+0000","last_fullsized":"2023-11-28T18:23:24.150194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:06.485631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913951+0000","last_change":"2023-11-28T18:23:08.130355+0000","last_active":"2023-11-28T18:23:10.913951+0000","last_peered":"2023-11-28T18:23:10.913951+0000","last_clean":"2023-11-28T18:23:10.913951+0000","last_became_active":"2023-11-28T18:23:08.129946+0000","last_became_peered":"2023-11-28T18:23:08.129946+0000","last_unstale":"2023-11-28T18:23:10.913951+0000","last_undegraded":"2023-11-28T18:23:10.913951+0000","last_fullsized":"2023-11-28T18:23:10.913951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:58.140191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.245973+0000","last_change":"2023-11-28T18:23:07.664827+0000","last_active":"2023-11-28T18:23:35.245973+0000","last_peered":"2023-11-28T18:23:35.245973+0000","last_clean":"2023-11-28T18:23:35.245973+0000","last_became_active":"2023-11-28T18:23:07.664503+0000","last_became_peered":"2023-11-28T18:23:07.664503+0000","last_unstale":"2023-11-28T18:23:35.245973+0000","last_undegraded":"2023-11-28T18:23:35.245973+0000","last_fullsized":"2023-11-28T18:23:35.245973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.238694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.123498+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.123197+0000","last_became_peered":"2023-11-28T18:23:08.123197+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:08:42.817848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250563+0000","last_change":"2023-11-28T18:23:07.673345+0000","last_active":"2023-11-28T18:23:35.250563+0000","last_peered":"2023-11-28T18:23:35.250563+0000","last_clean":"2023-11-28T18:23:35.250563+0000","last_became_active":"2023-11-28T18:23:07.672982+0000","last_became_peered":"2023-11-28T18:23:07.672982+0000","last_unstale":"2023-11-28T18:23:35.250563+0000","last_undegraded":"2023-11-28T18:23:35.250563+0000","last_fullsized":"2023-11-28T18:23:35.250563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:26.594232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637566+0000","last_change":"2023-11-28T18:23:08.129091+0000","last_active":"2023-11-28T18:23:09.637566+0000","last_peered":"2023-11-28T18:23:09.637566+0000","last_clean":"2023-11-28T18:23:09.637566+0000","last_became_active":"2023-11-28T18:23:08.128589+0000","last_became_peered":"2023-11-28T18:23:08.128589+0000","last_unstale":"2023-11-28T18:23:09.637566+0000","last_undegraded":"2023-11-28T18:23:09.637566+0000","last_fullsized":"2023-11-28T18:23:09.637566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:28.975412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.493169+0000","last_change":"2023-11-28T18:23:07.671874+0000","last_active":"2023-11-28T18:23:09.493169+0000","last_peered":"2023-11-28T18:23:09.493169+0000","last_clean":"2023-11-28T18:23:09.493169+0000","last_became_active":"2023-11-28T18:23:07.671500+0000","last_became_peered":"2023-11-28T18:23:07.671500+0000","last_unstale":"2023-11-28T18:23:09.493169+0000","last_undegraded":"2023-11-28T18:23:09.493169+0000","last_fullsized":"2023-11-28T18:23:09.493169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:42.922770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113612+0000","last_change":"2023-11-28T18:23:08.127419+0000","last_active":"2023-11-28T18:23:09.113612+0000","last_peered":"2023-11-28T18:23:09.113612+0000","last_clean":"2023-11-28T18:23:09.113612+0000","last_became_active":"2023-11-28T18:23:08.127157+0000","last_became_peered":"2023-11-28T18:23:08.127157+0000","last_unstale":"2023-11-28T18:23:09.113612+0000","last_undegraded":"2023-11-28T18:23:09.113612+0000","last_fullsized":"2023-11-28T18:23:09.113612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:08.473882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.173374+0000","last_change":"2023-11-28T18:23:07.684725+0000","last_active":"2023-11-28T18:23:30.173374+0000","last_peered":"2023-11-28T18:23:30.173374+0000","last_clean":"2023-11-28T18:23:30.173374+0000","last_became_active":"2023-11-28T18:23:07.684363+0000","last_became_peered":"2023-11-28T18:23:07.684363+0000","last_unstale":"2023-11-28T18:23:30.173374+0000","last_undegraded":"2023-11-28T18:23:30.173374+0000","last_fullsized":"2023-11-28T18:23:30.173374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:16.380789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105690+0000","last_change":"2023-11-28T18:23:08.120741+0000","last_active":"2023-11-28T18:23:09.105690+0000","last_peered":"2023-11-28T18:23:09.105690+0000","last_clean":"2023-11-28T18:23:09.105690+0000","last_became_active":"2023-11-28T18:23:08.120421+0000","last_became_peered":"2023-11-28T18:23:08.120421+0000","last_unstale":"2023-11-28T18:23:09.105690+0000","last_undegraded":"2023-11-28T18:23:09.105690+0000","last_fullsized":"2023-11-28T18:23:09.105690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:53:27.131631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.175382+0000","last_change":"2023-11-28T18:23:07.672505+0000","last_active":"2023-11-28T18:23:30.175382+0000","last_peered":"2023-11-28T18:23:30.175382+0000","last_clean":"2023-11-28T18:23:30.175382+0000","last_became_active":"2023-11-28T18:23:07.672204+0000","last_became_peered":"2023-11-28T18:23:07.672204+0000","last_unstale":"2023-11-28T18:23:30.175382+0000","last_undegraded":"2023-11-28T18:23:30.175382+0000","last_fullsized":"2023-11-28T18:23:30.175382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:01.341643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565869+0000","last_change":"2023-11-28T18:23:08.126198+0000","last_active":"2023-11-28T18:23:09.565869+0000","last_peered":"2023-11-28T18:23:09.565869+0000","last_clean":"2023-11-28T18:23:09.565869+0000","last_became_active":"2023-11-28T18:23:08.125935+0000","last_became_peered":"2023-11-28T18:23:08.125935+0000","last_unstale":"2023-11-28T18:23:09.565869+0000","last_undegraded":"2023-11-28T18:23:09.565869+0000","last_fullsized":"2023-11-28T18:23:09.565869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.473922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.252470+0000","last_change":"2023-11-28T18:23:07.672170+0000","last_active":"2023-11-28T18:23:35.252470+0000","last_peered":"2023-11-28T18:23:35.252470+0000","last_clean":"2023-11-28T18:23:35.252470+0000","last_became_active":"2023-11-28T18:23:07.671683+0000","last_became_peered":"2023-11-28T18:23:07.671683+0000","last_unstale":"2023-11-28T18:23:35.252470+0000","last_undegraded":"2023-11-28T18:23:35.252470+0000","last_fullsized":"2023-11-28T18:23:35.252470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:41.860341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914424+0000","last_change":"2023-11-28T18:23:08.119844+0000","last_active":"2023-11-28T18:23:10.914424+0000","last_peered":"2023-11-28T18:23:10.914424+0000","last_clean":"2023-11-28T18:23:10.914424+0000","last_became_active":"2023-11-28T18:23:08.119525+0000","last_became_peered":"2023-11-28T18:23:08.119525+0000","last_unstale":"2023-11-28T18:23:10.914424+0000","last_undegraded":"2023-11-28T18:23:10.914424+0000","last_fullsized":"2023-11-28T18:23:10.914424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:34:00.230528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.727126+0000","last_change":"2023-11-28T18:23:07.685651+0000","last_active":"2023-11-28T18:23:18.727126+0000","last_peered":"2023-11-28T18:23:18.727126+0000","last_clean":"2023-11-28T18:23:18.727126+0000","last_became_active":"2023-11-28T18:23:07.683869+0000","last_became_peered":"2023-11-28T18:23:07.683869+0000","last_unstale":"2023-11-28T18:23:18.727126+0000","last_undegraded":"2023-11-28T18:23:18.727126+0000","last_fullsized":"2023-11-28T18:23:18.727126+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:35.383852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637574+0000","last_change":"2023-11-28T18:23:08.130016+0000","last_active":"2023-11-28T18:23:09.637574+0000","last_peered":"2023-11-28T18:23:09.637574+0000","last_clean":"2023-11-28T18:23:09.637574+0000","last_became_active":"2023-11-28T18:23:08.129586+0000","last_became_peered":"2023-11-28T18:23:08.129586+0000","last_unstale":"2023-11-28T18:23:09.637574+0000","last_undegraded":"2023-11-28T18:23:09.637574+0000","last_fullsized":"2023-11-28T18:23:09.637574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:00.449941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.245267+0000","last_change":"2023-11-28T18:23:07.671773+0000","last_active":"2023-11-28T18:23:35.245267+0000","last_peered":"2023-11-28T18:23:35.245267+0000","last_clean":"2023-11-28T18:23:35.245267+0000","last_became_active":"2023-11-28T18:23:07.671465+0000","last_became_peered":"2023-11-28T18:23:07.671465+0000","last_unstale":"2023-11-28T18:23:35.245267+0000","last_undegraded":"2023-11-28T18:23:35.245267+0000","last_fullsized":"2023-11-28T18:23:35.245267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:44.862780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565865+0000","last_change":"2023-11-28T18:23:08.118146+0000","last_active":"2023-11-28T18:23:09.565865+0000","last_peered":"2023-11-28T18:23:09.565865+0000","last_clean":"2023-11-28T18:23:09.565865+0000","last_became_active":"2023-11-28T18:23:08.117848+0000","last_became_peered":"2023-11-28T18:23:08.117848+0000","last_unstale":"2023-11-28T18:23:09.565865+0000","last_undegraded":"2023-11-28T18:23:09.565865+0000","last_fullsized":"2023-11-28T18:23:09.565865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:33:20.745184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.169912+0000","last_change":"2023-11-28T18:23:07.676709+0000","last_active":"2023-11-28T18:23:30.169912+0000","last_peered":"2023-11-28T18:23:30.169912+0000","last_clean":"2023-11-28T18:23:30.169912+0000","last_became_active":"2023-11-28T18:23:07.676414+0000","last_became_peered":"2023-11-28T18:23:07.676414+0000","last_unstale":"2023-11-28T18:23:30.169912+0000","last_undegraded":"2023-11-28T18:23:30.169912+0000","last_fullsized":"2023-11-28T18:23:30.169912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.282005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565753+0000","last_change":"2023-11-28T18:23:08.112410+0000","last_active":"2023-11-28T18:23:09.565753+0000","last_peered":"2023-11-28T18:23:09.565753+0000","last_clean":"2023-11-28T18:23:09.565753+0000","last_became_active":"2023-11-28T18:23:08.112097+0000","last_became_peered":"2023-11-28T18:23:08.112097+0000","last_unstale":"2023-11-28T18:23:09.565753+0000","last_undegraded":"2023-11-28T18:23:09.565753+0000","last_fullsized":"2023-11-28T18:23:09.565753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:57.891789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.719092+0000","last_change":"2023-11-28T18:23:07.684818+0000","last_active":"2023-11-28T18:23:18.719092+0000","last_peered":"2023-11-28T18:23:18.719092+0000","last_clean":"2023-11-28T18:23:18.719092+0000","last_became_active":"2023-11-28T18:23:07.684502+0000","last_became_peered":"2023-11-28T18:23:07.684502+0000","last_unstale":"2023-11-28T18:23:18.719092+0000","last_undegraded":"2023-11-28T18:23:18.719092+0000","last_fullsized":"2023-11-28T18:23:18.719092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:16:21.223266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109527+0000","last_change":"2023-11-28T18:23:08.133972+0000","last_active":"2023-11-28T18:23:09.109527+0000","last_peered":"2023-11-28T18:23:09.109527+0000","last_clean":"2023-11-28T18:23:09.109527+0000","last_became_active":"2023-11-28T18:23:08.133747+0000","last_became_peered":"2023-11-28T18:23:08.133747+0000","last_unstale":"2023-11-28T18:23:09.109527+0000","last_undegraded":"2023-11-28T18:23:09.109527+0000","last_fullsized":"2023-11-28T18:23:09.109527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:24.968241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113538+0000","last_change":"2023-11-28T18:23:08.134117+0000","last_active":"2023-11-28T18:23:09.113538+0000","last_peered":"2023-11-28T18:23:09.113538+0000","last_clean":"2023-11-28T18:23:09.113538+0000","last_became_active":"2023-11-28T18:23:08.133801+0000","last_became_peered":"2023-11-28T18:23:08.133801+0000","last_unstale":"2023-11-28T18:23:09.113538+0000","last_undegraded":"2023-11-28T18:23:09.113538+0000","last_fullsized":"2023-11-28T18:23:09.113538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:02.370862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250642+0000","last_change":"2023-11-28T18:23:07.686399+0000","last_active":"2023-11-28T18:23:35.250642+0000","last_peered":"2023-11-28T18:23:35.250642+0000","last_clean":"2023-11-28T18:23:35.250642+0000","last_became_active":"2023-11-28T18:23:07.686051+0000","last_became_peered":"2023-11-28T18:23:07.686051+0000","last_unstale":"2023-11-28T18:23:35.250642+0000","last_undegraded":"2023-11-28T18:23:35.250642+0000","last_fullsized":"2023-11-28T18:23:35.250642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:11.797516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113317+0000","last_change":"2023-11-28T18:23:08.134870+0000","last_active":"2023-11-28T18:23:09.113317+0000","last_peered":"2023-11-28T18:23:09.113317+0000","last_clean":"2023-11-28T18:23:09.113317+0000","last_became_active":"2023-11-28T18:23:08.133732+0000","last_became_peered":"2023-11-28T18:23:08.133732+0000","last_unstale":"2023-11-28T18:23:09.113317+0000","last_undegraded":"2023-11-28T18:23:09.113317+0000","last_fullsized":"2023-11-28T18:23:09.113317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:01.364242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.252608+0000","last_change":"2023-11-28T18:23:07.682230+0000","last_active":"2023-11-28T18:23:35.252608+0000","last_peered":"2023-11-28T18:23:35.252608+0000","last_clean":"2023-11-28T18:23:35.252608+0000","last_became_active":"2023-11-28T18:23:07.681715+0000","last_became_peered":"2023-11-28T18:23:07.681715+0000","last_unstale":"2023-11-28T18:23:35.252608+0000","last_undegraded":"2023-11-28T18:23:35.252608+0000","last_fullsized":"2023-11-28T18:23:35.252608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:02.343996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914216+0000","last_change":"2023-11-28T18:23:08.128927+0000","last_active":"2023-11-28T18:23:10.914216+0000","last_peered":"2023-11-28T18:23:10.914216+0000","last_clean":"2023-11-28T18:23:10.914216+0000","last_became_active":"2023-11-28T18:23:08.127775+0000","last_became_peered":"2023-11-28T18:23:08.127775+0000","last_unstale":"2023-11-28T18:23:10.914216+0000","last_undegraded":"2023-11-28T18:23:10.914216+0000","last_fullsized":"2023-11-28T18:23:10.914216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:50:05.531589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.669760+0000","last_change":"2023-11-28T18:23:07.667929+0000","last_active":"2023-11-28T18:23:13.669760+0000","last_peered":"2023-11-28T18:23:13.669760+0000","last_clean":"2023-11-28T18:23:13.669760+0000","last_became_active":"2023-11-28T18:23:07.667666+0000","last_became_peered":"2023-11-28T18:23:07.667666+0000","last_unstale":"2023-11-28T18:23:13.669760+0000","last_undegraded":"2023-11-28T18:23:13.669760+0000","last_fullsized":"2023-11-28T18:23:13.669760+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:32.132092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565609+0000","last_change":"2023-11-28T18:23:08.126404+0000","last_active":"2023-11-28T18:23:09.565609+0000","last_peered":"2023-11-28T18:23:09.565609+0000","last_clean":"2023-11-28T18:23:09.565609+0000","last_became_active":"2023-11-28T18:23:08.125315+0000","last_became_peered":"2023-11-28T18:23:08.125315+0000","last_unstale":"2023-11-28T18:23:09.565609+0000","last_undegraded":"2023-11-28T18:23:09.565609+0000","last_fullsized":"2023-11-28T18:23:09.565609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:24:59.527642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":25,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250692+0000","last_change":"2023-11-28T18:23:07.677758+0000","last_active":"2023-11-28T18:23:35.250692+0000","last_peered":"2023-11-28T18:23:35.250692+0000","last_clean":"2023-11-28T18:23:35.250692+0000","last_became_active":"2023-11-28T18:23:07.676902+0000","last_became_peered":"2023-11-28T18:23:07.676902+0000","last_unstale":"2023-11-28T18:23:35.250692+0000","last_undegraded":"2023-11-28T18:23:35.250692+0000","last_fullsized":"2023-11-28T18:23:35.250692+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:50.953647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9794,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":835584,"data_stored":663818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":74192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":74842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":72240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":73346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":72812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":233472,"data_stored":72442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":827392,"data_stored":662152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T18:23:49.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T18:23:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12cd3b8640 1 Processor -- start 2023-11-28T18:23:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12cd3b8640 1 -- start start 2023-11-28T18:23:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12cd3b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c8153f00 0x7f12c80ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12cd3b8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12c80cb020 0x7f12c80b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12cd3b8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c815d420 con 0x7f12c80cb5a0 2023-11-28T18:23:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12cd3b8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c80ca900 con 0x7f12c80cb020 2023-11-28T18:23:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12cd3b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c815d700 con 0x7f12c8153f00 2023-11-28T18:23:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c67fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12c80cb020 0x7f12c80b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c67fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12c80cb020 0x7f12c80b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47494/0 (socket says 172.21.15.7:47494) 2023-11-28T18:23:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c67fc640 1 -- 172.21.15.7:0/2675528203 learned_addr learned my addr 172.21.15.7:0/2675528203 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c6ffd640 1 --2- 172.21.15.7:0/2675528203 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c8153f00 0x7f12c80ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c67fc640 1 -- 172.21.15.7:0/2675528203 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c8153f00 msgr2=0x7f12c80ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c67fc640 1 --2- 172.21.15.7:0/2675528203 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c8153f00 0x7f12c80ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c67fc640 1 -- 172.21.15.7:0/2675528203 >> v1:172.21.15.193:6789/0 conn(0x7f12c80cb5a0 legacy=0x7f12c80cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c67fc640 1 -- 172.21.15.7:0/2675528203 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c815d9e0 con 0x7f12c80cb020 2023-11-28T18:23:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.899+0000 7f12c6ffd640 1 --2- 172.21.15.7:0/2675528203 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c8153f00 0x7f12c80ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12c67fc640 1 --2- 172.21.15.7:0/2675528203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c80b1fc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f12c806c640 tx=0x7f12bc01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4faaa4a1d993d525 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12c5ffb640 1 -- 172.21.15.7:0/2675528203 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc0a8970 con 0x7f12c80cb020 2023-11-28T18:23:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12c5ffb640 1 -- 172.21.15.7:0/2675528203 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc0a8ad0 con 0x7f12c80cb020 2023-11-28T18:23:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12c5ffb640 1 -- 172.21.15.7:0/2675528203 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc0a8e60 con 0x7f12c80cb020 2023-11-28T18:23:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 -- 172.21.15.7:0/2675528203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 msgr2=0x7f12c80b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/2675528203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c80b1fc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f12c806c640 tx=0x7f12bc01ecb0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 -- 172.21.15.7:0/2675528203 shutdown_connections 2023-11-28T18:23:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/2675528203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c80b1fc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/2675528203 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c8153f00 0x7f12c80ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 -- 172.21.15.7:0/2675528203 >> 172.21.15.7:0/2675528203 conn(0x7f12c806c060 msgr2=0x7f12c80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 -- 172.21.15.7:0/2675528203 shutdown_connections 2023-11-28T18:23:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 -- 172.21.15.7:0/2675528203 wait complete. 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 Processor -- start 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.903+0000 7f12cd3b8640 1 -- start start 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c81f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c80cb5a0 0x7f12c80ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12c8153f00 0x7f12c8159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c8158260 con 0x7f12c80cb020 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c81e6770 con 0x7f12c8153f00 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c81e6a50 con 0x7f12c80cb5a0 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c81f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c81f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47498/0 (socket says 172.21.15.7:47498) 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 -- 172.21.15.7:0/3394046390 learned_addr learned my addr 172.21.15.7:0/3394046390 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c77fe640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12c8153f00 0x7f12c8159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c67fc640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c80cb5a0 0x7f12c80ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 -- 172.21.15.7:0/3394046390 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c80cb5a0 msgr2=0x7f12c80ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c80cb5a0 0x7f12c80ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 -- 172.21.15.7:0/3394046390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12c8153f00 msgr2=0x7f12c8159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12c8153f00 0x7f12c8159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 -- 172.21.15.7:0/3394046390 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc0094d0 con 0x7f12c80cb020 2023-11-28T18:23:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12c6ffd640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c81f3180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12b800c7f0 tx=0x7f12b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12af7fe640 1 -- 172.21.15.7:0/3394046390 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8007820 con 0x7f12c80cb020 2023-11-28T18:23:49.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12af7fe640 1 -- 172.21.15.7:0/3394046390 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b8007980 con 0x7f12c80cb020 2023-11-28T18:23:49.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12af7fe640 1 -- 172.21.15.7:0/3394046390 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8007d30 con 0x7f12c80cb020 2023-11-28T18:23:49.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c81e6d30 con 0x7f12c80cb020 2023-11-28T18:23:49.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c80cf030 con 0x7f12c80cb020 2023-11-28T18:23:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.907+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c81e6f70 con 0x7f12c80cb020 2023-11-28T18:23:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.911+0000 7f12af7fe640 1 -- 172.21.15.7:0/3394046390 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f12b80113f0 con 0x7f12c80cb020 2023-11-28T18:23:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.911+0000 7f12af7fe640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12b4066950 0x7f12b4068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.911+0000 7f12af7fe640 1 -- 172.21.15.7:0/3394046390 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f12b8084690 con 0x7f12c80cb020 2023-11-28T18:23:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.911+0000 7f12c67fc640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12b4066950 0x7f12b4068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.911+0000 7f12c67fc640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12b4066950 0x7f12b4068e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0b2000 tx=0x7f12bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:49.915+0000 7f12af7fe640 1 -- 172.21.15.7:0/3394046390 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f12c81e6f70 con 0x7f12c80cb020 2023-11-28T18:23:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.079+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 --> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12c815dfa0 con 0x7f12b4066950 2023-11-28T18:23:50.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.087+0000 7f12af7fe640 1 -- 172.21.15.7:0/3394046390 <== mgr.4102 v2:172.21.15.7:6832/264069339 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264664 (secure 0 0 0) 0x7f12bc0bb420 con 0x7f12b4066950 2023-11-28T18:23:50.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:50.098 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-28T18:23:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12b4066950 msgr2=0x7f12b4068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12b4066950 0x7f12b4068e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0b2000 tx=0x7f12bc008040 comp rx=0 tx=0).stop 2023-11-28T18:23:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 msgr2=0x7f12c81f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c81f3180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12b800c7f0 tx=0x7f12b800ccb0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 shutdown_connections 2023-11-28T18:23:50.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12b4066950 0x7f12b4068e10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12c8153f00 0x7f12c8159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12c80cb5a0 0x7f12c80ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 --2- 172.21.15.7:0/3394046390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12c80cb020 0x7f12c81f3180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 >> 172.21.15.7:0/3394046390 conn(0x7f12c806c060 msgr2=0x7f12c8151840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:50.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 shutdown_connections 2023-11-28T18:23:50.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.091+0000 7f12cd3b8640 1 -- 172.21.15.7:0/3394046390 wait complete. 2023-11-28T18:23:50.117 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-11-28T18:23:49.470780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600162,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":371,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235784,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738936,"statfs":{"total":773094113280,"available":772852670464,"internally_reserved":0,"allocated":3088384,"data_stored":1765844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1760,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001731"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001273+0000","last_change":"2023-11-28T18:23:08.134633+0000","last_active":"2023-11-28T18:23:11.001273+0000","last_peered":"2023-11-28T18:23:11.001273+0000","last_clean":"2023-11-28T18:23:11.001273+0000","last_became_active":"2023-11-28T18:23:08.134320+0000","last_became_peered":"2023-11-28T18:23:08.134320+0000","last_unstale":"2023-11-28T18:23:11.001273+0000","last_undegraded":"2023-11-28T18:23:11.001273+0000","last_fullsized":"2023-11-28T18:23:11.001273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:54.995718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105857+0000","last_change":"2023-11-28T18:23:08.122460+0000","last_active":"2023-11-28T18:23:09.105857+0000","last_peered":"2023-11-28T18:23:09.105857+0000","last_clean":"2023-11-28T18:23:09.105857+0000","last_became_active":"2023-11-28T18:23:08.122153+0000","last_became_peered":"2023-11-28T18:23:08.122153+0000","last_unstale":"2023-11-28T18:23:09.105857+0000","last_undegraded":"2023-11-28T18:23:09.105857+0000","last_fullsized":"2023-11-28T18:23:09.105857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:17:31.074978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914207+0000","last_change":"2023-11-28T18:23:08.110187+0000","last_active":"2023-11-28T18:23:10.914207+0000","last_peered":"2023-11-28T18:23:10.914207+0000","last_clean":"2023-11-28T18:23:10.914207+0000","last_became_active":"2023-11-28T18:23:08.109915+0000","last_became_peered":"2023-11-28T18:23:08.109915+0000","last_unstale":"2023-11-28T18:23:10.914207+0000","last_undegraded":"2023-11-28T18:23:10.914207+0000","last_fullsized":"2023-11-28T18:23:10.914207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:32.098408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113335+0000","last_change":"2023-11-28T18:23:08.122000+0000","last_active":"2023-11-28T18:23:09.113335+0000","last_peered":"2023-11-28T18:23:09.113335+0000","last_clean":"2023-11-28T18:23:09.113335+0000","last_became_active":"2023-11-28T18:23:08.121652+0000","last_became_peered":"2023-11-28T18:23:08.121652+0000","last_unstale":"2023-11-28T18:23:09.113335+0000","last_undegraded":"2023-11-28T18:23:09.113335+0000","last_fullsized":"2023-11-28T18:23:09.113335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:15.540606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000810+0000","last_change":"2023-11-28T18:23:08.126587+0000","last_active":"2023-11-28T18:23:11.000810+0000","last_peered":"2023-11-28T18:23:11.000810+0000","last_clean":"2023-11-28T18:23:11.000810+0000","last_became_active":"2023-11-28T18:23:08.126313+0000","last_became_peered":"2023-11-28T18:23:08.126313+0000","last_unstale":"2023-11-28T18:23:11.000810+0000","last_undegraded":"2023-11-28T18:23:11.000810+0000","last_fullsized":"2023-11-28T18:23:11.000810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.956197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113040+0000","last_change":"2023-11-28T18:23:08.122878+0000","last_active":"2023-11-28T18:23:09.113040+0000","last_peered":"2023-11-28T18:23:09.113040+0000","last_clean":"2023-11-28T18:23:09.113040+0000","last_became_active":"2023-11-28T18:23:08.122609+0000","last_became_peered":"2023-11-28T18:23:08.122609+0000","last_unstale":"2023-11-28T18:23:09.113040+0000","last_undegraded":"2023-11-28T18:23:09.113040+0000","last_fullsized":"2023-11-28T18:23:09.113040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:00.610074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000604+0000","last_change":"2023-11-28T18:23:08.133485+0000","last_active":"2023-11-28T18:23:11.000604+0000","last_peered":"2023-11-28T18:23:11.000604+0000","last_clean":"2023-11-28T18:23:11.000604+0000","last_became_active":"2023-11-28T18:23:08.133211+0000","last_became_peered":"2023-11-28T18:23:08.133211+0000","last_unstale":"2023-11-28T18:23:11.000604+0000","last_undegraded":"2023-11-28T18:23:11.000604+0000","last_fullsized":"2023-11-28T18:23:11.000604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:04:33.507685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565355+0000","last_change":"2023-11-28T18:23:08.117903+0000","last_active":"2023-11-28T18:23:09.565355+0000","last_peered":"2023-11-28T18:23:09.565355+0000","last_clean":"2023-11-28T18:23:09.565355+0000","last_became_active":"2023-11-28T18:23:08.117482+0000","last_became_peered":"2023-11-28T18:23:08.117482+0000","last_unstale":"2023-11-28T18:23:09.565355+0000","last_undegraded":"2023-11-28T18:23:09.565355+0000","last_fullsized":"2023-11-28T18:23:09.565355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:45.293402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001351+0000","last_change":"2023-11-28T18:23:08.135606+0000","last_active":"2023-11-28T18:23:11.001351+0000","last_peered":"2023-11-28T18:23:11.001351+0000","last_clean":"2023-11-28T18:23:11.001351+0000","last_became_active":"2023-11-28T18:23:08.135328+0000","last_became_peered":"2023-11-28T18:23:08.135328+0000","last_unstale":"2023-11-28T18:23:11.001351+0000","last_undegraded":"2023-11-28T18:23:11.001351+0000","last_fullsized":"2023-11-28T18:23:11.001351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:25.011869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110046+0000","last_change":"2023-11-28T18:23:08.123993+0000","last_active":"2023-11-28T18:23:09.110046+0000","last_peered":"2023-11-28T18:23:09.110046+0000","last_clean":"2023-11-28T18:23:09.110046+0000","last_became_active":"2023-11-28T18:23:08.123583+0000","last_became_peered":"2023-11-28T18:23:08.123583+0000","last_unstale":"2023-11-28T18:23:09.110046+0000","last_undegraded":"2023-11-28T18:23:09.110046+0000","last_fullsized":"2023-11-28T18:23:09.110046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:16:33.089307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:08.120983+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:08.120486+0000","last_became_peered":"2023-11-28T18:23:08.120486+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:46:12.881629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637634+0000","last_change":"2023-11-28T18:23:08.130169+0000","last_active":"2023-11-28T18:23:09.637634+0000","last_peered":"2023-11-28T18:23:09.637634+0000","last_clean":"2023-11-28T18:23:09.637634+0000","last_became_active":"2023-11-28T18:23:08.129771+0000","last_became_peered":"2023-11-28T18:23:08.129771+0000","last_unstale":"2023-11-28T18:23:09.637634+0000","last_undegraded":"2023-11-28T18:23:09.637634+0000","last_fullsized":"2023-11-28T18:23:09.637634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:45.753322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565505+0000","last_change":"2023-11-28T18:23:08.123268+0000","last_active":"2023-11-28T18:23:09.565505+0000","last_peered":"2023-11-28T18:23:09.565505+0000","last_clean":"2023-11-28T18:23:09.565505+0000","last_became_active":"2023-11-28T18:23:08.122978+0000","last_became_peered":"2023-11-28T18:23:08.122978+0000","last_unstale":"2023-11-28T18:23:09.565505+0000","last_undegraded":"2023-11-28T18:23:09.565505+0000","last_fullsized":"2023-11-28T18:23:09.565505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:15:01.320918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913943+0000","last_change":"2023-11-28T18:23:08.118361+0000","last_active":"2023-11-28T18:23:10.913943+0000","last_peered":"2023-11-28T18:23:10.913943+0000","last_clean":"2023-11-28T18:23:10.913943+0000","last_became_active":"2023-11-28T18:23:08.117973+0000","last_became_peered":"2023-11-28T18:23:08.117973+0000","last_unstale":"2023-11-28T18:23:10.913943+0000","last_undegraded":"2023-11-28T18:23:10.913943+0000","last_fullsized":"2023-11-28T18:23:10.913943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:57.485937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105530+0000","last_change":"2023-11-28T18:23:08.139744+0000","last_active":"2023-11-28T18:23:09.105530+0000","last_peered":"2023-11-28T18:23:09.105530+0000","last_clean":"2023-11-28T18:23:09.105530+0000","last_became_active":"2023-11-28T18:23:08.139464+0000","last_became_peered":"2023-11-28T18:23:08.139464+0000","last_unstale":"2023-11-28T18:23:09.105530+0000","last_undegraded":"2023-11-28T18:23:09.105530+0000","last_fullsized":"2023-11-28T18:23:09.105530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:19:34.780979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113036+0000","last_change":"2023-11-28T18:23:08.119103+0000","last_active":"2023-11-28T18:23:09.113036+0000","last_peered":"2023-11-28T18:23:09.113036+0000","last_clean":"2023-11-28T18:23:09.113036+0000","last_became_active":"2023-11-28T18:23:08.116080+0000","last_became_peered":"2023-11-28T18:23:08.116080+0000","last_unstale":"2023-11-28T18:23:09.113036+0000","last_undegraded":"2023-11-28T18:23:09.113036+0000","last_fullsized":"2023-11-28T18:23:09.113036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:44.279413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113688+0000","last_change":"2023-11-28T18:23:08.126817+0000","last_active":"2023-11-28T18:23:09.113688+0000","last_peered":"2023-11-28T18:23:09.113688+0000","last_clean":"2023-11-28T18:23:09.113688+0000","last_became_active":"2023-11-28T18:23:08.126540+0000","last_became_peered":"2023-11-28T18:23:08.126540+0000","last_unstale":"2023-11-28T18:23:09.113688+0000","last_undegraded":"2023-11-28T18:23:09.113688+0000","last_fullsized":"2023-11-28T18:23:09.113688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:08.807847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271940+0000","last_change":"2023-11-28T18:23:08.121775+0000","last_active":"2023-11-28T18:23:10.271940+0000","last_peered":"2023-11-28T18:23:10.271940+0000","last_clean":"2023-11-28T18:23:10.271940+0000","last_became_active":"2023-11-28T18:23:08.121466+0000","last_became_peered":"2023-11-28T18:23:08.121466+0000","last_unstale":"2023-11-28T18:23:10.271940+0000","last_undegraded":"2023-11-28T18:23:10.271940+0000","last_fullsized":"2023-11-28T18:23:10.271940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:32:51.612113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001115+0000","last_change":"2023-11-28T18:23:08.138282+0000","last_active":"2023-11-28T18:23:11.001115+0000","last_peered":"2023-11-28T18:23:11.001115+0000","last_clean":"2023-11-28T18:23:11.001115+0000","last_became_active":"2023-11-28T18:23:08.137171+0000","last_became_peered":"2023-11-28T18:23:08.137171+0000","last_unstale":"2023-11-28T18:23:11.001115+0000","last_undegraded":"2023-11-28T18:23:11.001115+0000","last_fullsized":"2023-11-28T18:23:11.001115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:39:07.113697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110124+0000","last_change":"2023-11-28T18:23:08.133521+0000","last_active":"2023-11-28T18:23:09.110124+0000","last_peered":"2023-11-28T18:23:09.110124+0000","last_clean":"2023-11-28T18:23:09.110124+0000","last_became_active":"2023-11-28T18:23:08.133248+0000","last_became_peered":"2023-11-28T18:23:08.133248+0000","last_unstale":"2023-11-28T18:23:09.110124+0000","last_undegraded":"2023-11-28T18:23:09.110124+0000","last_fullsized":"2023-11-28T18:23:09.110124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:17:34.585808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914195+0000","last_change":"2023-11-28T18:23:08.129150+0000","last_active":"2023-11-28T18:23:10.914195+0000","last_peered":"2023-11-28T18:23:10.914195+0000","last_clean":"2023-11-28T18:23:10.914195+0000","last_became_active":"2023-11-28T18:23:08.128622+0000","last_became_peered":"2023-11-28T18:23:08.128622+0000","last_unstale":"2023-11-28T18:23:10.914195+0000","last_undegraded":"2023-11-28T18:23:10.914195+0000","last_fullsized":"2023-11-28T18:23:10.914195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:33.197161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113274+0000","last_change":"2023-11-28T18:23:08.131333+0000","last_active":"2023-11-28T18:23:09.113274+0000","last_peered":"2023-11-28T18:23:09.113274+0000","last_clean":"2023-11-28T18:23:09.113274+0000","last_became_active":"2023-11-28T18:23:08.130917+0000","last_became_peered":"2023-11-28T18:23:08.130917+0000","last_unstale":"2023-11-28T18:23:09.113274+0000","last_undegraded":"2023-11-28T18:23:09.113274+0000","last_fullsized":"2023-11-28T18:23:09.113274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:04:48.114417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565507+0000","last_change":"2023-11-28T18:23:08.132355+0000","last_active":"2023-11-28T18:23:09.565507+0000","last_peered":"2023-11-28T18:23:09.565507+0000","last_clean":"2023-11-28T18:23:09.565507+0000","last_became_active":"2023-11-28T18:23:08.132016+0000","last_became_peered":"2023-11-28T18:23:08.132016+0000","last_unstale":"2023-11-28T18:23:09.565507+0000","last_undegraded":"2023-11-28T18:23:09.565507+0000","last_fullsized":"2023-11-28T18:23:09.565507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:11.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113066+0000","last_change":"2023-11-28T18:23:08.119935+0000","last_active":"2023-11-28T18:23:09.113066+0000","last_peered":"2023-11-28T18:23:09.113066+0000","last_clean":"2023-11-28T18:23:09.113066+0000","last_became_active":"2023-11-28T18:23:08.118372+0000","last_became_peered":"2023-11-28T18:23:08.118372+0000","last_unstale":"2023-11-28T18:23:09.113066+0000","last_undegraded":"2023-11-28T18:23:09.113066+0000","last_fullsized":"2023-11-28T18:23:09.113066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:52.339024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.132272+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.131985+0000","last_became_peered":"2023-11-28T18:23:08.131985+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:44.594691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110084+0000","last_change":"2023-11-28T18:23:08.127816+0000","last_active":"2023-11-28T18:23:09.110084+0000","last_peered":"2023-11-28T18:23:09.110084+0000","last_clean":"2023-11-28T18:23:09.110084+0000","last_became_active":"2023-11-28T18:23:08.127380+0000","last_became_peered":"2023-11-28T18:23:08.127380+0000","last_unstale":"2023-11-28T18:23:09.110084+0000","last_undegraded":"2023-11-28T18:23:09.110084+0000","last_fullsized":"2023-11-28T18:23:09.110084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:31.242361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105706+0000","last_change":"2023-11-28T18:23:08.128188+0000","last_active":"2023-11-28T18:23:09.105706+0000","last_peered":"2023-11-28T18:23:09.105706+0000","last_clean":"2023-11-28T18:23:09.105706+0000","last_became_active":"2023-11-28T18:23:08.127860+0000","last_became_peered":"2023-11-28T18:23:08.127860+0000","last_unstale":"2023-11-28T18:23:09.105706+0000","last_undegraded":"2023-11-28T18:23:09.105706+0000","last_fullsized":"2023-11-28T18:23:09.105706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:12.095768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565712+0000","last_change":"2023-11-28T18:23:08.132381+0000","last_active":"2023-11-28T18:23:09.565712+0000","last_peered":"2023-11-28T18:23:09.565712+0000","last_clean":"2023-11-28T18:23:09.565712+0000","last_became_active":"2023-11-28T18:23:08.132072+0000","last_became_peered":"2023-11-28T18:23:08.132072+0000","last_unstale":"2023-11-28T18:23:09.565712+0000","last_undegraded":"2023-11-28T18:23:09.565712+0000","last_fullsized":"2023-11-28T18:23:09.565712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:16:38.666702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914263+0000","last_change":"2023-11-28T18:23:08.115379+0000","last_active":"2023-11-28T18:23:10.914263+0000","last_peered":"2023-11-28T18:23:10.914263+0000","last_clean":"2023-11-28T18:23:10.914263+0000","last_became_active":"2023-11-28T18:23:08.115236+0000","last_became_peered":"2023-11-28T18:23:08.115236+0000","last_unstale":"2023-11-28T18:23:10.914263+0000","last_undegraded":"2023-11-28T18:23:10.914263+0000","last_fullsized":"2023-11-28T18:23:10.914263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:26:52.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113293+0000","last_change":"2023-11-28T18:23:08.121402+0000","last_active":"2023-11-28T18:23:09.113293+0000","last_peered":"2023-11-28T18:23:09.113293+0000","last_clean":"2023-11-28T18:23:09.113293+0000","last_became_active":"2023-11-28T18:23:08.121129+0000","last_became_peered":"2023-11-28T18:23:08.121129+0000","last_unstale":"2023-11-28T18:23:09.113293+0000","last_undegraded":"2023-11-28T18:23:09.113293+0000","last_fullsized":"2023-11-28T18:23:09.113293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:02.927891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271829+0000","last_change":"2023-11-28T18:23:08.121594+0000","last_active":"2023-11-28T18:23:10.271829+0000","last_peered":"2023-11-28T18:23:10.271829+0000","last_clean":"2023-11-28T18:23:10.271829+0000","last_became_active":"2023-11-28T18:23:08.119009+0000","last_became_peered":"2023-11-28T18:23:08.119009+0000","last_unstale":"2023-11-28T18:23:10.271829+0000","last_undegraded":"2023-11-28T18:23:10.271829+0000","last_fullsized":"2023-11-28T18:23:10.271829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:38.801632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109676+0000","last_change":"2023-11-28T18:23:08.115403+0000","last_active":"2023-11-28T18:23:09.109676+0000","last_peered":"2023-11-28T18:23:09.109676+0000","last_clean":"2023-11-28T18:23:09.109676+0000","last_became_active":"2023-11-28T18:23:08.114955+0000","last_became_peered":"2023-11-28T18:23:08.114955+0000","last_unstale":"2023-11-28T18:23:09.109676+0000","last_undegraded":"2023-11-28T18:23:09.109676+0000","last_fullsized":"2023-11-28T18:23:09.109676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:41.242830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637404+0000","last_change":"2023-11-28T18:23:08.129745+0000","last_active":"2023-11-28T18:23:09.637404+0000","last_peered":"2023-11-28T18:23:09.637404+0000","last_clean":"2023-11-28T18:23:09.637404+0000","last_became_active":"2023-11-28T18:23:08.129388+0000","last_became_peered":"2023-11-28T18:23:08.129388+0000","last_unstale":"2023-11-28T18:23:09.637404+0000","last_undegraded":"2023-11-28T18:23:09.637404+0000","last_fullsized":"2023-11-28T18:23:09.637404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:50.570821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.743781+0000","last_change":"2023-11-28T18:23:07.664773+0000","last_active":"2023-11-28T18:23:18.743781+0000","last_peered":"2023-11-28T18:23:18.743781+0000","last_clean":"2023-11-28T18:23:18.743781+0000","last_became_active":"2023-11-28T18:23:07.664478+0000","last_became_peered":"2023-11-28T18:23:07.664478+0000","last_unstale":"2023-11-28T18:23:18.743781+0000","last_undegraded":"2023-11-28T18:23:18.743781+0000","last_fullsized":"2023-11-28T18:23:18.743781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:28.322454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914337+0000","last_change":"2023-11-28T18:23:08.129441+0000","last_active":"2023-11-28T18:23:10.914337+0000","last_peered":"2023-11-28T18:23:10.914337+0000","last_clean":"2023-11-28T18:23:10.914337+0000","last_became_active":"2023-11-28T18:23:08.129143+0000","last_became_peered":"2023-11-28T18:23:08.129143+0000","last_unstale":"2023-11-28T18:23:10.914337+0000","last_undegraded":"2023-11-28T18:23:10.914337+0000","last_fullsized":"2023-11-28T18:23:10.914337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:36.540819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.173128+0000","last_change":"2023-11-28T18:23:07.668192+0000","last_active":"2023-11-28T18:23:30.173128+0000","last_peered":"2023-11-28T18:23:30.173128+0000","last_clean":"2023-11-28T18:23:30.173128+0000","last_became_active":"2023-11-28T18:23:07.667780+0000","last_became_peered":"2023-11-28T18:23:07.667780+0000","last_unstale":"2023-11-28T18:23:30.173128+0000","last_undegraded":"2023-11-28T18:23:30.173128+0000","last_fullsized":"2023-11-28T18:23:30.173128+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:24.148981+0000","last_change":"2023-11-28T18:23:07.665608+0000","last_active":"2023-11-28T18:23:24.148981+0000","last_peered":"2023-11-28T18:23:24.148981+0000","last_clean":"2023-11-28T18:23:24.148981+0000","last_became_active":"2023-11-28T18:23:07.665361+0000","last_became_peered":"2023-11-28T18:23:07.665361+0000","last_unstale":"2023-11-28T18:23:24.148981+0000","last_undegraded":"2023-11-28T18:23:24.148981+0000","last_fullsized":"2023-11-28T18:23:24.148981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:17.279597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.118091+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.117524+0000","last_became_peered":"2023-11-28T18:23:08.117524+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:04.054226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171679+0000","last_change":"2023-11-28T18:23:07.667847+0000","last_active":"2023-11-28T18:23:30.171679+0000","last_peered":"2023-11-28T18:23:30.171679+0000","last_clean":"2023-11-28T18:23:30.171679+0000","last_became_active":"2023-11-28T18:23:07.667466+0000","last_became_peered":"2023-11-28T18:23:07.667466+0000","last_unstale":"2023-11-28T18:23:30.171679+0000","last_undegraded":"2023-11-28T18:23:30.171679+0000","last_fullsized":"2023-11-28T18:23:30.171679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:06.807146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.127497+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.127216+0000","last_became_peered":"2023-11-28T18:23:08.127216+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:03:14.999484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172207+0000","last_change":"2023-11-28T18:23:07.681698+0000","last_active":"2023-11-28T18:23:30.172207+0000","last_peered":"2023-11-28T18:23:30.172207+0000","last_clean":"2023-11-28T18:23:30.172207+0000","last_became_active":"2023-11-28T18:23:07.681365+0000","last_became_peered":"2023-11-28T18:23:07.681365+0000","last_unstale":"2023-11-28T18:23:30.172207+0000","last_undegraded":"2023-11-28T18:23:30.172207+0000","last_fullsized":"2023-11-28T18:23:30.172207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:21.985029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113361+0000","last_change":"2023-11-28T18:23:08.132797+0000","last_active":"2023-11-28T18:23:09.113361+0000","last_peered":"2023-11-28T18:23:09.113361+0000","last_clean":"2023-11-28T18:23:09.113361+0000","last_became_active":"2023-11-28T18:23:08.132525+0000","last_became_peered":"2023-11-28T18:23:08.132525+0000","last_unstale":"2023-11-28T18:23:09.113361+0000","last_undegraded":"2023-11-28T18:23:09.113361+0000","last_fullsized":"2023-11-28T18:23:09.113361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:12:10.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.248118+0000","last_change":"2023-11-28T18:23:07.672337+0000","last_active":"2023-11-28T18:23:35.248118+0000","last_peered":"2023-11-28T18:23:35.248118+0000","last_clean":"2023-11-28T18:23:35.248118+0000","last_became_active":"2023-11-28T18:23:07.671922+0000","last_became_peered":"2023-11-28T18:23:07.671922+0000","last_unstale":"2023-11-28T18:23:35.248118+0000","last_undegraded":"2023-11-28T18:23:35.248118+0000","last_fullsized":"2023-11-28T18:23:35.248118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:22.936770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914223+0000","last_change":"2023-11-28T18:23:08.127919+0000","last_active":"2023-11-28T18:23:10.914223+0000","last_peered":"2023-11-28T18:23:10.914223+0000","last_clean":"2023-11-28T18:23:10.914223+0000","last_became_active":"2023-11-28T18:23:08.127602+0000","last_became_peered":"2023-11-28T18:23:08.127602+0000","last_unstale":"2023-11-28T18:23:10.914223+0000","last_undegraded":"2023-11-28T18:23:10.914223+0000","last_fullsized":"2023-11-28T18:23:10.914223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:32:45.967018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.734841+0000","last_change":"2023-11-28T18:23:07.679219+0000","last_active":"2023-11-28T18:23:18.734841+0000","last_peered":"2023-11-28T18:23:18.734841+0000","last_clean":"2023-11-28T18:23:18.734841+0000","last_became_active":"2023-11-28T18:23:07.678920+0000","last_became_peered":"2023-11-28T18:23:07.678920+0000","last_unstale":"2023-11-28T18:23:18.734841+0000","last_undegraded":"2023-11-28T18:23:18.734841+0000","last_fullsized":"2023-11-28T18:23:18.734841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:02.332690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637820+0000","last_change":"2023-11-28T18:23:08.132429+0000","last_active":"2023-11-28T18:23:09.637820+0000","last_peered":"2023-11-28T18:23:09.637820+0000","last_clean":"2023-11-28T18:23:09.637820+0000","last_became_active":"2023-11-28T18:23:08.132106+0000","last_became_peered":"2023-11-28T18:23:08.132106+0000","last_unstale":"2023-11-28T18:23:09.637820+0000","last_undegraded":"2023-11-28T18:23:09.637820+0000","last_fullsized":"2023-11-28T18:23:09.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:52.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'12","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:43.822203+0000","last_change":"2023-11-28T18:23:07.667479+0000","last_active":"2023-11-28T18:23:43.822203+0000","last_peered":"2023-11-28T18:23:43.822203+0000","last_clean":"2023-11-28T18:23:43.822203+0000","last_became_active":"2023-11-28T18:23:07.665310+0000","last_became_peered":"2023-11-28T18:23:07.665310+0000","last_unstale":"2023-11-28T18:23:43.822203+0000","last_undegraded":"2023-11-28T18:23:43.822203+0000","last_fullsized":"2023-11-28T18:23:43.822203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:03.174425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2052,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000928+0000","last_change":"2023-11-28T18:23:08.136812+0000","last_active":"2023-11-28T18:23:11.000928+0000","last_peered":"2023-11-28T18:23:11.000928+0000","last_clean":"2023-11-28T18:23:11.000928+0000","last_became_active":"2023-11-28T18:23:08.136525+0000","last_became_peered":"2023-11-28T18:23:08.136525+0000","last_unstale":"2023-11-28T18:23:11.000928+0000","last_undegraded":"2023-11-28T18:23:11.000928+0000","last_fullsized":"2023-11-28T18:23:11.000928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:07.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.249898+0000","last_change":"2023-11-28T18:23:07.660519+0000","last_active":"2023-11-28T18:23:35.249898+0000","last_peered":"2023-11-28T18:23:35.249898+0000","last_clean":"2023-11-28T18:23:35.249898+0000","last_became_active":"2023-11-28T18:23:07.660205+0000","last_became_peered":"2023-11-28T18:23:07.660205+0000","last_unstale":"2023-11-28T18:23:35.249898+0000","last_undegraded":"2023-11-28T18:23:35.249898+0000","last_fullsized":"2023-11-28T18:23:35.249898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:04:54.125737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914264+0000","last_change":"2023-11-28T18:23:08.127921+0000","last_active":"2023-11-28T18:23:10.914264+0000","last_peered":"2023-11-28T18:23:10.914264+0000","last_clean":"2023-11-28T18:23:10.914264+0000","last_became_active":"2023-11-28T18:23:08.127678+0000","last_became_peered":"2023-11-28T18:23:08.127678+0000","last_unstale":"2023-11-28T18:23:10.914264+0000","last_undegraded":"2023-11-28T18:23:10.914264+0000","last_fullsized":"2023-11-28T18:23:10.914264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:38:47.175911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171696+0000","last_change":"2023-11-28T18:23:07.671589+0000","last_active":"2023-11-28T18:23:30.171696+0000","last_peered":"2023-11-28T18:23:30.171696+0000","last_clean":"2023-11-28T18:23:30.171696+0000","last_became_active":"2023-11-28T18:23:07.671291+0000","last_became_peered":"2023-11-28T18:23:07.671291+0000","last_unstale":"2023-11-28T18:23:30.171696+0000","last_undegraded":"2023-11-28T18:23:30.171696+0000","last_fullsized":"2023-11-28T18:23:30.171696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:23.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000866+0000","last_change":"2023-11-28T18:23:08.141316+0000","last_active":"2023-11-28T18:23:11.000866+0000","last_peered":"2023-11-28T18:23:11.000866+0000","last_clean":"2023-11-28T18:23:11.000866+0000","last_became_active":"2023-11-28T18:23:08.141011+0000","last_became_peered":"2023-11-28T18:23:08.141011+0000","last_unstale":"2023-11-28T18:23:11.000866+0000","last_undegraded":"2023-11-28T18:23:11.000866+0000","last_fullsized":"2023-11-28T18:23:11.000866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:13:25.849340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172815+0000","last_change":"2023-11-28T18:23:07.671192+0000","last_active":"2023-11-28T18:23:30.172815+0000","last_peered":"2023-11-28T18:23:30.172815+0000","last_clean":"2023-11-28T18:23:30.172815+0000","last_became_active":"2023-11-28T18:23:07.670812+0000","last_became_peered":"2023-11-28T18:23:07.670812+0000","last_unstale":"2023-11-28T18:23:30.172815+0000","last_undegraded":"2023-11-28T18:23:30.172815+0000","last_fullsized":"2023-11-28T18:23:30.172815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:28.806061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109506+0000","last_change":"2023-11-28T18:23:08.115229+0000","last_active":"2023-11-28T18:23:09.109506+0000","last_peered":"2023-11-28T18:23:09.109506+0000","last_clean":"2023-11-28T18:23:09.109506+0000","last_became_active":"2023-11-28T18:23:08.114541+0000","last_became_peered":"2023-11-28T18:23:08.114541+0000","last_unstale":"2023-11-28T18:23:09.109506+0000","last_undegraded":"2023-11-28T18:23:09.109506+0000","last_fullsized":"2023-11-28T18:23:09.109506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:38.816731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":29,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171599+0000","last_change":"2023-11-28T18:23:07.673554+0000","last_active":"2023-11-28T18:23:30.171599+0000","last_peered":"2023-11-28T18:23:30.171599+0000","last_clean":"2023-11-28T18:23:30.171599+0000","last_became_active":"2023-11-28T18:23:07.672923+0000","last_became_peered":"2023-11-28T18:23:07.672923+0000","last_unstale":"2023-11-28T18:23:30.171599+0000","last_undegraded":"2023-11-28T18:23:30.171599+0000","last_fullsized":"2023-11-28T18:23:30.171599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:49.805572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000699+0000","last_change":"2023-11-28T18:23:08.141169+0000","last_active":"2023-11-28T18:23:11.000699+0000","last_peered":"2023-11-28T18:23:11.000699+0000","last_clean":"2023-11-28T18:23:11.000699+0000","last_became_active":"2023-11-28T18:23:08.140801+0000","last_became_peered":"2023-11-28T18:23:08.140801+0000","last_unstale":"2023-11-28T18:23:11.000699+0000","last_undegraded":"2023-11-28T18:23:11.000699+0000","last_fullsized":"2023-11-28T18:23:11.000699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:16.669717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.667707+0000","last_change":"2023-11-28T18:23:07.680094+0000","last_active":"2023-11-28T18:23:13.667707+0000","last_peered":"2023-11-28T18:23:13.667707+0000","last_clean":"2023-11-28T18:23:13.667707+0000","last_became_active":"2023-11-28T18:23:07.679801+0000","last_became_peered":"2023-11-28T18:23:07.679801+0000","last_unstale":"2023-11-28T18:23:13.667707+0000","last_undegraded":"2023-11-28T18:23:13.667707+0000","last_fullsized":"2023-11-28T18:23:13.667707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:02.916155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001208+0000","last_change":"2023-11-28T18:23:08.137825+0000","last_active":"2023-11-28T18:23:11.001208+0000","last_peered":"2023-11-28T18:23:11.001208+0000","last_clean":"2023-11-28T18:23:11.001208+0000","last_became_active":"2023-11-28T18:23:08.137499+0000","last_became_peered":"2023-11-28T18:23:08.137499+0000","last_unstale":"2023-11-28T18:23:11.001208+0000","last_undegraded":"2023-11-28T18:23:11.001208+0000","last_fullsized":"2023-11-28T18:23:11.001208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:42.807830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.244409+0000","last_change":"2023-11-28T18:23:07.670359+0000","last_active":"2023-11-28T18:23:35.244409+0000","last_peered":"2023-11-28T18:23:35.244409+0000","last_clean":"2023-11-28T18:23:35.244409+0000","last_became_active":"2023-11-28T18:23:07.669687+0000","last_became_peered":"2023-11-28T18:23:07.669687+0000","last_unstale":"2023-11-28T18:23:35.244409+0000","last_undegraded":"2023-11-28T18:23:35.244409+0000","last_fullsized":"2023-11-28T18:23:35.244409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:45.450623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001353+0000","last_change":"2023-11-28T18:23:08.141300+0000","last_active":"2023-11-28T18:23:11.001353+0000","last_peered":"2023-11-28T18:23:11.001353+0000","last_clean":"2023-11-28T18:23:11.001353+0000","last_became_active":"2023-11-28T18:23:08.141037+0000","last_became_peered":"2023-11-28T18:23:08.141037+0000","last_unstale":"2023-11-28T18:23:11.001353+0000","last_undegraded":"2023-11-28T18:23:11.001353+0000","last_fullsized":"2023-11-28T18:23:11.001353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:11.093903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172901+0000","last_change":"2023-11-28T18:23:07.673086+0000","last_active":"2023-11-28T18:23:30.172901+0000","last_peered":"2023-11-28T18:23:30.172901+0000","last_clean":"2023-11-28T18:23:30.172901+0000","last_became_active":"2023-11-28T18:23:07.672773+0000","last_became_peered":"2023-11-28T18:23:07.672773+0000","last_unstale":"2023-11-28T18:23:30.172901+0000","last_undegraded":"2023-11-28T18:23:30.172901+0000","last_fullsized":"2023-11-28T18:23:30.172901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:34.962428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113586+0000","last_change":"2023-11-28T18:23:08.121987+0000","last_active":"2023-11-28T18:23:09.113586+0000","last_peered":"2023-11-28T18:23:09.113586+0000","last_clean":"2023-11-28T18:23:09.113586+0000","last_became_active":"2023-11-28T18:23:08.121279+0000","last_became_peered":"2023-11-28T18:23:08.121279+0000","last_unstale":"2023-11-28T18:23:09.113586+0000","last_undegraded":"2023-11-28T18:23:09.113586+0000","last_fullsized":"2023-11-28T18:23:09.113586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:47.973973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.174488+0000","last_change":"2023-11-28T18:23:07.681312+0000","last_active":"2023-11-28T18:23:30.174488+0000","last_peered":"2023-11-28T18:23:30.174488+0000","last_clean":"2023-11-28T18:23:30.174488+0000","last_became_active":"2023-11-28T18:23:07.680806+0000","last_became_peered":"2023-11-28T18:23:07.680806+0000","last_unstale":"2023-11-28T18:23:30.174488+0000","last_undegraded":"2023-11-28T18:23:30.174488+0000","last_fullsized":"2023-11-28T18:23:30.174488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:58:41.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001014+0000","last_change":"2023-11-28T18:23:08.137874+0000","last_active":"2023-11-28T18:23:11.001014+0000","last_peered":"2023-11-28T18:23:11.001014+0000","last_clean":"2023-11-28T18:23:11.001014+0000","last_became_active":"2023-11-28T18:23:08.137104+0000","last_became_peered":"2023-11-28T18:23:08.137104+0000","last_unstale":"2023-11-28T18:23:11.001014+0000","last_undegraded":"2023-11-28T18:23:11.001014+0000","last_fullsized":"2023-11-28T18:23:11.001014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:39:11.857836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117053+0000","last_change":"2023-11-28T18:22:56.774057+0000","last_active":"2023-11-28T18:23:08.117053+0000","last_peered":"2023-11-28T18:23:08.117053+0000","last_clean":"2023-11-28T18:23:08.117053+0000","last_became_active":"2023-11-28T18:22:56.772763+0000","last_became_peered":"2023-11-28T18:22:56.772763+0000","last_unstale":"2023-11-28T18:23:08.117053+0000","last_undegraded":"2023-11-28T18:23:08.117053+0000","last_fullsized":"2023-11-28T18:23:08.117053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_clean_scrub_stamp":"2023-11-28T18:22:55.591288+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:05.470408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172368+0000","last_change":"2023-11-28T18:23:07.674072+0000","last_active":"2023-11-28T18:23:30.172368+0000","last_peered":"2023-11-28T18:23:30.172368+0000","last_clean":"2023-11-28T18:23:30.172368+0000","last_became_active":"2023-11-28T18:23:07.673220+0000","last_became_peered":"2023-11-28T18:23:07.673220+0000","last_unstale":"2023-11-28T18:23:30.172368+0000","last_undegraded":"2023-11-28T18:23:30.172368+0000","last_fullsized":"2023-11-28T18:23:30.172368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:58.524112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565598+0000","last_change":"2023-11-28T18:23:08.130989+0000","last_active":"2023-11-28T18:23:09.565598+0000","last_peered":"2023-11-28T18:23:09.565598+0000","last_clean":"2023-11-28T18:23:09.565598+0000","last_became_active":"2023-11-28T18:23:08.130646+0000","last_became_peered":"2023-11-28T18:23:08.130646+0000","last_unstale":"2023-11-28T18:23:09.565598+0000","last_undegraded":"2023-11-28T18:23:09.565598+0000","last_fullsized":"2023-11-28T18:23:09.565598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:50.371045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:24.150194+0000","last_change":"2023-11-28T18:23:07.686046+0000","last_active":"2023-11-28T18:23:24.150194+0000","last_peered":"2023-11-28T18:23:24.150194+0000","last_clean":"2023-11-28T18:23:24.150194+0000","last_became_active":"2023-11-28T18:23:07.685739+0000","last_became_peered":"2023-11-28T18:23:07.685739+0000","last_unstale":"2023-11-28T18:23:24.150194+0000","last_undegraded":"2023-11-28T18:23:24.150194+0000","last_fullsized":"2023-11-28T18:23:24.150194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:06.485631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913951+0000","last_change":"2023-11-28T18:23:08.130355+0000","last_active":"2023-11-28T18:23:10.913951+0000","last_peered":"2023-11-28T18:23:10.913951+0000","last_clean":"2023-11-28T18:23:10.913951+0000","last_became_active":"2023-11-28T18:23:08.129946+0000","last_became_peered":"2023-11-28T18:23:08.129946+0000","last_unstale":"2023-11-28T18:23:10.913951+0000","last_undegraded":"2023-11-28T18:23:10.913951+0000","last_fullsized":"2023-11-28T18:23:10.913951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:58.140191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.245973+0000","last_change":"2023-11-28T18:23:07.664827+0000","last_active":"2023-11-28T18:23:35.245973+0000","last_peered":"2023-11-28T18:23:35.245973+0000","last_clean":"2023-11-28T18:23:35.245973+0000","last_became_active":"2023-11-28T18:23:07.664503+0000","last_became_peered":"2023-11-28T18:23:07.664503+0000","last_unstale":"2023-11-28T18:23:35.245973+0000","last_undegraded":"2023-11-28T18:23:35.245973+0000","last_fullsized":"2023-11-28T18:23:35.245973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.238694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.123498+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.123197+0000","last_became_peered":"2023-11-28T18:23:08.123197+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:08:42.817848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250563+0000","last_change":"2023-11-28T18:23:07.673345+0000","last_active":"2023-11-28T18:23:35.250563+0000","last_peered":"2023-11-28T18:23:35.250563+0000","last_clean":"2023-11-28T18:23:35.250563+0000","last_became_active":"2023-11-28T18:23:07.672982+0000","last_became_peered":"2023-11-28T18:23:07.672982+0000","last_unstale":"2023-11-28T18:23:35.250563+0000","last_undegraded":"2023-11-28T18:23:35.250563+0000","last_fullsized":"2023-11-28T18:23:35.250563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:26.594232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637566+0000","last_change":"2023-11-28T18:23:08.129091+0000","last_active":"2023-11-28T18:23:09.637566+0000","last_peered":"2023-11-28T18:23:09.637566+0000","last_clean":"2023-11-28T18:23:09.637566+0000","last_became_active":"2023-11-28T18:23:08.128589+0000","last_became_peered":"2023-11-28T18:23:08.128589+0000","last_unstale":"2023-11-28T18:23:09.637566+0000","last_undegraded":"2023-11-28T18:23:09.637566+0000","last_fullsized":"2023-11-28T18:23:09.637566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:28.975412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.493169+0000","last_change":"2023-11-28T18:23:07.671874+0000","last_active":"2023-11-28T18:23:09.493169+0000","last_peered":"2023-11-28T18:23:09.493169+0000","last_clean":"2023-11-28T18:23:09.493169+0000","last_became_active":"2023-11-28T18:23:07.671500+0000","last_became_peered":"2023-11-28T18:23:07.671500+0000","last_unstale":"2023-11-28T18:23:09.493169+0000","last_undegraded":"2023-11-28T18:23:09.493169+0000","last_fullsized":"2023-11-28T18:23:09.493169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:42.922770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113612+0000","last_change":"2023-11-28T18:23:08.127419+0000","last_active":"2023-11-28T18:23:09.113612+0000","last_peered":"2023-11-28T18:23:09.113612+0000","last_clean":"2023-11-28T18:23:09.113612+0000","last_became_active":"2023-11-28T18:23:08.127157+0000","last_became_peered":"2023-11-28T18:23:08.127157+0000","last_unstale":"2023-11-28T18:23:09.113612+0000","last_undegraded":"2023-11-28T18:23:09.113612+0000","last_fullsized":"2023-11-28T18:23:09.113612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:08.473882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.173374+0000","last_change":"2023-11-28T18:23:07.684725+0000","last_active":"2023-11-28T18:23:30.173374+0000","last_peered":"2023-11-28T18:23:30.173374+0000","last_clean":"2023-11-28T18:23:30.173374+0000","last_became_active":"2023-11-28T18:23:07.684363+0000","last_became_peered":"2023-11-28T18:23:07.684363+0000","last_unstale":"2023-11-28T18:23:30.173374+0000","last_undegraded":"2023-11-28T18:23:30.173374+0000","last_fullsized":"2023-11-28T18:23:30.173374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:16.380789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105690+0000","last_change":"2023-11-28T18:23:08.120741+0000","last_active":"2023-11-28T18:23:09.105690+0000","last_peered":"2023-11-28T18:23:09.105690+0000","last_clean":"2023-11-28T18:23:09.105690+0000","last_became_active":"2023-11-28T18:23:08.120421+0000","last_became_peered":"2023-11-28T18:23:08.120421+0000","last_unstale":"2023-11-28T18:23:09.105690+0000","last_undegraded":"2023-11-28T18:23:09.105690+0000","last_fullsized":"2023-11-28T18:23:09.105690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:53:27.131631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.175382+0000","last_change":"2023-11-28T18:23:07.672505+0000","last_active":"2023-11-28T18:23:30.175382+0000","last_peered":"2023-11-28T18:23:30.175382+0000","last_clean":"2023-11-28T18:23:30.175382+0000","last_became_active":"2023-11-28T18:23:07.672204+0000","last_became_peered":"2023-11-28T18:23:07.672204+0000","last_unstale":"2023-11-28T18:23:30.175382+0000","last_undegraded":"2023-11-28T18:23:30.175382+0000","last_fullsized":"2023-11-28T18:23:30.175382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:01.341643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565869+0000","last_change":"2023-11-28T18:23:08.126198+0000","last_active":"2023-11-28T18:23:09.565869+0000","last_peered":"2023-11-28T18:23:09.565869+0000","last_clean":"2023-11-28T18:23:09.565869+0000","last_became_active":"2023-11-28T18:23:08.125935+0000","last_became_peered":"2023-11-28T18:23:08.125935+0000","last_unstale":"2023-11-28T18:23:09.565869+0000","last_undegraded":"2023-11-28T18:23:09.565869+0000","last_fullsized":"2023-11-28T18:23:09.565869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.473922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.252470+0000","last_change":"2023-11-28T18:23:07.672170+0000","last_active":"2023-11-28T18:23:35.252470+0000","last_peered":"2023-11-28T18:23:35.252470+0000","last_clean":"2023-11-28T18:23:35.252470+0000","last_became_active":"2023-11-28T18:23:07.671683+0000","last_became_peered":"2023-11-28T18:23:07.671683+0000","last_unstale":"2023-11-28T18:23:35.252470+0000","last_undegraded":"2023-11-28T18:23:35.252470+0000","last_fullsized":"2023-11-28T18:23:35.252470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:41.860341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914424+0000","last_change":"2023-11-28T18:23:08.119844+0000","last_active":"2023-11-28T18:23:10.914424+0000","last_peered":"2023-11-28T18:23:10.914424+0000","last_clean":"2023-11-28T18:23:10.914424+0000","last_became_active":"2023-11-28T18:23:08.119525+0000","last_became_peered":"2023-11-28T18:23:08.119525+0000","last_unstale":"2023-11-28T18:23:10.914424+0000","last_undegraded":"2023-11-28T18:23:10.914424+0000","last_fullsized":"2023-11-28T18:23:10.914424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:34:00.230528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.727126+0000","last_change":"2023-11-28T18:23:07.685651+0000","last_active":"2023-11-28T18:23:18.727126+0000","last_peered":"2023-11-28T18:23:18.727126+0000","last_clean":"2023-11-28T18:23:18.727126+0000","last_became_active":"2023-11-28T18:23:07.683869+0000","last_became_peered":"2023-11-28T18:23:07.683869+0000","last_unstale":"2023-11-28T18:23:18.727126+0000","last_undegraded":"2023-11-28T18:23:18.727126+0000","last_fullsized":"2023-11-28T18:23:18.727126+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:35.383852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637574+0000","last_change":"2023-11-28T18:23:08.130016+0000","last_active":"2023-11-28T18:23:09.637574+0000","last_peered":"2023-11-28T18:23:09.637574+0000","last_clean":"2023-11-28T18:23:09.637574+0000","last_became_active":"2023-11-28T18:23:08.129586+0000","last_became_peered":"2023-11-28T18:23:08.129586+0000","last_unstale":"2023-11-28T18:23:09.637574+0000","last_undegraded":"2023-11-28T18:23:09.637574+0000","last_fullsized":"2023-11-28T18:23:09.637574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:00.449941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.245267+0000","last_change":"2023-11-28T18:23:07.671773+0000","last_active":"2023-11-28T18:23:35.245267+0000","last_peered":"2023-11-28T18:23:35.245267+0000","last_clean":"2023-11-28T18:23:35.245267+0000","last_became_active":"2023-11-28T18:23:07.671465+0000","last_became_peered":"2023-11-28T18:23:07.671465+0000","last_unstale":"2023-11-28T18:23:35.245267+0000","last_undegraded":"2023-11-28T18:23:35.245267+0000","last_fullsized":"2023-11-28T18:23:35.245267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:44.862780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565865+0000","last_change":"2023-11-28T18:23:08.118146+0000","last_active":"2023-11-28T18:23:09.565865+0000","last_peered":"2023-11-28T18:23:09.565865+0000","last_clean":"2023-11-28T18:23:09.565865+0000","last_became_active":"2023-11-28T18:23:08.117848+0000","last_became_peered":"2023-11-28T18:23:08.117848+0000","last_unstale":"2023-11-28T18:23:09.565865+0000","last_undegraded":"2023-11-28T18:23:09.565865+0000","last_fullsized":"2023-11-28T18:23:09.565865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:33:20.745184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.169912+0000","last_change":"2023-11-28T18:23:07.676709+0000","last_active":"2023-11-28T18:23:30.169912+0000","last_peered":"2023-11-28T18:23:30.169912+0000","last_clean":"2023-11-28T18:23:30.169912+0000","last_became_active":"2023-11-28T18:23:07.676414+0000","last_became_peered":"2023-11-28T18:23:07.676414+0000","last_unstale":"2023-11-28T18:23:30.169912+0000","last_undegraded":"2023-11-28T18:23:30.169912+0000","last_fullsized":"2023-11-28T18:23:30.169912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.282005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565753+0000","last_change":"2023-11-28T18:23:08.112410+0000","last_active":"2023-11-28T18:23:09.565753+0000","last_peered":"2023-11-28T18:23:09.565753+0000","last_clean":"2023-11-28T18:23:09.565753+0000","last_became_active":"2023-11-28T18:23:08.112097+0000","last_became_peered":"2023-11-28T18:23:08.112097+0000","last_unstale":"2023-11-28T18:23:09.565753+0000","last_undegraded":"2023-11-28T18:23:09.565753+0000","last_fullsized":"2023-11-28T18:23:09.565753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:57.891789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.719092+0000","last_change":"2023-11-28T18:23:07.684818+0000","last_active":"2023-11-28T18:23:18.719092+0000","last_peered":"2023-11-28T18:23:18.719092+0000","last_clean":"2023-11-28T18:23:18.719092+0000","last_became_active":"2023-11-28T18:23:07.684502+0000","last_became_peered":"2023-11-28T18:23:07.684502+0000","last_unstale":"2023-11-28T18:23:18.719092+0000","last_undegraded":"2023-11-28T18:23:18.719092+0000","last_fullsized":"2023-11-28T18:23:18.719092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:16:21.223266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109527+0000","last_change":"2023-11-28T18:23:08.133972+0000","last_active":"2023-11-28T18:23:09.109527+0000","last_peered":"2023-11-28T18:23:09.109527+0000","last_clean":"2023-11-28T18:23:09.109527+0000","last_became_active":"2023-11-28T18:23:08.133747+0000","last_became_peered":"2023-11-28T18:23:08.133747+0000","last_unstale":"2023-11-28T18:23:09.109527+0000","last_undegraded":"2023-11-28T18:23:09.109527+0000","last_fullsized":"2023-11-28T18:23:09.109527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:24.968241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113538+0000","last_change":"2023-11-28T18:23:08.134117+0000","last_active":"2023-11-28T18:23:09.113538+0000","last_peered":"2023-11-28T18:23:09.113538+0000","last_clean":"2023-11-28T18:23:09.113538+0000","last_became_active":"2023-11-28T18:23:08.133801+0000","last_became_peered":"2023-11-28T18:23:08.133801+0000","last_unstale":"2023-11-28T18:23:09.113538+0000","last_undegraded":"2023-11-28T18:23:09.113538+0000","last_fullsized":"2023-11-28T18:23:09.113538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:02.370862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250642+0000","last_change":"2023-11-28T18:23:07.686399+0000","last_active":"2023-11-28T18:23:35.250642+0000","last_peered":"2023-11-28T18:23:35.250642+0000","last_clean":"2023-11-28T18:23:35.250642+0000","last_became_active":"2023-11-28T18:23:07.686051+0000","last_became_peered":"2023-11-28T18:23:07.686051+0000","last_unstale":"2023-11-28T18:23:35.250642+0000","last_undegraded":"2023-11-28T18:23:35.250642+0000","last_fullsized":"2023-11-28T18:23:35.250642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:11.797516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113317+0000","last_change":"2023-11-28T18:23:08.134870+0000","last_active":"2023-11-28T18:23:09.113317+0000","last_peered":"2023-11-28T18:23:09.113317+0000","last_clean":"2023-11-28T18:23:09.113317+0000","last_became_active":"2023-11-28T18:23:08.133732+0000","last_became_peered":"2023-11-28T18:23:08.133732+0000","last_unstale":"2023-11-28T18:23:09.113317+0000","last_undegraded":"2023-11-28T18:23:09.113317+0000","last_fullsized":"2023-11-28T18:23:09.113317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:01.364242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.252608+0000","last_change":"2023-11-28T18:23:07.682230+0000","last_active":"2023-11-28T18:23:35.252608+0000","last_peered":"2023-11-28T18:23:35.252608+0000","last_clean":"2023-11-28T18:23:35.252608+0000","last_became_active":"2023-11-28T18:23:07.681715+0000","last_became_peered":"2023-11-28T18:23:07.681715+0000","last_unstale":"2023-11-28T18:23:35.252608+0000","last_undegraded":"2023-11-28T18:23:35.252608+0000","last_fullsized":"2023-11-28T18:23:35.252608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:02.343996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914216+0000","last_change":"2023-11-28T18:23:08.128927+0000","last_active":"2023-11-28T18:23:10.914216+0000","last_peered":"2023-11-28T18:23:10.914216+0000","last_clean":"2023-11-28T18:23:10.914216+0000","last_became_active":"2023-11-28T18:23:08.127775+0000","last_became_peered":"2023-11-28T18:23:08.127775+0000","last_unstale":"2023-11-28T18:23:10.914216+0000","last_undegraded":"2023-11-28T18:23:10.914216+0000","last_fullsized":"2023-11-28T18:23:10.914216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:50:05.531589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.669760+0000","last_change":"2023-11-28T18:23:07.667929+0000","last_active":"2023-11-28T18:23:13.669760+0000","last_peered":"2023-11-28T18:23:13.669760+0000","last_clean":"2023-11-28T18:23:13.669760+0000","last_became_active":"2023-11-28T18:23:07.667666+0000","last_became_peered":"2023-11-28T18:23:07.667666+0000","last_unstale":"2023-11-28T18:23:13.669760+0000","last_undegraded":"2023-11-28T18:23:13.669760+0000","last_fullsized":"2023-11-28T18:23:13.669760+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:32.132092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565609+0000","last_change":"2023-11-28T18:23:08.126404+0000","last_active":"2023-11-28T18:23:09.565609+0000","last_peered":"2023-11-28T18:23:09.565609+0000","last_clean":"2023-11-28T18:23:09.565609+0000","last_became_active":"2023-11-28T18:23:08.125315+0000","last_became_peered":"2023-11-28T18:23:08.125315+0000","last_unstale":"2023-11-28T18:23:09.565609+0000","last_undegraded":"2023-11-28T18:23:09.565609+0000","last_fullsized":"2023-11-28T18:23:09.565609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:24:59.527642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":25,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250692+0000","last_change":"2023-11-28T18:23:07.677758+0000","last_active":"2023-11-28T18:23:35.250692+0000","last_peered":"2023-11-28T18:23:35.250692+0000","last_clean":"2023-11-28T18:23:35.250692+0000","last_became_active":"2023-11-28T18:23:07.676902+0000","last_became_peered":"2023-11-28T18:23:07.676902+0000","last_unstale":"2023-11-28T18:23:35.250692+0000","last_undegraded":"2023-11-28T18:23:35.250692+0000","last_fullsized":"2023-11-28T18:23:35.250692+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:50.953647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9794,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":835584,"data_stored":663818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":74192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":74842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":72240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":73346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":72812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":233472,"data_stored":72442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":827392,"data_stored":662152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T18:23:50.123 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-11-28T18:23:50.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T18:23:50.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f241532b640 1 Processor -- start 2023-11-28T18:23:50.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f241532b640 1 -- start start 2023-11-28T18:23:50.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f241532b640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2410153ee0 0x7f24100cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f241532b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2410159c90 0x7f241015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f241532b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f241015def0 0x7f24100ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f241532b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241015d240 con 0x7f2410159c90 2023-11-28T18:23:50.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f241532b640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241015d520 con 0x7f2410153ee0 2023-11-28T18:23:50.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f241532b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24100c4bc0 con 0x7f241015def0 2023-11-28T18:23:50.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2410159c90 0x7f241015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2410153ee0 0x7f24100cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f2414b2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f241015def0 0x7f24100ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2410159c90 0x7f241015d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47510/0 (socket says 172.21.15.7:47510) 2023-11-28T18:23:50.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240f7fe640 1 -- 172.21.15.7:0/3399161845 learned_addr learned my addr 172.21.15.7:0/3399161845 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:50.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240f7fe640 1 -- 172.21.15.7:0/3399161845 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f241015def0 msgr2=0x7f24100ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:50.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240f7fe640 1 --2- 172.21.15.7:0/3399161845 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f241015def0 0x7f24100ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240f7fe640 1 -- 172.21.15.7:0/3399161845 >> v2:172.21.15.193:3300/0 conn(0x7f2410153ee0 msgr2=0x7f24100cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:50.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240f7fe640 1 --2- 172.21.15.7:0/3399161845 >> v2:172.21.15.193:3300/0 conn(0x7f2410153ee0 0x7f24100cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.303+0000 7f240f7fe640 1 -- 172.21.15.7:0/3399161845 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241015a240 con 0x7f2410159c90 2023-11-28T18:23:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f240f7fe640 1 --2- 172.21.15.7:0/3399161845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2410159c90 0x7f241015d9b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2410072af0 tx=0x7f240401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a2b3c40d080b2f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f240effd640 1 -- 172.21.15.7:0/3399161845 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24040a89b0 con 0x7f2410159c90 2023-11-28T18:23:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f240effd640 1 -- 172.21.15.7:0/3399161845 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24040a8b10 con 0x7f2410159c90 2023-11-28T18:23:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f240effd640 1 -- 172.21.15.7:0/3399161845 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24040a8ea0 con 0x7f2410159c90 2023-11-28T18:23:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 -- 172.21.15.7:0/3399161845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2410159c90 msgr2=0x7f241015d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 --2- 172.21.15.7:0/3399161845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2410159c90 0x7f241015d9b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2410072af0 tx=0x7f240401ecf0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 -- 172.21.15.7:0/3399161845 shutdown_connections 2023-11-28T18:23:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 --2- 172.21.15.7:0/3399161845 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f241015def0 0x7f24100ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 --2- 172.21.15.7:0/3399161845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2410159c90 0x7f241015d9b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 --2- 172.21.15.7:0/3399161845 >> v2:172.21.15.193:3300/0 conn(0x7f2410153ee0 0x7f24100cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 -- 172.21.15.7:0/3399161845 >> 172.21.15.7:0/3399161845 conn(0x7f2410071e20 msgr2=0x7f2410072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 -- 172.21.15.7:0/3399161845 shutdown_connections 2023-11-28T18:23:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.307+0000 7f241532b640 1 -- 172.21.15.7:0/3399161845 wait complete. 2023-11-28T18:23:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 Processor -- start 2023-11-28T18:23:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 -- start start 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2410153ee0 0x7f24100b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2410159c90 0x7f24101f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f241015def0 0x7f24101f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241015be60 con 0x7f241015def0 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24101f2d40 con 0x7f2410153ee0 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241015c0b0 con 0x7f2410159c90 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f241015def0 0x7f24101f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f241015def0 0x7f24101f3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47518/0 (socket says 172.21.15.7:47518) 2023-11-28T18:23:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 -- 172.21.15.7:0/1230675536 learned_addr learned my addr 172.21.15.7:0/1230675536 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:50.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f240f7fe640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2410159c90 0x7f24101f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f240ffff640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2410153ee0 0x7f24100b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 -- 172.21.15.7:0/1230675536 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2410159c90 msgr2=0x7f24101f31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2410159c90 0x7f24101f31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 -- 172.21.15.7:0/1230675536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2410153ee0 msgr2=0x7f24100b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2410153ee0 0x7f24100b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 -- 172.21.15.7:0/1230675536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24040094d0 con 0x7f241015def0 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f2414b2a640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f241015def0 0x7f24101f3730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2408009e30 tx=0x7f240800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f240cff9640 1 -- 172.21.15.7:0/1230675536 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408009ca0 con 0x7f241015def0 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f240cff9640 1 -- 172.21.15.7:0/1230675536 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240800f040 con 0x7f241015def0 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241015c730 con 0x7f241015def0 2023-11-28T18:23:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.311+0000 7f240cff9640 1 -- 172.21.15.7:0/1230675536 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24080146c0 con 0x7f241015def0 2023-11-28T18:23:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.315+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241014f550 con 0x7f241015def0 2023-11-28T18:23:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.315+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24101f2ed0 con 0x7f241015def0 2023-11-28T18:23:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.315+0000 7f240cff9640 1 -- 172.21.15.7:0/1230675536 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2408014820 con 0x7f241015def0 2023-11-28T18:23:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.315+0000 7f240cff9640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23fc0668d0 0x7f23fc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.315+0000 7f240cff9640 1 -- 172.21.15.7:0/1230675536 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f24080852d0 con 0x7f241015def0 2023-11-28T18:23:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.315+0000 7f240ffff640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23fc0668d0 0x7f23fc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.315+0000 7f240ffff640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23fc0668d0 0x7f23fc068d90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2400009a40 tx=0x7f2400009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:50.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.319+0000 7f240cff9640 1 -- 172.21.15.7:0/1230675536 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2408052540 con 0x7f241015def0 2023-11-28T18:23:50.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.471+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 --> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f241015aac0 con 0x7f23fc0668d0 2023-11-28T18:23:50.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.479+0000 7f240cff9640 1 -- 172.21.15.7:0/1230675536 <== mgr.4102 v2:172.21.15.7:6832/264069339 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264664 (secure 0 0 0) 0x7f2400011a00 con 0x7f23fc0668d0 2023-11-28T18:23:50.489 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:50.490 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-28T18:23:50.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23fc0668d0 msgr2=0x7f23fc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23fc0668d0 0x7f23fc068d90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2400009a40 tx=0x7f2400009190 comp rx=0 tx=0).stop 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f241015def0 msgr2=0x7f24101f3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f241015def0 0x7f24101f3730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2408009e30 tx=0x7f240800c490 comp rx=0 tx=0).stop 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 shutdown_connections 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23fc0668d0 0x7f23fc068d90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f241015def0 0x7f24101f3730 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2410159c90 0x7f24101f31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 --2- 172.21.15.7:0/1230675536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2410153ee0 0x7f24100b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 >> 172.21.15.7:0/1230675536 conn(0x7f2410071e20 msgr2=0x7f24100c4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:50.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 shutdown_connections 2023-11-28T18:23:50.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.483+0000 7f241532b640 1 -- 172.21.15.7:0/1230675536 wait complete. 2023-11-28T18:23:50.509 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-11-28T18:23:49.470780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600162,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":371,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235784,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738936,"statfs":{"total":773094113280,"available":772852670464,"internally_reserved":0,"allocated":3088384,"data_stored":1765844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1760,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001731"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001273+0000","last_change":"2023-11-28T18:23:08.134633+0000","last_active":"2023-11-28T18:23:11.001273+0000","last_peered":"2023-11-28T18:23:11.001273+0000","last_clean":"2023-11-28T18:23:11.001273+0000","last_became_active":"2023-11-28T18:23:08.134320+0000","last_became_peered":"2023-11-28T18:23:08.134320+0000","last_unstale":"2023-11-28T18:23:11.001273+0000","last_undegraded":"2023-11-28T18:23:11.001273+0000","last_fullsized":"2023-11-28T18:23:11.001273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:54.995718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105857+0000","last_change":"2023-11-28T18:23:08.122460+0000","last_active":"2023-11-28T18:23:09.105857+0000","last_peered":"2023-11-28T18:23:09.105857+0000","last_clean":"2023-11-28T18:23:09.105857+0000","last_became_active":"2023-11-28T18:23:08.122153+0000","last_became_peered":"2023-11-28T18:23:08.122153+0000","last_unstale":"2023-11-28T18:23:09.105857+0000","last_undegraded":"2023-11-28T18:23:09.105857+0000","last_fullsized":"2023-11-28T18:23:09.105857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:17:31.074978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914207+0000","last_change":"2023-11-28T18:23:08.110187+0000","last_active":"2023-11-28T18:23:10.914207+0000","last_peered":"2023-11-28T18:23:10.914207+0000","last_clean":"2023-11-28T18:23:10.914207+0000","last_became_active":"2023-11-28T18:23:08.109915+0000","last_became_peered":"2023-11-28T18:23:08.109915+0000","last_unstale":"2023-11-28T18:23:10.914207+0000","last_undegraded":"2023-11-28T18:23:10.914207+0000","last_fullsized":"2023-11-28T18:23:10.914207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:32.098408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113335+0000","last_change":"2023-11-28T18:23:08.122000+0000","last_active":"2023-11-28T18:23:09.113335+0000","last_peered":"2023-11-28T18:23:09.113335+0000","last_clean":"2023-11-28T18:23:09.113335+0000","last_became_active":"2023-11-28T18:23:08.121652+0000","last_became_peered":"2023-11-28T18:23:08.121652+0000","last_unstale":"2023-11-28T18:23:09.113335+0000","last_undegraded":"2023-11-28T18:23:09.113335+0000","last_fullsized":"2023-11-28T18:23:09.113335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:15.540606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000810+0000","last_change":"2023-11-28T18:23:08.126587+0000","last_active":"2023-11-28T18:23:11.000810+0000","last_peered":"2023-11-28T18:23:11.000810+0000","last_clean":"2023-11-28T18:23:11.000810+0000","last_became_active":"2023-11-28T18:23:08.126313+0000","last_became_peered":"2023-11-28T18:23:08.126313+0000","last_unstale":"2023-11-28T18:23:11.000810+0000","last_undegraded":"2023-11-28T18:23:11.000810+0000","last_fullsized":"2023-11-28T18:23:11.000810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.956197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113040+0000","last_change":"2023-11-28T18:23:08.122878+0000","last_active":"2023-11-28T18:23:09.113040+0000","last_peered":"2023-11-28T18:23:09.113040+0000","last_clean":"2023-11-28T18:23:09.113040+0000","last_became_active":"2023-11-28T18:23:08.122609+0000","last_became_peered":"2023-11-28T18:23:08.122609+0000","last_unstale":"2023-11-28T18:23:09.113040+0000","last_undegraded":"2023-11-28T18:23:09.113040+0000","last_fullsized":"2023-11-28T18:23:09.113040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:00.610074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000604+0000","last_change":"2023-11-28T18:23:08.133485+0000","last_active":"2023-11-28T18:23:11.000604+0000","last_peered":"2023-11-28T18:23:11.000604+0000","last_clean":"2023-11-28T18:23:11.000604+0000","last_became_active":"2023-11-28T18:23:08.133211+0000","last_became_peered":"2023-11-28T18:23:08.133211+0000","last_unstale":"2023-11-28T18:23:11.000604+0000","last_undegraded":"2023-11-28T18:23:11.000604+0000","last_fullsized":"2023-11-28T18:23:11.000604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:04:33.507685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565355+0000","last_change":"2023-11-28T18:23:08.117903+0000","last_active":"2023-11-28T18:23:09.565355+0000","last_peered":"2023-11-28T18:23:09.565355+0000","last_clean":"2023-11-28T18:23:09.565355+0000","last_became_active":"2023-11-28T18:23:08.117482+0000","last_became_peered":"2023-11-28T18:23:08.117482+0000","last_unstale":"2023-11-28T18:23:09.565355+0000","last_undegraded":"2023-11-28T18:23:09.565355+0000","last_fullsized":"2023-11-28T18:23:09.565355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:45.293402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001351+0000","last_change":"2023-11-28T18:23:08.135606+0000","last_active":"2023-11-28T18:23:11.001351+0000","last_peered":"2023-11-28T18:23:11.001351+0000","last_clean":"2023-11-28T18:23:11.001351+0000","last_became_active":"2023-11-28T18:23:08.135328+0000","last_became_peered":"2023-11-28T18:23:08.135328+0000","last_unstale":"2023-11-28T18:23:11.001351+0000","last_undegraded":"2023-11-28T18:23:11.001351+0000","last_fullsized":"2023-11-28T18:23:11.001351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:25.011869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110046+0000","last_change":"2023-11-28T18:23:08.123993+0000","last_active":"2023-11-28T18:23:09.110046+0000","last_peered":"2023-11-28T18:23:09.110046+0000","last_clean":"2023-11-28T18:23:09.110046+0000","last_became_active":"2023-11-28T18:23:08.123583+0000","last_became_peered":"2023-11-28T18:23:08.123583+0000","last_unstale":"2023-11-28T18:23:09.110046+0000","last_undegraded":"2023-11-28T18:23:09.110046+0000","last_fullsized":"2023-11-28T18:23:09.110046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:16:33.089307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109780+0000","last_change":"2023-11-28T18:23:08.120983+0000","last_active":"2023-11-28T18:23:09.109780+0000","last_peered":"2023-11-28T18:23:09.109780+0000","last_clean":"2023-11-28T18:23:09.109780+0000","last_became_active":"2023-11-28T18:23:08.120486+0000","last_became_peered":"2023-11-28T18:23:08.120486+0000","last_unstale":"2023-11-28T18:23:09.109780+0000","last_undegraded":"2023-11-28T18:23:09.109780+0000","last_fullsized":"2023-11-28T18:23:09.109780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:46:12.881629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637634+0000","last_change":"2023-11-28T18:23:08.130169+0000","last_active":"2023-11-28T18:23:09.637634+0000","last_peered":"2023-11-28T18:23:09.637634+0000","last_clean":"2023-11-28T18:23:09.637634+0000","last_became_active":"2023-11-28T18:23:08.129771+0000","last_became_peered":"2023-11-28T18:23:08.129771+0000","last_unstale":"2023-11-28T18:23:09.637634+0000","last_undegraded":"2023-11-28T18:23:09.637634+0000","last_fullsized":"2023-11-28T18:23:09.637634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:45.753322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565505+0000","last_change":"2023-11-28T18:23:08.123268+0000","last_active":"2023-11-28T18:23:09.565505+0000","last_peered":"2023-11-28T18:23:09.565505+0000","last_clean":"2023-11-28T18:23:09.565505+0000","last_became_active":"2023-11-28T18:23:08.122978+0000","last_became_peered":"2023-11-28T18:23:08.122978+0000","last_unstale":"2023-11-28T18:23:09.565505+0000","last_undegraded":"2023-11-28T18:23:09.565505+0000","last_fullsized":"2023-11-28T18:23:09.565505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:15:01.320918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913943+0000","last_change":"2023-11-28T18:23:08.118361+0000","last_active":"2023-11-28T18:23:10.913943+0000","last_peered":"2023-11-28T18:23:10.913943+0000","last_clean":"2023-11-28T18:23:10.913943+0000","last_became_active":"2023-11-28T18:23:08.117973+0000","last_became_peered":"2023-11-28T18:23:08.117973+0000","last_unstale":"2023-11-28T18:23:10.913943+0000","last_undegraded":"2023-11-28T18:23:10.913943+0000","last_fullsized":"2023-11-28T18:23:10.913943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:57.485937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105530+0000","last_change":"2023-11-28T18:23:08.139744+0000","last_active":"2023-11-28T18:23:09.105530+0000","last_peered":"2023-11-28T18:23:09.105530+0000","last_clean":"2023-11-28T18:23:09.105530+0000","last_became_active":"2023-11-28T18:23:08.139464+0000","last_became_peered":"2023-11-28T18:23:08.139464+0000","last_unstale":"2023-11-28T18:23:09.105530+0000","last_undegraded":"2023-11-28T18:23:09.105530+0000","last_fullsized":"2023-11-28T18:23:09.105530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:19:34.780979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113036+0000","last_change":"2023-11-28T18:23:08.119103+0000","last_active":"2023-11-28T18:23:09.113036+0000","last_peered":"2023-11-28T18:23:09.113036+0000","last_clean":"2023-11-28T18:23:09.113036+0000","last_became_active":"2023-11-28T18:23:08.116080+0000","last_became_peered":"2023-11-28T18:23:08.116080+0000","last_unstale":"2023-11-28T18:23:09.113036+0000","last_undegraded":"2023-11-28T18:23:09.113036+0000","last_fullsized":"2023-11-28T18:23:09.113036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:44.279413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113688+0000","last_change":"2023-11-28T18:23:08.126817+0000","last_active":"2023-11-28T18:23:09.113688+0000","last_peered":"2023-11-28T18:23:09.113688+0000","last_clean":"2023-11-28T18:23:09.113688+0000","last_became_active":"2023-11-28T18:23:08.126540+0000","last_became_peered":"2023-11-28T18:23:08.126540+0000","last_unstale":"2023-11-28T18:23:09.113688+0000","last_undegraded":"2023-11-28T18:23:09.113688+0000","last_fullsized":"2023-11-28T18:23:09.113688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:08.807847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271940+0000","last_change":"2023-11-28T18:23:08.121775+0000","last_active":"2023-11-28T18:23:10.271940+0000","last_peered":"2023-11-28T18:23:10.271940+0000","last_clean":"2023-11-28T18:23:10.271940+0000","last_became_active":"2023-11-28T18:23:08.121466+0000","last_became_peered":"2023-11-28T18:23:08.121466+0000","last_unstale":"2023-11-28T18:23:10.271940+0000","last_undegraded":"2023-11-28T18:23:10.271940+0000","last_fullsized":"2023-11-28T18:23:10.271940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:32:51.612113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001115+0000","last_change":"2023-11-28T18:23:08.138282+0000","last_active":"2023-11-28T18:23:11.001115+0000","last_peered":"2023-11-28T18:23:11.001115+0000","last_clean":"2023-11-28T18:23:11.001115+0000","last_became_active":"2023-11-28T18:23:08.137171+0000","last_became_peered":"2023-11-28T18:23:08.137171+0000","last_unstale":"2023-11-28T18:23:11.001115+0000","last_undegraded":"2023-11-28T18:23:11.001115+0000","last_fullsized":"2023-11-28T18:23:11.001115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:39:07.113697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110124+0000","last_change":"2023-11-28T18:23:08.133521+0000","last_active":"2023-11-28T18:23:09.110124+0000","last_peered":"2023-11-28T18:23:09.110124+0000","last_clean":"2023-11-28T18:23:09.110124+0000","last_became_active":"2023-11-28T18:23:08.133248+0000","last_became_peered":"2023-11-28T18:23:08.133248+0000","last_unstale":"2023-11-28T18:23:09.110124+0000","last_undegraded":"2023-11-28T18:23:09.110124+0000","last_fullsized":"2023-11-28T18:23:09.110124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:17:34.585808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914195+0000","last_change":"2023-11-28T18:23:08.129150+0000","last_active":"2023-11-28T18:23:10.914195+0000","last_peered":"2023-11-28T18:23:10.914195+0000","last_clean":"2023-11-28T18:23:10.914195+0000","last_became_active":"2023-11-28T18:23:08.128622+0000","last_became_peered":"2023-11-28T18:23:08.128622+0000","last_unstale":"2023-11-28T18:23:10.914195+0000","last_undegraded":"2023-11-28T18:23:10.914195+0000","last_fullsized":"2023-11-28T18:23:10.914195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:33.197161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113274+0000","last_change":"2023-11-28T18:23:08.131333+0000","last_active":"2023-11-28T18:23:09.113274+0000","last_peered":"2023-11-28T18:23:09.113274+0000","last_clean":"2023-11-28T18:23:09.113274+0000","last_became_active":"2023-11-28T18:23:08.130917+0000","last_became_peered":"2023-11-28T18:23:08.130917+0000","last_unstale":"2023-11-28T18:23:09.113274+0000","last_undegraded":"2023-11-28T18:23:09.113274+0000","last_fullsized":"2023-11-28T18:23:09.113274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:04:48.114417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565507+0000","last_change":"2023-11-28T18:23:08.132355+0000","last_active":"2023-11-28T18:23:09.565507+0000","last_peered":"2023-11-28T18:23:09.565507+0000","last_clean":"2023-11-28T18:23:09.565507+0000","last_became_active":"2023-11-28T18:23:08.132016+0000","last_became_peered":"2023-11-28T18:23:08.132016+0000","last_unstale":"2023-11-28T18:23:09.565507+0000","last_undegraded":"2023-11-28T18:23:09.565507+0000","last_fullsized":"2023-11-28T18:23:09.565507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:11.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113066+0000","last_change":"2023-11-28T18:23:08.119935+0000","last_active":"2023-11-28T18:23:09.113066+0000","last_peered":"2023-11-28T18:23:09.113066+0000","last_clean":"2023-11-28T18:23:09.113066+0000","last_became_active":"2023-11-28T18:23:08.118372+0000","last_became_peered":"2023-11-28T18:23:08.118372+0000","last_unstale":"2023-11-28T18:23:09.113066+0000","last_undegraded":"2023-11-28T18:23:09.113066+0000","last_fullsized":"2023-11-28T18:23:09.113066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:52.339024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.132272+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.131985+0000","last_became_peered":"2023-11-28T18:23:08.131985+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:44.594691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.110084+0000","last_change":"2023-11-28T18:23:08.127816+0000","last_active":"2023-11-28T18:23:09.110084+0000","last_peered":"2023-11-28T18:23:09.110084+0000","last_clean":"2023-11-28T18:23:09.110084+0000","last_became_active":"2023-11-28T18:23:08.127380+0000","last_became_peered":"2023-11-28T18:23:08.127380+0000","last_unstale":"2023-11-28T18:23:09.110084+0000","last_undegraded":"2023-11-28T18:23:09.110084+0000","last_fullsized":"2023-11-28T18:23:09.110084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:31.242361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105706+0000","last_change":"2023-11-28T18:23:08.128188+0000","last_active":"2023-11-28T18:23:09.105706+0000","last_peered":"2023-11-28T18:23:09.105706+0000","last_clean":"2023-11-28T18:23:09.105706+0000","last_became_active":"2023-11-28T18:23:08.127860+0000","last_became_peered":"2023-11-28T18:23:08.127860+0000","last_unstale":"2023-11-28T18:23:09.105706+0000","last_undegraded":"2023-11-28T18:23:09.105706+0000","last_fullsized":"2023-11-28T18:23:09.105706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:12.095768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565712+0000","last_change":"2023-11-28T18:23:08.132381+0000","last_active":"2023-11-28T18:23:09.565712+0000","last_peered":"2023-11-28T18:23:09.565712+0000","last_clean":"2023-11-28T18:23:09.565712+0000","last_became_active":"2023-11-28T18:23:08.132072+0000","last_became_peered":"2023-11-28T18:23:08.132072+0000","last_unstale":"2023-11-28T18:23:09.565712+0000","last_undegraded":"2023-11-28T18:23:09.565712+0000","last_fullsized":"2023-11-28T18:23:09.565712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:16:38.666702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914263+0000","last_change":"2023-11-28T18:23:08.115379+0000","last_active":"2023-11-28T18:23:10.914263+0000","last_peered":"2023-11-28T18:23:10.914263+0000","last_clean":"2023-11-28T18:23:10.914263+0000","last_became_active":"2023-11-28T18:23:08.115236+0000","last_became_peered":"2023-11-28T18:23:08.115236+0000","last_unstale":"2023-11-28T18:23:10.914263+0000","last_undegraded":"2023-11-28T18:23:10.914263+0000","last_fullsized":"2023-11-28T18:23:10.914263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:26:52.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113293+0000","last_change":"2023-11-28T18:23:08.121402+0000","last_active":"2023-11-28T18:23:09.113293+0000","last_peered":"2023-11-28T18:23:09.113293+0000","last_clean":"2023-11-28T18:23:09.113293+0000","last_became_active":"2023-11-28T18:23:08.121129+0000","last_became_peered":"2023-11-28T18:23:08.121129+0000","last_unstale":"2023-11-28T18:23:09.113293+0000","last_undegraded":"2023-11-28T18:23:09.113293+0000","last_fullsized":"2023-11-28T18:23:09.113293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:02.927891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.271829+0000","last_change":"2023-11-28T18:23:08.121594+0000","last_active":"2023-11-28T18:23:10.271829+0000","last_peered":"2023-11-28T18:23:10.271829+0000","last_clean":"2023-11-28T18:23:10.271829+0000","last_became_active":"2023-11-28T18:23:08.119009+0000","last_became_peered":"2023-11-28T18:23:08.119009+0000","last_unstale":"2023-11-28T18:23:10.271829+0000","last_undegraded":"2023-11-28T18:23:10.271829+0000","last_fullsized":"2023-11-28T18:23:10.271829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:38.801632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109676+0000","last_change":"2023-11-28T18:23:08.115403+0000","last_active":"2023-11-28T18:23:09.109676+0000","last_peered":"2023-11-28T18:23:09.109676+0000","last_clean":"2023-11-28T18:23:09.109676+0000","last_became_active":"2023-11-28T18:23:08.114955+0000","last_became_peered":"2023-11-28T18:23:08.114955+0000","last_unstale":"2023-11-28T18:23:09.109676+0000","last_undegraded":"2023-11-28T18:23:09.109676+0000","last_fullsized":"2023-11-28T18:23:09.109676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:41.242830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637404+0000","last_change":"2023-11-28T18:23:08.129745+0000","last_active":"2023-11-28T18:23:09.637404+0000","last_peered":"2023-11-28T18:23:09.637404+0000","last_clean":"2023-11-28T18:23:09.637404+0000","last_became_active":"2023-11-28T18:23:08.129388+0000","last_became_peered":"2023-11-28T18:23:08.129388+0000","last_unstale":"2023-11-28T18:23:09.637404+0000","last_undegraded":"2023-11-28T18:23:09.637404+0000","last_fullsized":"2023-11-28T18:23:09.637404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:50.570821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.743781+0000","last_change":"2023-11-28T18:23:07.664773+0000","last_active":"2023-11-28T18:23:18.743781+0000","last_peered":"2023-11-28T18:23:18.743781+0000","last_clean":"2023-11-28T18:23:18.743781+0000","last_became_active":"2023-11-28T18:23:07.664478+0000","last_became_peered":"2023-11-28T18:23:07.664478+0000","last_unstale":"2023-11-28T18:23:18.743781+0000","last_undegraded":"2023-11-28T18:23:18.743781+0000","last_fullsized":"2023-11-28T18:23:18.743781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:28.322454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914337+0000","last_change":"2023-11-28T18:23:08.129441+0000","last_active":"2023-11-28T18:23:10.914337+0000","last_peered":"2023-11-28T18:23:10.914337+0000","last_clean":"2023-11-28T18:23:10.914337+0000","last_became_active":"2023-11-28T18:23:08.129143+0000","last_became_peered":"2023-11-28T18:23:08.129143+0000","last_unstale":"2023-11-28T18:23:10.914337+0000","last_undegraded":"2023-11-28T18:23:10.914337+0000","last_fullsized":"2023-11-28T18:23:10.914337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:36.540819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.173128+0000","last_change":"2023-11-28T18:23:07.668192+0000","last_active":"2023-11-28T18:23:30.173128+0000","last_peered":"2023-11-28T18:23:30.173128+0000","last_clean":"2023-11-28T18:23:30.173128+0000","last_became_active":"2023-11-28T18:23:07.667780+0000","last_became_peered":"2023-11-28T18:23:07.667780+0000","last_unstale":"2023-11-28T18:23:30.173128+0000","last_undegraded":"2023-11-28T18:23:30.173128+0000","last_fullsized":"2023-11-28T18:23:30.173128+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:24.148981+0000","last_change":"2023-11-28T18:23:07.665608+0000","last_active":"2023-11-28T18:23:24.148981+0000","last_peered":"2023-11-28T18:23:24.148981+0000","last_clean":"2023-11-28T18:23:24.148981+0000","last_became_active":"2023-11-28T18:23:07.665361+0000","last_became_peered":"2023-11-28T18:23:07.665361+0000","last_unstale":"2023-11-28T18:23:24.148981+0000","last_undegraded":"2023-11-28T18:23:24.148981+0000","last_fullsized":"2023-11-28T18:23:24.148981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:17.279597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565782+0000","last_change":"2023-11-28T18:23:08.118091+0000","last_active":"2023-11-28T18:23:09.565782+0000","last_peered":"2023-11-28T18:23:09.565782+0000","last_clean":"2023-11-28T18:23:09.565782+0000","last_became_active":"2023-11-28T18:23:08.117524+0000","last_became_peered":"2023-11-28T18:23:08.117524+0000","last_unstale":"2023-11-28T18:23:09.565782+0000","last_undegraded":"2023-11-28T18:23:09.565782+0000","last_fullsized":"2023-11-28T18:23:09.565782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:04.054226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171679+0000","last_change":"2023-11-28T18:23:07.667847+0000","last_active":"2023-11-28T18:23:30.171679+0000","last_peered":"2023-11-28T18:23:30.171679+0000","last_clean":"2023-11-28T18:23:30.171679+0000","last_became_active":"2023-11-28T18:23:07.667466+0000","last_became_peered":"2023-11-28T18:23:07.667466+0000","last_unstale":"2023-11-28T18:23:30.171679+0000","last_undegraded":"2023-11-28T18:23:30.171679+0000","last_fullsized":"2023-11-28T18:23:30.171679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:06.807146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.127497+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.127216+0000","last_became_peered":"2023-11-28T18:23:08.127216+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:03:14.999484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172207+0000","last_change":"2023-11-28T18:23:07.681698+0000","last_active":"2023-11-28T18:23:30.172207+0000","last_peered":"2023-11-28T18:23:30.172207+0000","last_clean":"2023-11-28T18:23:30.172207+0000","last_became_active":"2023-11-28T18:23:07.681365+0000","last_became_peered":"2023-11-28T18:23:07.681365+0000","last_unstale":"2023-11-28T18:23:30.172207+0000","last_undegraded":"2023-11-28T18:23:30.172207+0000","last_fullsized":"2023-11-28T18:23:30.172207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:21.985029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113361+0000","last_change":"2023-11-28T18:23:08.132797+0000","last_active":"2023-11-28T18:23:09.113361+0000","last_peered":"2023-11-28T18:23:09.113361+0000","last_clean":"2023-11-28T18:23:09.113361+0000","last_became_active":"2023-11-28T18:23:08.132525+0000","last_became_peered":"2023-11-28T18:23:08.132525+0000","last_unstale":"2023-11-28T18:23:09.113361+0000","last_undegraded":"2023-11-28T18:23:09.113361+0000","last_fullsized":"2023-11-28T18:23:09.113361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:12:10.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.248118+0000","last_change":"2023-11-28T18:23:07.672337+0000","last_active":"2023-11-28T18:23:35.248118+0000","last_peered":"2023-11-28T18:23:35.248118+0000","last_clean":"2023-11-28T18:23:35.248118+0000","last_became_active":"2023-11-28T18:23:07.671922+0000","last_became_peered":"2023-11-28T18:23:07.671922+0000","last_unstale":"2023-11-28T18:23:35.248118+0000","last_undegraded":"2023-11-28T18:23:35.248118+0000","last_fullsized":"2023-11-28T18:23:35.248118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:22.936770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914223+0000","last_change":"2023-11-28T18:23:08.127919+0000","last_active":"2023-11-28T18:23:10.914223+0000","last_peered":"2023-11-28T18:23:10.914223+0000","last_clean":"2023-11-28T18:23:10.914223+0000","last_became_active":"2023-11-28T18:23:08.127602+0000","last_became_peered":"2023-11-28T18:23:08.127602+0000","last_unstale":"2023-11-28T18:23:10.914223+0000","last_undegraded":"2023-11-28T18:23:10.914223+0000","last_fullsized":"2023-11-28T18:23:10.914223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:32:45.967018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.734841+0000","last_change":"2023-11-28T18:23:07.679219+0000","last_active":"2023-11-28T18:23:18.734841+0000","last_peered":"2023-11-28T18:23:18.734841+0000","last_clean":"2023-11-28T18:23:18.734841+0000","last_became_active":"2023-11-28T18:23:07.678920+0000","last_became_peered":"2023-11-28T18:23:07.678920+0000","last_unstale":"2023-11-28T18:23:18.734841+0000","last_undegraded":"2023-11-28T18:23:18.734841+0000","last_fullsized":"2023-11-28T18:23:18.734841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:59:02.332690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637820+0000","last_change":"2023-11-28T18:23:08.132429+0000","last_active":"2023-11-28T18:23:09.637820+0000","last_peered":"2023-11-28T18:23:09.637820+0000","last_clean":"2023-11-28T18:23:09.637820+0000","last_became_active":"2023-11-28T18:23:08.132106+0000","last_became_peered":"2023-11-28T18:23:08.132106+0000","last_unstale":"2023-11-28T18:23:09.637820+0000","last_undegraded":"2023-11-28T18:23:09.637820+0000","last_fullsized":"2023-11-28T18:23:09.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:52.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'12","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:43.822203+0000","last_change":"2023-11-28T18:23:07.667479+0000","last_active":"2023-11-28T18:23:43.822203+0000","last_peered":"2023-11-28T18:23:43.822203+0000","last_clean":"2023-11-28T18:23:43.822203+0000","last_became_active":"2023-11-28T18:23:07.665310+0000","last_became_peered":"2023-11-28T18:23:07.665310+0000","last_unstale":"2023-11-28T18:23:43.822203+0000","last_undegraded":"2023-11-28T18:23:43.822203+0000","last_fullsized":"2023-11-28T18:23:43.822203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:03.174425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2052,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000928+0000","last_change":"2023-11-28T18:23:08.136812+0000","last_active":"2023-11-28T18:23:11.000928+0000","last_peered":"2023-11-28T18:23:11.000928+0000","last_clean":"2023-11-28T18:23:11.000928+0000","last_became_active":"2023-11-28T18:23:08.136525+0000","last_became_peered":"2023-11-28T18:23:08.136525+0000","last_unstale":"2023-11-28T18:23:11.000928+0000","last_undegraded":"2023-11-28T18:23:11.000928+0000","last_fullsized":"2023-11-28T18:23:11.000928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:07.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.249898+0000","last_change":"2023-11-28T18:23:07.660519+0000","last_active":"2023-11-28T18:23:35.249898+0000","last_peered":"2023-11-28T18:23:35.249898+0000","last_clean":"2023-11-28T18:23:35.249898+0000","last_became_active":"2023-11-28T18:23:07.660205+0000","last_became_peered":"2023-11-28T18:23:07.660205+0000","last_unstale":"2023-11-28T18:23:35.249898+0000","last_undegraded":"2023-11-28T18:23:35.249898+0000","last_fullsized":"2023-11-28T18:23:35.249898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:04:54.125737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914264+0000","last_change":"2023-11-28T18:23:08.127921+0000","last_active":"2023-11-28T18:23:10.914264+0000","last_peered":"2023-11-28T18:23:10.914264+0000","last_clean":"2023-11-28T18:23:10.914264+0000","last_became_active":"2023-11-28T18:23:08.127678+0000","last_became_peered":"2023-11-28T18:23:08.127678+0000","last_unstale":"2023-11-28T18:23:10.914264+0000","last_undegraded":"2023-11-28T18:23:10.914264+0000","last_fullsized":"2023-11-28T18:23:10.914264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:38:47.175911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171696+0000","last_change":"2023-11-28T18:23:07.671589+0000","last_active":"2023-11-28T18:23:30.171696+0000","last_peered":"2023-11-28T18:23:30.171696+0000","last_clean":"2023-11-28T18:23:30.171696+0000","last_became_active":"2023-11-28T18:23:07.671291+0000","last_became_peered":"2023-11-28T18:23:07.671291+0000","last_unstale":"2023-11-28T18:23:30.171696+0000","last_undegraded":"2023-11-28T18:23:30.171696+0000","last_fullsized":"2023-11-28T18:23:30.171696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:23.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000866+0000","last_change":"2023-11-28T18:23:08.141316+0000","last_active":"2023-11-28T18:23:11.000866+0000","last_peered":"2023-11-28T18:23:11.000866+0000","last_clean":"2023-11-28T18:23:11.000866+0000","last_became_active":"2023-11-28T18:23:08.141011+0000","last_became_peered":"2023-11-28T18:23:08.141011+0000","last_unstale":"2023-11-28T18:23:11.000866+0000","last_undegraded":"2023-11-28T18:23:11.000866+0000","last_fullsized":"2023-11-28T18:23:11.000866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:13:25.849340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172815+0000","last_change":"2023-11-28T18:23:07.671192+0000","last_active":"2023-11-28T18:23:30.172815+0000","last_peered":"2023-11-28T18:23:30.172815+0000","last_clean":"2023-11-28T18:23:30.172815+0000","last_became_active":"2023-11-28T18:23:07.670812+0000","last_became_peered":"2023-11-28T18:23:07.670812+0000","last_unstale":"2023-11-28T18:23:30.172815+0000","last_undegraded":"2023-11-28T18:23:30.172815+0000","last_fullsized":"2023-11-28T18:23:30.172815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:28.806061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109506+0000","last_change":"2023-11-28T18:23:08.115229+0000","last_active":"2023-11-28T18:23:09.109506+0000","last_peered":"2023-11-28T18:23:09.109506+0000","last_clean":"2023-11-28T18:23:09.109506+0000","last_became_active":"2023-11-28T18:23:08.114541+0000","last_became_peered":"2023-11-28T18:23:08.114541+0000","last_unstale":"2023-11-28T18:23:09.109506+0000","last_undegraded":"2023-11-28T18:23:09.109506+0000","last_fullsized":"2023-11-28T18:23:09.109506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:38.816731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":29,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.171599+0000","last_change":"2023-11-28T18:23:07.673554+0000","last_active":"2023-11-28T18:23:30.171599+0000","last_peered":"2023-11-28T18:23:30.171599+0000","last_clean":"2023-11-28T18:23:30.171599+0000","last_became_active":"2023-11-28T18:23:07.672923+0000","last_became_peered":"2023-11-28T18:23:07.672923+0000","last_unstale":"2023-11-28T18:23:30.171599+0000","last_undegraded":"2023-11-28T18:23:30.171599+0000","last_fullsized":"2023-11-28T18:23:30.171599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:49.805572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.000699+0000","last_change":"2023-11-28T18:23:08.141169+0000","last_active":"2023-11-28T18:23:11.000699+0000","last_peered":"2023-11-28T18:23:11.000699+0000","last_clean":"2023-11-28T18:23:11.000699+0000","last_became_active":"2023-11-28T18:23:08.140801+0000","last_became_peered":"2023-11-28T18:23:08.140801+0000","last_unstale":"2023-11-28T18:23:11.000699+0000","last_undegraded":"2023-11-28T18:23:11.000699+0000","last_fullsized":"2023-11-28T18:23:11.000699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:16.669717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.667707+0000","last_change":"2023-11-28T18:23:07.680094+0000","last_active":"2023-11-28T18:23:13.667707+0000","last_peered":"2023-11-28T18:23:13.667707+0000","last_clean":"2023-11-28T18:23:13.667707+0000","last_became_active":"2023-11-28T18:23:07.679801+0000","last_became_peered":"2023-11-28T18:23:07.679801+0000","last_unstale":"2023-11-28T18:23:13.667707+0000","last_undegraded":"2023-11-28T18:23:13.667707+0000","last_fullsized":"2023-11-28T18:23:13.667707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:02.916155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001208+0000","last_change":"2023-11-28T18:23:08.137825+0000","last_active":"2023-11-28T18:23:11.001208+0000","last_peered":"2023-11-28T18:23:11.001208+0000","last_clean":"2023-11-28T18:23:11.001208+0000","last_became_active":"2023-11-28T18:23:08.137499+0000","last_became_peered":"2023-11-28T18:23:08.137499+0000","last_unstale":"2023-11-28T18:23:11.001208+0000","last_undegraded":"2023-11-28T18:23:11.001208+0000","last_fullsized":"2023-11-28T18:23:11.001208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:42.807830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.244409+0000","last_change":"2023-11-28T18:23:07.670359+0000","last_active":"2023-11-28T18:23:35.244409+0000","last_peered":"2023-11-28T18:23:35.244409+0000","last_clean":"2023-11-28T18:23:35.244409+0000","last_became_active":"2023-11-28T18:23:07.669687+0000","last_became_peered":"2023-11-28T18:23:07.669687+0000","last_unstale":"2023-11-28T18:23:35.244409+0000","last_undegraded":"2023-11-28T18:23:35.244409+0000","last_fullsized":"2023-11-28T18:23:35.244409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:45.450623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001353+0000","last_change":"2023-11-28T18:23:08.141300+0000","last_active":"2023-11-28T18:23:11.001353+0000","last_peered":"2023-11-28T18:23:11.001353+0000","last_clean":"2023-11-28T18:23:11.001353+0000","last_became_active":"2023-11-28T18:23:08.141037+0000","last_became_peered":"2023-11-28T18:23:08.141037+0000","last_unstale":"2023-11-28T18:23:11.001353+0000","last_undegraded":"2023-11-28T18:23:11.001353+0000","last_fullsized":"2023-11-28T18:23:11.001353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:11.093903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172901+0000","last_change":"2023-11-28T18:23:07.673086+0000","last_active":"2023-11-28T18:23:30.172901+0000","last_peered":"2023-11-28T18:23:30.172901+0000","last_clean":"2023-11-28T18:23:30.172901+0000","last_became_active":"2023-11-28T18:23:07.672773+0000","last_became_peered":"2023-11-28T18:23:07.672773+0000","last_unstale":"2023-11-28T18:23:30.172901+0000","last_undegraded":"2023-11-28T18:23:30.172901+0000","last_fullsized":"2023-11-28T18:23:30.172901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:34.962428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113586+0000","last_change":"2023-11-28T18:23:08.121987+0000","last_active":"2023-11-28T18:23:09.113586+0000","last_peered":"2023-11-28T18:23:09.113586+0000","last_clean":"2023-11-28T18:23:09.113586+0000","last_became_active":"2023-11-28T18:23:08.121279+0000","last_became_peered":"2023-11-28T18:23:08.121279+0000","last_unstale":"2023-11-28T18:23:09.113586+0000","last_undegraded":"2023-11-28T18:23:09.113586+0000","last_fullsized":"2023-11-28T18:23:09.113586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:47.973973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.174488+0000","last_change":"2023-11-28T18:23:07.681312+0000","last_active":"2023-11-28T18:23:30.174488+0000","last_peered":"2023-11-28T18:23:30.174488+0000","last_clean":"2023-11-28T18:23:30.174488+0000","last_became_active":"2023-11-28T18:23:07.680806+0000","last_became_peered":"2023-11-28T18:23:07.680806+0000","last_unstale":"2023-11-28T18:23:30.174488+0000","last_undegraded":"2023-11-28T18:23:30.174488+0000","last_fullsized":"2023-11-28T18:23:30.174488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:58:41.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:11.001014+0000","last_change":"2023-11-28T18:23:08.137874+0000","last_active":"2023-11-28T18:23:11.001014+0000","last_peered":"2023-11-28T18:23:11.001014+0000","last_clean":"2023-11-28T18:23:11.001014+0000","last_became_active":"2023-11-28T18:23:08.137104+0000","last_became_peered":"2023-11-28T18:23:08.137104+0000","last_unstale":"2023-11-28T18:23:11.001014+0000","last_undegraded":"2023-11-28T18:23:11.001014+0000","last_fullsized":"2023-11-28T18:23:11.001014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:39:11.857836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T18:23:08.117053+0000","last_change":"2023-11-28T18:22:56.774057+0000","last_active":"2023-11-28T18:23:08.117053+0000","last_peered":"2023-11-28T18:23:08.117053+0000","last_clean":"2023-11-28T18:23:08.117053+0000","last_became_active":"2023-11-28T18:22:56.772763+0000","last_became_peered":"2023-11-28T18:22:56.772763+0000","last_unstale":"2023-11-28T18:23:08.117053+0000","last_undegraded":"2023-11-28T18:23:08.117053+0000","last_fullsized":"2023-11-28T18:23:08.117053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:22:55.591288+0000","last_clean_scrub_stamp":"2023-11-28T18:22:55.591288+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:05.470408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.172368+0000","last_change":"2023-11-28T18:23:07.674072+0000","last_active":"2023-11-28T18:23:30.172368+0000","last_peered":"2023-11-28T18:23:30.172368+0000","last_clean":"2023-11-28T18:23:30.172368+0000","last_became_active":"2023-11-28T18:23:07.673220+0000","last_became_peered":"2023-11-28T18:23:07.673220+0000","last_unstale":"2023-11-28T18:23:30.172368+0000","last_undegraded":"2023-11-28T18:23:30.172368+0000","last_fullsized":"2023-11-28T18:23:30.172368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:58.524112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565598+0000","last_change":"2023-11-28T18:23:08.130989+0000","last_active":"2023-11-28T18:23:09.565598+0000","last_peered":"2023-11-28T18:23:09.565598+0000","last_clean":"2023-11-28T18:23:09.565598+0000","last_became_active":"2023-11-28T18:23:08.130646+0000","last_became_peered":"2023-11-28T18:23:08.130646+0000","last_unstale":"2023-11-28T18:23:09.565598+0000","last_undegraded":"2023-11-28T18:23:09.565598+0000","last_fullsized":"2023-11-28T18:23:09.565598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:50.371045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:24.150194+0000","last_change":"2023-11-28T18:23:07.686046+0000","last_active":"2023-11-28T18:23:24.150194+0000","last_peered":"2023-11-28T18:23:24.150194+0000","last_clean":"2023-11-28T18:23:24.150194+0000","last_became_active":"2023-11-28T18:23:07.685739+0000","last_became_peered":"2023-11-28T18:23:07.685739+0000","last_unstale":"2023-11-28T18:23:24.150194+0000","last_undegraded":"2023-11-28T18:23:24.150194+0000","last_fullsized":"2023-11-28T18:23:24.150194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:06.485631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.913951+0000","last_change":"2023-11-28T18:23:08.130355+0000","last_active":"2023-11-28T18:23:10.913951+0000","last_peered":"2023-11-28T18:23:10.913951+0000","last_clean":"2023-11-28T18:23:10.913951+0000","last_became_active":"2023-11-28T18:23:08.129946+0000","last_became_peered":"2023-11-28T18:23:08.129946+0000","last_unstale":"2023-11-28T18:23:10.913951+0000","last_undegraded":"2023-11-28T18:23:10.913951+0000","last_fullsized":"2023-11-28T18:23:10.913951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:58.140191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.245973+0000","last_change":"2023-11-28T18:23:07.664827+0000","last_active":"2023-11-28T18:23:35.245973+0000","last_peered":"2023-11-28T18:23:35.245973+0000","last_clean":"2023-11-28T18:23:35.245973+0000","last_became_active":"2023-11-28T18:23:07.664503+0000","last_became_peered":"2023-11-28T18:23:07.664503+0000","last_unstale":"2023-11-28T18:23:35.245973+0000","last_undegraded":"2023-11-28T18:23:35.245973+0000","last_fullsized":"2023-11-28T18:23:35.245973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.238694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105529+0000","last_change":"2023-11-28T18:23:08.123498+0000","last_active":"2023-11-28T18:23:09.105529+0000","last_peered":"2023-11-28T18:23:09.105529+0000","last_clean":"2023-11-28T18:23:09.105529+0000","last_became_active":"2023-11-28T18:23:08.123197+0000","last_became_peered":"2023-11-28T18:23:08.123197+0000","last_unstale":"2023-11-28T18:23:09.105529+0000","last_undegraded":"2023-11-28T18:23:09.105529+0000","last_fullsized":"2023-11-28T18:23:09.105529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:08:42.817848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250563+0000","last_change":"2023-11-28T18:23:07.673345+0000","last_active":"2023-11-28T18:23:35.250563+0000","last_peered":"2023-11-28T18:23:35.250563+0000","last_clean":"2023-11-28T18:23:35.250563+0000","last_became_active":"2023-11-28T18:23:07.672982+0000","last_became_peered":"2023-11-28T18:23:07.672982+0000","last_unstale":"2023-11-28T18:23:35.250563+0000","last_undegraded":"2023-11-28T18:23:35.250563+0000","last_fullsized":"2023-11-28T18:23:35.250563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:26.594232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637566+0000","last_change":"2023-11-28T18:23:08.129091+0000","last_active":"2023-11-28T18:23:09.637566+0000","last_peered":"2023-11-28T18:23:09.637566+0000","last_clean":"2023-11-28T18:23:09.637566+0000","last_became_active":"2023-11-28T18:23:08.128589+0000","last_became_peered":"2023-11-28T18:23:08.128589+0000","last_unstale":"2023-11-28T18:23:09.637566+0000","last_undegraded":"2023-11-28T18:23:09.637566+0000","last_fullsized":"2023-11-28T18:23:09.637566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:28.975412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.493169+0000","last_change":"2023-11-28T18:23:07.671874+0000","last_active":"2023-11-28T18:23:09.493169+0000","last_peered":"2023-11-28T18:23:09.493169+0000","last_clean":"2023-11-28T18:23:09.493169+0000","last_became_active":"2023-11-28T18:23:07.671500+0000","last_became_peered":"2023-11-28T18:23:07.671500+0000","last_unstale":"2023-11-28T18:23:09.493169+0000","last_undegraded":"2023-11-28T18:23:09.493169+0000","last_fullsized":"2023-11-28T18:23:09.493169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:42.922770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113612+0000","last_change":"2023-11-28T18:23:08.127419+0000","last_active":"2023-11-28T18:23:09.113612+0000","last_peered":"2023-11-28T18:23:09.113612+0000","last_clean":"2023-11-28T18:23:09.113612+0000","last_became_active":"2023-11-28T18:23:08.127157+0000","last_became_peered":"2023-11-28T18:23:08.127157+0000","last_unstale":"2023-11-28T18:23:09.113612+0000","last_undegraded":"2023-11-28T18:23:09.113612+0000","last_fullsized":"2023-11-28T18:23:09.113612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:08.473882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.173374+0000","last_change":"2023-11-28T18:23:07.684725+0000","last_active":"2023-11-28T18:23:30.173374+0000","last_peered":"2023-11-28T18:23:30.173374+0000","last_clean":"2023-11-28T18:23:30.173374+0000","last_became_active":"2023-11-28T18:23:07.684363+0000","last_became_peered":"2023-11-28T18:23:07.684363+0000","last_unstale":"2023-11-28T18:23:30.173374+0000","last_undegraded":"2023-11-28T18:23:30.173374+0000","last_fullsized":"2023-11-28T18:23:30.173374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:16.380789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.105690+0000","last_change":"2023-11-28T18:23:08.120741+0000","last_active":"2023-11-28T18:23:09.105690+0000","last_peered":"2023-11-28T18:23:09.105690+0000","last_clean":"2023-11-28T18:23:09.105690+0000","last_became_active":"2023-11-28T18:23:08.120421+0000","last_became_peered":"2023-11-28T18:23:08.120421+0000","last_unstale":"2023-11-28T18:23:09.105690+0000","last_undegraded":"2023-11-28T18:23:09.105690+0000","last_fullsized":"2023-11-28T18:23:09.105690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:53:27.131631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.175382+0000","last_change":"2023-11-28T18:23:07.672505+0000","last_active":"2023-11-28T18:23:30.175382+0000","last_peered":"2023-11-28T18:23:30.175382+0000","last_clean":"2023-11-28T18:23:30.175382+0000","last_became_active":"2023-11-28T18:23:07.672204+0000","last_became_peered":"2023-11-28T18:23:07.672204+0000","last_unstale":"2023-11-28T18:23:30.175382+0000","last_undegraded":"2023-11-28T18:23:30.175382+0000","last_fullsized":"2023-11-28T18:23:30.175382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:01.341643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565869+0000","last_change":"2023-11-28T18:23:08.126198+0000","last_active":"2023-11-28T18:23:09.565869+0000","last_peered":"2023-11-28T18:23:09.565869+0000","last_clean":"2023-11-28T18:23:09.565869+0000","last_became_active":"2023-11-28T18:23:08.125935+0000","last_became_peered":"2023-11-28T18:23:08.125935+0000","last_unstale":"2023-11-28T18:23:09.565869+0000","last_undegraded":"2023-11-28T18:23:09.565869+0000","last_fullsized":"2023-11-28T18:23:09.565869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.473922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.252470+0000","last_change":"2023-11-28T18:23:07.672170+0000","last_active":"2023-11-28T18:23:35.252470+0000","last_peered":"2023-11-28T18:23:35.252470+0000","last_clean":"2023-11-28T18:23:35.252470+0000","last_became_active":"2023-11-28T18:23:07.671683+0000","last_became_peered":"2023-11-28T18:23:07.671683+0000","last_unstale":"2023-11-28T18:23:35.252470+0000","last_undegraded":"2023-11-28T18:23:35.252470+0000","last_fullsized":"2023-11-28T18:23:35.252470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:41.860341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914424+0000","last_change":"2023-11-28T18:23:08.119844+0000","last_active":"2023-11-28T18:23:10.914424+0000","last_peered":"2023-11-28T18:23:10.914424+0000","last_clean":"2023-11-28T18:23:10.914424+0000","last_became_active":"2023-11-28T18:23:08.119525+0000","last_became_peered":"2023-11-28T18:23:08.119525+0000","last_unstale":"2023-11-28T18:23:10.914424+0000","last_undegraded":"2023-11-28T18:23:10.914424+0000","last_fullsized":"2023-11-28T18:23:10.914424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:34:00.230528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.727126+0000","last_change":"2023-11-28T18:23:07.685651+0000","last_active":"2023-11-28T18:23:18.727126+0000","last_peered":"2023-11-28T18:23:18.727126+0000","last_clean":"2023-11-28T18:23:18.727126+0000","last_became_active":"2023-11-28T18:23:07.683869+0000","last_became_peered":"2023-11-28T18:23:07.683869+0000","last_unstale":"2023-11-28T18:23:18.727126+0000","last_undegraded":"2023-11-28T18:23:18.727126+0000","last_fullsized":"2023-11-28T18:23:18.727126+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:35.383852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.637574+0000","last_change":"2023-11-28T18:23:08.130016+0000","last_active":"2023-11-28T18:23:09.637574+0000","last_peered":"2023-11-28T18:23:09.637574+0000","last_clean":"2023-11-28T18:23:09.637574+0000","last_became_active":"2023-11-28T18:23:08.129586+0000","last_became_peered":"2023-11-28T18:23:08.129586+0000","last_unstale":"2023-11-28T18:23:09.637574+0000","last_undegraded":"2023-11-28T18:23:09.637574+0000","last_fullsized":"2023-11-28T18:23:09.637574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:00.449941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.245267+0000","last_change":"2023-11-28T18:23:07.671773+0000","last_active":"2023-11-28T18:23:35.245267+0000","last_peered":"2023-11-28T18:23:35.245267+0000","last_clean":"2023-11-28T18:23:35.245267+0000","last_became_active":"2023-11-28T18:23:07.671465+0000","last_became_peered":"2023-11-28T18:23:07.671465+0000","last_unstale":"2023-11-28T18:23:35.245267+0000","last_undegraded":"2023-11-28T18:23:35.245267+0000","last_fullsized":"2023-11-28T18:23:35.245267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:44.862780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565865+0000","last_change":"2023-11-28T18:23:08.118146+0000","last_active":"2023-11-28T18:23:09.565865+0000","last_peered":"2023-11-28T18:23:09.565865+0000","last_clean":"2023-11-28T18:23:09.565865+0000","last_became_active":"2023-11-28T18:23:08.117848+0000","last_became_peered":"2023-11-28T18:23:08.117848+0000","last_unstale":"2023-11-28T18:23:09.565865+0000","last_undegraded":"2023-11-28T18:23:09.565865+0000","last_fullsized":"2023-11-28T18:23:09.565865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:33:20.745184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:30.169912+0000","last_change":"2023-11-28T18:23:07.676709+0000","last_active":"2023-11-28T18:23:30.169912+0000","last_peered":"2023-11-28T18:23:30.169912+0000","last_clean":"2023-11-28T18:23:30.169912+0000","last_became_active":"2023-11-28T18:23:07.676414+0000","last_became_peered":"2023-11-28T18:23:07.676414+0000","last_unstale":"2023-11-28T18:23:30.169912+0000","last_undegraded":"2023-11-28T18:23:30.169912+0000","last_fullsized":"2023-11-28T18:23:30.169912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:58.282005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565753+0000","last_change":"2023-11-28T18:23:08.112410+0000","last_active":"2023-11-28T18:23:09.565753+0000","last_peered":"2023-11-28T18:23:09.565753+0000","last_clean":"2023-11-28T18:23:09.565753+0000","last_became_active":"2023-11-28T18:23:08.112097+0000","last_became_peered":"2023-11-28T18:23:08.112097+0000","last_unstale":"2023-11-28T18:23:09.565753+0000","last_undegraded":"2023-11-28T18:23:09.565753+0000","last_fullsized":"2023-11-28T18:23:09.565753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:57.891789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:18.719092+0000","last_change":"2023-11-28T18:23:07.684818+0000","last_active":"2023-11-28T18:23:18.719092+0000","last_peered":"2023-11-28T18:23:18.719092+0000","last_clean":"2023-11-28T18:23:18.719092+0000","last_became_active":"2023-11-28T18:23:07.684502+0000","last_became_peered":"2023-11-28T18:23:07.684502+0000","last_unstale":"2023-11-28T18:23:18.719092+0000","last_undegraded":"2023-11-28T18:23:18.719092+0000","last_fullsized":"2023-11-28T18:23:18.719092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:16:21.223266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.109527+0000","last_change":"2023-11-28T18:23:08.133972+0000","last_active":"2023-11-28T18:23:09.109527+0000","last_peered":"2023-11-28T18:23:09.109527+0000","last_clean":"2023-11-28T18:23:09.109527+0000","last_became_active":"2023-11-28T18:23:08.133747+0000","last_became_peered":"2023-11-28T18:23:08.133747+0000","last_unstale":"2023-11-28T18:23:09.109527+0000","last_undegraded":"2023-11-28T18:23:09.109527+0000","last_fullsized":"2023-11-28T18:23:09.109527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:24.968241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113538+0000","last_change":"2023-11-28T18:23:08.134117+0000","last_active":"2023-11-28T18:23:09.113538+0000","last_peered":"2023-11-28T18:23:09.113538+0000","last_clean":"2023-11-28T18:23:09.113538+0000","last_became_active":"2023-11-28T18:23:08.133801+0000","last_became_peered":"2023-11-28T18:23:08.133801+0000","last_unstale":"2023-11-28T18:23:09.113538+0000","last_undegraded":"2023-11-28T18:23:09.113538+0000","last_fullsized":"2023-11-28T18:23:09.113538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:24:02.370862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250642+0000","last_change":"2023-11-28T18:23:07.686399+0000","last_active":"2023-11-28T18:23:35.250642+0000","last_peered":"2023-11-28T18:23:35.250642+0000","last_clean":"2023-11-28T18:23:35.250642+0000","last_became_active":"2023-11-28T18:23:07.686051+0000","last_became_peered":"2023-11-28T18:23:07.686051+0000","last_unstale":"2023-11-28T18:23:35.250642+0000","last_undegraded":"2023-11-28T18:23:35.250642+0000","last_fullsized":"2023-11-28T18:23:35.250642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:11.797516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.113317+0000","last_change":"2023-11-28T18:23:08.134870+0000","last_active":"2023-11-28T18:23:09.113317+0000","last_peered":"2023-11-28T18:23:09.113317+0000","last_clean":"2023-11-28T18:23:09.113317+0000","last_became_active":"2023-11-28T18:23:08.133732+0000","last_became_peered":"2023-11-28T18:23:08.133732+0000","last_unstale":"2023-11-28T18:23:09.113317+0000","last_undegraded":"2023-11-28T18:23:09.113317+0000","last_fullsized":"2023-11-28T18:23:09.113317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:01.364242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.252608+0000","last_change":"2023-11-28T18:23:07.682230+0000","last_active":"2023-11-28T18:23:35.252608+0000","last_peered":"2023-11-28T18:23:35.252608+0000","last_clean":"2023-11-28T18:23:35.252608+0000","last_became_active":"2023-11-28T18:23:07.681715+0000","last_became_peered":"2023-11-28T18:23:07.681715+0000","last_unstale":"2023-11-28T18:23:35.252608+0000","last_undegraded":"2023-11-28T18:23:35.252608+0000","last_fullsized":"2023-11-28T18:23:35.252608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:02.343996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:10.914216+0000","last_change":"2023-11-28T18:23:08.128927+0000","last_active":"2023-11-28T18:23:10.914216+0000","last_peered":"2023-11-28T18:23:10.914216+0000","last_clean":"2023-11-28T18:23:10.914216+0000","last_became_active":"2023-11-28T18:23:08.127775+0000","last_became_peered":"2023-11-28T18:23:08.127775+0000","last_unstale":"2023-11-28T18:23:10.914216+0000","last_undegraded":"2023-11-28T18:23:10.914216+0000","last_fullsized":"2023-11-28T18:23:10.914216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:50:05.531589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:13.669760+0000","last_change":"2023-11-28T18:23:07.667929+0000","last_active":"2023-11-28T18:23:13.669760+0000","last_peered":"2023-11-28T18:23:13.669760+0000","last_clean":"2023-11-28T18:23:13.669760+0000","last_became_active":"2023-11-28T18:23:07.667666+0000","last_became_peered":"2023-11-28T18:23:07.667666+0000","last_unstale":"2023-11-28T18:23:13.669760+0000","last_undegraded":"2023-11-28T18:23:13.669760+0000","last_fullsized":"2023-11-28T18:23:13.669760+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:32.132092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:09.565609+0000","last_change":"2023-11-28T18:23:08.126404+0000","last_active":"2023-11-28T18:23:09.565609+0000","last_peered":"2023-11-28T18:23:09.565609+0000","last_clean":"2023-11-28T18:23:09.565609+0000","last_became_active":"2023-11-28T18:23:08.125315+0000","last_became_peered":"2023-11-28T18:23:08.125315+0000","last_unstale":"2023-11-28T18:23:09.565609+0000","last_undegraded":"2023-11-28T18:23:09.565609+0000","last_fullsized":"2023-11-28T18:23:09.565609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:07.640407+0000","last_clean_scrub_stamp":"2023-11-28T18:23:07.640407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:24:59.527642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":25,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T18:23:35.250692+0000","last_change":"2023-11-28T18:23:07.677758+0000","last_active":"2023-11-28T18:23:35.250692+0000","last_peered":"2023-11-28T18:23:35.250692+0000","last_clean":"2023-11-28T18:23:35.250692+0000","last_became_active":"2023-11-28T18:23:07.676902+0000","last_became_peered":"2023-11-28T18:23:07.676902+0000","last_unstale":"2023-11-28T18:23:35.250692+0000","last_undegraded":"2023-11-28T18:23:35.250692+0000","last_fullsized":"2023-11-28T18:23:35.250692+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T18:23:06.633326+0000","last_clean_scrub_stamp":"2023-11-28T18:23:06.633326+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:50.953647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9794,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":835584,"data_stored":663818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":74192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":74842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":72240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":73346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":72812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":233472,"data_stored":72442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":827392,"data_stored":662152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T18:23:50.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:50.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.687+0000 7fef893f8640 1 Processor -- start 2023-11-28T18:23:50.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.687+0000 7fef893f8640 1 -- start start 2023-11-28T18:23:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.687+0000 7fef893f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef840ca4d0 0x7fef8415a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef893f8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef8415e380 con 0x7fef841540e0 2023-11-28T18:23:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef893f8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef8415d7e0 con 0x7fef840cf6c0 2023-11-28T18:23:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef893f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8415ade0 con 0x7fef840ca4d0 2023-11-28T18:23:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef82ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fef841540e0 0x7fef8415dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57482/0 (socket says 172.21.15.7:57482) 2023-11-28T18:23:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef82ffd640 1 -- 172.21.15.7:0/3451131195 learned_addr learned my addr 172.21.15.7:0/3451131195 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:50.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef7bfff640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef840ca4d0 0x7fef8415a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820120629 0 0) 0x7fef74001120 con 0x7fef841540e0 2023-11-28T18:23:50.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef70003530 con 0x7fef841540e0 2023-11-28T18:23:50.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348212856 0 0) 0x7fef70003530 con 0x7fef841540e0 2023-11-28T18:23:50.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef840ca4d0 msgr2=0x7fef8415a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef840ca4d0 0x7fef8415a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 >> v1:172.21.15.193:6789/0 conn(0x7fef840cf6c0 legacy=0x7fef84164760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef840ca8f0 con 0x7fef841540e0 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fef74001810 con 0x7fef841540e0 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 >> v1:172.21.15.7:6789/0 conn(0x7fef841540e0 legacy=0x7fef8415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef700039b0 0x7fef70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700047d0 0x7fef70004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef840ca4d0 0x7fef70005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8415ade0 con 0x7fef700039b0 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7000ba40 con 0x7fef700047d0 2023-11-28T18:23:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7000bc00 con 0x7fef840ca4d0 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef82ffd640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef700039b0 0x7fef70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef7bfff640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700047d0 0x7fef70004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef837fe640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef840ca4d0 0x7fef70005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef82ffd640 1 -- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef840ca4d0 msgr2=0x7fef70005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef82ffd640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef840ca4d0 0x7fef70005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef82ffd640 1 -- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700047d0 msgr2=0x7fef70004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef82ffd640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700047d0 0x7fef70004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.691+0000 7fef82ffd640 1 -- 172.21.15.7:0/3451131195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef70003530 con 0x7fef700039b0 2023-11-28T18:23:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef82ffd640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef700039b0 0x7fef70003dd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fef7400c160 tx=0x7fef7401cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cc3584a26e02a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74005ae0 con 0x7fef700039b0 2023-11-28T18:23:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef74005c40 con 0x7fef700039b0 2023-11-28T18:23:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef827fc640 1 -- 172.21.15.7:0/3451131195 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74007070 con 0x7fef700039b0 2023-11-28T18:23:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef893f8640 1 -- 172.21.15.7:0/3451131195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef700039b0 msgr2=0x7fef70003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef893f8640 1 --2- 172.21.15.7:0/3451131195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef700039b0 0x7fef70003dd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fef7400c160 tx=0x7fef7401cbc0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef837fe640 1 -- 172.21.15.7:0/3451131195 reap_dead start 2023-11-28T18:23:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef893f8640 1 -- 172.21.15.7:0/3451131195 shutdown_connections 2023-11-28T18:23:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef893f8640 1 -- 172.21.15.7:0/3451131195 >> 172.21.15.7:0/3451131195 conn(0x7fef840662e0 msgr2=0x7fef840666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef893f8640 1 -- 172.21.15.7:0/3451131195 shutdown_connections 2023-11-28T18:23:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef893f8640 1 -- 172.21.15.7:0/3451131195 wait complete. 2023-11-28T18:23:50.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef893f8640 1 Processor -- start 2023-11-28T18:23:50.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.695+0000 7fef893f8640 1 -- start start 2023-11-28T18:23:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700039b0 0x7fef84159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef841548b0 0x7fef841f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef841f7380 0x7fef84149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef84165cd0 con 0x7fef841f7380 2023-11-28T18:23:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef841f66d0 con 0x7fef700039b0 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef841f69b0 con 0x7fef841548b0 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef841f7380 0x7fef84149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef841f7380 0x7fef84149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47546/0 (socket says 172.21.15.7:47546) 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 -- 172.21.15.7:0/2675910210 learned_addr learned my addr 172.21.15.7:0/2675910210 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef82ffd640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700039b0 0x7fef84159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef7bfff640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef841548b0 0x7fef841f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 -- 172.21.15.7:0/2675910210 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef841548b0 msgr2=0x7fef841f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef841548b0 0x7fef841f6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 -- 172.21.15.7:0/2675910210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700039b0 msgr2=0x7fef84159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700039b0 0x7fef84159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 -- 172.21.15.7:0/2675910210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef840b37c0 con 0x7fef841f7380 2023-11-28T18:23:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef837fe640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef841f7380 0x7fef84149f20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fef840cb550 tx=0x7fef7c00fcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:50.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef7b7fe640 1 -- 172.21.15.7:0/2675910210 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c006d70 con 0x7fef841f7380 2023-11-28T18:23:50.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef7b7fe640 1 -- 172.21.15.7:0/2675910210 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c003980 con 0x7fef841f7380 2023-11-28T18:23:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef7b7fe640 1 -- 172.21.15.7:0/2675910210 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c003d30 con 0x7fef841f7380 2023-11-28T18:23:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8415c430 con 0x7fef841f7380 2023-11-28T18:23:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8415c9f0 con 0x7fef841f7380 2023-11-28T18:23:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef8404e890 con 0x7fef841f7380 2023-11-28T18:23:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.699+0000 7fef7b7fe640 1 -- 172.21.15.7:0/2675910210 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fef7c005110 con 0x7fef841f7380 2023-11-28T18:23:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.703+0000 7fef7b7fe640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef64066920 0x7fef64068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.703+0000 7fef7b7fe640 1 -- 172.21.15.7:0/2675910210 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fef7c083d80 con 0x7fef841f7380 2023-11-28T18:23:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.703+0000 7fef82ffd640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef64066920 0x7fef64068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.703+0000 7fef82ffd640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef64066920 0x7fef64068de0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef740ab000 tx=0x7fef740a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.707+0000 7fef7b7fe640 1 -- 172.21.15.7:0/2675910210 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fef7c0510b0 con 0x7fef841f7380 2023-11-28T18:23:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.863+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef841f6af0 con 0x7fef841f7380 2023-11-28T18:23:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.863+0000 7fef7b7fe640 1 -- 172.21.15.7:0/2675910210 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16694 (secure 0 0 0) 0x7fef7c085090 con 0x7fef841f7380 2023-11-28T18:23:50.871 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:50.871 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:49.291754+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef64066920 msgr2=0x7fef64068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef64066920 0x7fef64068de0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef740ab000 tx=0x7fef740a7070 comp rx=0 tx=0).stop 2023-11-28T18:23:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef841f7380 msgr2=0x7fef84149f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef841f7380 0x7fef84149f20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fef840cb550 tx=0x7fef7c00fcd0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 shutdown_connections 2023-11-28T18:23:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef64066920 0x7fef64068de0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef841f7380 0x7fef84149f20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef841548b0 0x7fef841f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 --2- 172.21.15.7:0/2675910210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef700039b0 0x7fef84159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 >> 172.21.15.7:0/2675910210 conn(0x7fef840662e0 msgr2=0x7fef84152900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 shutdown_connections 2023-11-28T18:23:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:50.867+0000 7fef893f8640 1 -- 172.21.15.7:0/2675910210 wait complete. 2023-11-28T18:23:50.892 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T18:23:05.706694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T18:23:07.048910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:23:50.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:23:51.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.031+0000 7ff2074ea640 1 Processor -- start 2023-11-28T18:23:51.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.031+0000 7ff2074ea640 1 -- start start 2023-11-28T18:23:51.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff2074ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff200150e50 0x7ff2000c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff2074ea640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff200157530 0x7ff2000c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff2074ea640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff20006a5f0 0x7ff20006aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff2074ea640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2000c7120 con 0x7ff20006a5f0 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff2074ea640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20015ad50 con 0x7ff200157530 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff2074ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff200151230 con 0x7ff200150e50 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205a87640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff20006a5f0 0x7ff20006aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205a87640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff20006a5f0 0x7ff20006aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47558/0 (socket says 172.21.15.7:47558) 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205a87640 1 -- 172.21.15.7:0/1353205154 learned_addr learned my addr 172.21.15.7:0/1353205154 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205286640 1 --2- 172.21.15.7:0/1353205154 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff200150e50 0x7ff2000c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff204a85640 1 --2- 172.21.15.7:0/1353205154 >> v2:172.21.15.193:3300/0 conn(0x7ff200157530 0x7ff2000c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205a87640 1 -- 172.21.15.7:0/1353205154 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff200150e50 msgr2=0x7ff2000c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205a87640 1 --2- 172.21.15.7:0/1353205154 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff200150e50 0x7ff2000c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205a87640 1 -- 172.21.15.7:0/1353205154 >> v2:172.21.15.193:3300/0 conn(0x7ff200157530 msgr2=0x7ff2000c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:51.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205a87640 1 --2- 172.21.15.7:0/1353205154 >> v2:172.21.15.193:3300/0 conn(0x7ff200157530 0x7ff2000c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff205a87640 1 -- 172.21.15.7:0/1353205154 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff200157a00 con 0x7ff20006a5f0 2023-11-28T18:23:51.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.035+0000 7ff204a85640 1 --2- 172.21.15.7:0/1353205154 >> v2:172.21.15.193:3300/0 conn(0x7ff200157530 0x7ff2000c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:23:51.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff205a87640 1 --2- 172.21.15.7:0/1353205154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff20006a5f0 0x7ff20006aaa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff20006c5e0 tx=0x7ff1f801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6585693a9da2f23 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff1f7fff640 1 -- 172.21.15.7:0/1353205154 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80af400 con 0x7ff20006a5f0 2023-11-28T18:23:51.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff1f7fff640 1 -- 172.21.15.7:0/1353205154 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f80ab050 con 0x7ff20006a5f0 2023-11-28T18:23:51.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff1f7fff640 1 -- 172.21.15.7:0/1353205154 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80af780 con 0x7ff20006a5f0 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- 172.21.15.7:0/1353205154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff20006a5f0 msgr2=0x7ff20006aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 --2- 172.21.15.7:0/1353205154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff20006a5f0 0x7ff20006aaa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff20006c5e0 tx=0x7ff1f801f1c0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- 172.21.15.7:0/1353205154 shutdown_connections 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 --2- 172.21.15.7:0/1353205154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff20006a5f0 0x7ff20006aaa0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 --2- 172.21.15.7:0/1353205154 >> v2:172.21.15.193:3300/0 conn(0x7ff200157530 0x7ff2000c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 --2- 172.21.15.7:0/1353205154 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff200150e50 0x7ff2000c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- 172.21.15.7:0/1353205154 >> 172.21.15.7:0/1353205154 conn(0x7ff2000cc490 msgr2=0x7ff2000cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- 172.21.15.7:0/1353205154 shutdown_connections 2023-11-28T18:23:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- 172.21.15.7:0/1353205154 wait complete. 2023-11-28T18:23:51.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 Processor -- start 2023-11-28T18:23:51.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- start start 2023-11-28T18:23:51.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff20006a5f0 0x7ff20014b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff200150e50 0x7ff20014de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200157530 0x7ff2001f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2001f2770 con 0x7ff200150e50 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20014c4e0 con 0x7ff200157530 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.039+0000 7ff2074ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20014c7c0 con 0x7ff20006a5f0 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff200150e50 0x7ff20014de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff200150e50 0x7ff20014de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47570/0 (socket says 172.21.15.7:47570) 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 -- 172.21.15.7:0/3877657635 learned_addr learned my addr 172.21.15.7:0/3877657635 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff205286640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff20006a5f0 0x7ff20014b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff205a87640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200157530 0x7ff2001f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 -- 172.21.15.7:0/3877657635 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff20006a5f0 msgr2=0x7ff20014b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff20006a5f0 0x7ff20014b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 -- 172.21.15.7:0/3877657635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200157530 msgr2=0x7ff2001f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200157530 0x7ff2001f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 -- 172.21.15.7:0/3877657635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f8009b30 con 0x7ff200150e50 2023-11-28T18:23:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff204a85640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff200150e50 0x7ff20014de20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000c7f0 tx=0x7ff1f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff1f5ffb640 1 -- 172.21.15.7:0/3877657635 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f000cea0 con 0x7ff200150e50 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff1f5ffb640 1 -- 172.21.15.7:0/3877657635 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f0007a30 con 0x7ff200150e50 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff1f5ffb640 1 -- 172.21.15.7:0/3877657635 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0007dc0 con 0x7ff200150e50 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20014caa0 con 0x7ff200150e50 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20014be80 con 0x7ff200150e50 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20014cfc0 con 0x7ff200150e50 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff1f5ffb640 1 -- 172.21.15.7:0/3877657635 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff1f00113f0 con 0x7ff200150e50 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.043+0000 7ff1f5ffb640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e4066920 0x7ff1e4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.047+0000 7ff1f5ffb640 1 -- 172.21.15.7:0/3877657635 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff1f0084620 con 0x7ff200150e50 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.047+0000 7ff205286640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e4066920 0x7ff1e4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.047+0000 7ff205286640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e4066920 0x7ff1e4068de0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0045f0 tx=0x7ff1fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.047+0000 7ff1f5ffb640 1 -- 172.21.15.7:0/3877657635 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff1f00517e0 con 0x7ff200150e50 2023-11-28T18:23:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.207+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff20004e890 con 0x7ff200150e50 2023-11-28T18:23:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.207+0000 7ff1f5ffb640 1 -- 172.21.15.7:0/3877657635 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1f0054e00 con 0x7ff200150e50 2023-11-28T18:23:51.213 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:23:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.207+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e4066920 msgr2=0x7ff1e4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.207+0000 7ff2074ea640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e4066920 0x7ff1e4068de0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0045f0 tx=0x7ff1fc008040 comp rx=0 tx=0).stop 2023-11-28T18:23:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.207+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff200150e50 msgr2=0x7ff20014de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.207+0000 7ff2074ea640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff200150e50 0x7ff20014de20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000c7f0 tx=0x7ff1f000ccb0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.211+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 shutdown_connections 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.211+0000 7ff2074ea640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e4066920 0x7ff1e4068de0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.211+0000 7ff2074ea640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200157530 0x7ff2001f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.211+0000 7ff2074ea640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff200150e50 0x7ff20014de20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.211+0000 7ff2074ea640 1 --2- 172.21.15.7:0/3877657635 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff20006a5f0 0x7ff20014b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.211+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 >> 172.21.15.7:0/3877657635 conn(0x7ff2000cc490 msgr2=0x7ff2001500b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.211+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 shutdown_connections 2023-11-28T18:23:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.211+0000 7ff2074ea640 1 -- 172.21.15.7:0/3877657635 wait complete. 2023-11-28T18:23:51.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T18:23:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.375+0000 7ff7ae1d1640 1 Processor -- start 2023-11-28T18:23:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.375+0000 7ff7ae1d1640 1 -- start start 2023-11-28T18:23:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7ae1d1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a815db70 0x7ff7a815df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7ae1d1640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff7a80cd560 0x7ff7a80cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7ae1d1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a815ae90 con 0x7ff7a81540d0 2023-11-28T18:23:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7ae1d1640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a815e490 con 0x7ff7a80cd560 2023-11-28T18:23:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7ae1d1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a815a370 con 0x7ff7a815db70 2023-11-28T18:23:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a77fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff7a81540d0 0x7ff7a8072940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57486/0 (socket says 172.21.15.7:57486) 2023-11-28T18:23:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a77fe640 1 -- 172.21.15.7:0/935667689 learned_addr learned my addr 172.21.15.7:0/935667689 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a7fff640 1 --2- 172.21.15.7:0/935667689 >> v2:172.21.15.193:3300/0 conn(0x7ff7a80cd560 0x7ff7a80cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a815db70 0x7ff7a815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160252544 0 0) 0x7ff7a815ae90 con 0x7ff7a81540d0 2023-11-28T18:23:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff790003530 con 0x7ff7a81540d0 2023-11-28T18:23:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245190319 0 0) 0x7ff790003530 con 0x7ff7a81540d0 2023-11-28T18:23:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a815db70 msgr2=0x7ff7a815df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a815db70 0x7ff7a815df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 >> v2:172.21.15.193:3300/0 conn(0x7ff7a80cd560 msgr2=0x7ff7a80cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 --2- 172.21.15.7:0/935667689 >> v2:172.21.15.193:3300/0 conn(0x7ff7a80cd560 0x7ff7a80cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a815a610 con 0x7ff7a81540d0 2023-11-28T18:23:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7ff798001680 con 0x7ff7a81540d0 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a7fff640 1 --2- 172.21.15.7:0/935667689 >> v2:172.21.15.193:3300/0 conn(0x7ff7a80cd560 0x7ff7a80cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 >> v1:172.21.15.7:6789/0 conn(0x7ff7a81540d0 legacy=0x7ff7a8072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a815db70 0x7ff790003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790004d10 0x7ff790005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a815e490 con 0x7ff790003f00 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a815a370 con 0x7ff790004d10 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79000b990 con 0x7ff7a815db70 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a77fe640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a815db70 0x7ff790003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a7fff640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790004d10 0x7ff790005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a6ffd640 1 -- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a815db70 msgr2=0x7ff790003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a815db70 0x7ff790003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a6ffd640 1 -- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790004d10 msgr2=0x7ff790005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790004d10 0x7ff790005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a6ffd640 1 -- 172.21.15.7:0/935667689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff790003530 con 0x7ff790003f00 2023-11-28T18:23:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.379+0000 7ff7a7fff640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790004d10 0x7ff790005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7900047d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff79c009ae0 tx=0x7ff79c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95bf4f3e2f119fa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c0a5640 con 0x7ff790003f00 2023-11-28T18:23:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c0a57a0 con 0x7ff790003f00 2023-11-28T18:23:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7a67fc640 1 -- 172.21.15.7:0/935667689 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c0a5b50 con 0x7ff790003f00 2023-11-28T18:23:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/935667689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 msgr2=0x7ff7900047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7ae1d1640 1 --2- 172.21.15.7:0/935667689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7900047d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff79c009ae0 tx=0x7ff79c01c9f0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7a7fff640 1 -- 172.21.15.7:0/935667689 reap_dead start 2023-11-28T18:23:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/935667689 shutdown_connections 2023-11-28T18:23:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/935667689 >> 172.21.15.7:0/935667689 conn(0x7ff7a80662e0 msgr2=0x7ff7a80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/935667689 shutdown_connections 2023-11-28T18:23:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.383+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/935667689 wait complete. 2023-11-28T18:23:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 Processor -- start 2023-11-28T18:23:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 -- start start 2023-11-28T18:23:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7a80ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7a815cef0 0x7ff7a81483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a80cbfa0 0x7ff7a81f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a80cbb80 con 0x7ff790003f00 2023-11-28T18:23:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a815cc30 con 0x7ff7a815cef0 2023-11-28T18:23:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a815b9b0 con 0x7ff7a80cbfa0 2023-11-28T18:23:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7a80ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7a80ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47584/0 (socket says 172.21.15.7:47584) 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 -- 172.21.15.7:0/742207701 learned_addr learned my addr 172.21.15.7:0/742207701 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7a815cef0 0x7ff7a81483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a7fff640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a80cbfa0 0x7ff7a81f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 -- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a80cbfa0 msgr2=0x7ff7a81f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a80cbfa0 0x7ff7a81f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 -- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7a815cef0 msgr2=0x7ff7a81483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7a815cef0 0x7ff7a81483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 -- 172.21.15.7:0/742207701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a815b110 con 0x7ff790003f00 2023-11-28T18:23:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7a815cef0 0x7ff7a81483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:51.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7a77fe640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7a80ccbe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff798012540 tx=0x7ff798012a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff78ffff640 1 -- 172.21.15.7:0/742207701 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798014810 con 0x7ff790003f00 2023-11-28T18:23:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff78ffff640 1 -- 172.21.15.7:0/742207701 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7980149b0 con 0x7ff790003f00 2023-11-28T18:23:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff78ffff640 1 -- 172.21.15.7:0/742207701 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798014d60 con 0x7ff790003f00 2023-11-28T18:23:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a815b3f0 con 0x7ff790003f00 2023-11-28T18:23:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a80cb7a0 con 0x7ff790003f00 2023-11-28T18:23:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.387+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a80cb600 con 0x7ff790003f00 2023-11-28T18:23:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.391+0000 7ff78ffff640 1 -- 172.21.15.7:0/742207701 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff798016480 con 0x7ff790003f00 2023-11-28T18:23:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.391+0000 7ff78ffff640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff794066900 0x7ff794068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.391+0000 7ff78ffff640 1 -- 172.21.15.7:0/742207701 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff798087980 con 0x7ff790003f00 2023-11-28T18:23:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.391+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff794066900 0x7ff794068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.391+0000 7ff7a6ffd640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff794066900 0x7ff794068dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0af000 tx=0x7ff79c0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.395+0000 7ff78ffff640 1 -- 172.21.15.7:0/742207701 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff7a80cb600 con 0x7ff790003f00 2023-11-28T18:23:51.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.551+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff7a80cb600 con 0x7ff790003f00 2023-11-28T18:23:51.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.551+0000 7ff78ffff640 1 -- 172.21.15.7:0/742207701 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff7a80cb600 con 0x7ff790003f00 2023-11-28T18:23:51.559 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-11-28T18:23:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff794066900 msgr2=0x7ff794068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff794066900 0x7ff794068dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0af000 tx=0x7ff79c0046d0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 msgr2=0x7ff7a80ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7a80ccbe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff798012540 tx=0x7ff798012a00 comp rx=0 tx=0).stop 2023-11-28T18:23:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 shutdown_connections 2023-11-28T18:23:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff794066900 0x7ff794068dc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7a80cbfa0 0x7ff7a81f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7a815cef0 0x7ff7a81483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 --2- 172.21.15.7:0/742207701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff790003f00 0x7ff7a80ccbe0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 >> 172.21.15.7:0/742207701 conn(0x7ff7a80662e0 msgr2=0x7ff7a80c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:51.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 shutdown_connections 2023-11-28T18:23:51.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.555+0000 7ff7ae1d1640 1 -- 172.21.15.7:0/742207701 wait complete. 2023-11-28T18:23:51.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T18:23:51.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1edcbd8640 1 Processor -- start 2023-11-28T18:23:51.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1edcbd8640 1 -- start start 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1edcbd8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1ed8154230 0x7f1ed80cb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1edcbd8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1ed8154610 0x7f1ed80cb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1edcbd8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 0x7f1ed80cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1edcbd8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed815aff0 con 0x7f1ed8154230 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1edcbd8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed80cc010 con 0x7f1ed8154610 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1edcbd8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed815a470 con 0x7f1ed80d0150 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1ed8154230 0x7f1ed80cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1ed8154230 0x7f1ed80cb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47600/0 (socket says 172.21.15.7:47600) 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed659c640 1 -- 172.21.15.7:0/145754801 learned_addr learned my addr 172.21.15.7:0/145754801 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed5d9b640 1 --2- 172.21.15.7:0/145754801 >> v2:172.21.15.193:3300/0 conn(0x7f1ed8154610 0x7f1ed80cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed6d9d640 1 --2- 172.21.15.7:0/145754801 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 0x7f1ed80cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed659c640 1 -- 172.21.15.7:0/145754801 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 msgr2=0x7f1ed80cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed659c640 1 --2- 172.21.15.7:0/145754801 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 0x7f1ed80cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed659c640 1 -- 172.21.15.7:0/145754801 >> v2:172.21.15.193:3300/0 conn(0x7f1ed8154610 msgr2=0x7f1ed80cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed659c640 1 --2- 172.21.15.7:0/145754801 >> v2:172.21.15.193:3300/0 conn(0x7f1ed8154610 0x7f1ed80cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed659c640 1 -- 172.21.15.7:0/145754801 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed815a750 con 0x7f1ed8154230 2023-11-28T18:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.723+0000 7f1ed5d9b640 1 --2- 172.21.15.7:0/145754801 >> v2:172.21.15.193:3300/0 conn(0x7f1ed8154610 0x7f1ed80cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1ed659c640 1 --2- 172.21.15.7:0/145754801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154230 0x7f1ed80cb310 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80c9570 tx=0x7f1ec801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723bcd8b9f798a3e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1ed559a640 1 -- 172.21.15.7:0/145754801 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80ab600 con 0x7f1ed8154230 2023-11-28T18:23:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1ed559a640 1 -- 172.21.15.7:0/145754801 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec80ab760 con 0x7f1ed8154230 2023-11-28T18:23:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1ed559a640 1 -- 172.21.15.7:0/145754801 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80abaf0 con 0x7f1ed8154230 2023-11-28T18:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 -- 172.21.15.7:0/145754801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154230 msgr2=0x7f1ed80cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/145754801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154230 0x7f1ed80cb310 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80c9570 tx=0x7f1ec801c440 comp rx=0 tx=0).stop 2023-11-28T18:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 -- 172.21.15.7:0/145754801 shutdown_connections 2023-11-28T18:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/145754801 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 0x7f1ed80cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/145754801 >> v2:172.21.15.193:3300/0 conn(0x7f1ed8154610 0x7f1ed80cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/145754801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154230 0x7f1ed80cb310 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 -- 172.21.15.7:0/145754801 >> 172.21.15.7:0/145754801 conn(0x7f1ed80662e0 msgr2=0x7f1ed80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 -- 172.21.15.7:0/145754801 shutdown_connections 2023-11-28T18:23:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.727+0000 7f1edcbd8640 1 -- 172.21.15.7:0/145754801 wait complete. 2023-11-28T18:23:51.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 Processor -- start 2023-11-28T18:23:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 -- start start 2023-11-28T18:23:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 0x7f1ed80c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ed8154230 0x7f1ed80ce690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154610 0x7f1ed80cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed80cd610 con 0x7f1ed8154610 2023-11-28T18:23:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed81f3d00 con 0x7f1ed8154230 2023-11-28T18:23:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed80cd8d0 con 0x7f1ed80d0150 2023-11-28T18:23:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154610 0x7f1ed80cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154610 0x7f1ed80cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47614/0 (socket says 172.21.15.7:47614) 2023-11-28T18:23:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 -- 172.21.15.7:0/3469201068 learned_addr learned my addr 172.21.15.7:0/3469201068 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed659c640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 0x7f1ed80c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed5d9b640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ed8154230 0x7f1ed80ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 -- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 msgr2=0x7f1ed80c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 0x7f1ed80c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 -- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ed8154230 msgr2=0x7f1ed80ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ed8154230 0x7f1ed80ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 -- 172.21.15.7:0/3469201068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80094d0 con 0x7f1ed8154610 2023-11-28T18:23:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed5d9b640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ed8154230 0x7f1ed80ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ed6d9d640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154610 0x7f1ed80cebd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0007d10 tx=0x7f1ed000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ec6ffd640 1 -- 172.21.15.7:0/3469201068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0014420 con 0x7f1ed8154610 2023-11-28T18:23:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ec6ffd640 1 -- 172.21.15.7:0/3469201068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed0014580 con 0x7f1ed8154610 2023-11-28T18:23:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed80cdbb0 con 0x7f1ed8154610 2023-11-28T18:23:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1ec6ffd640 1 -- 172.21.15.7:0/3469201068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0014930 con 0x7f1ed8154610 2023-11-28T18:23:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.731+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed80ce170 con 0x7f1ed8154610 2023-11-28T18:23:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.735+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed80ce390 con 0x7f1ed8154610 2023-11-28T18:23:51.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.735+0000 7f1ec6ffd640 1 -- 172.21.15.7:0/3469201068 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1ed0014a90 con 0x7f1ed8154610 2023-11-28T18:23:51.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.735+0000 7f1ec6ffd640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1eb8066900 0x7f1eb8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:51.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.735+0000 7f1ec6ffd640 1 -- 172.21.15.7:0/3469201068 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1ed0084d20 con 0x7f1ed8154610 2023-11-28T18:23:51.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.735+0000 7f1ed659c640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1eb8066900 0x7f1eb8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:51.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.735+0000 7f1ed659c640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1eb8066900 0x7f1eb8068dc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80af000 tx=0x7f1ec80b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:51.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.739+0000 7f1ec6ffd640 1 -- 172.21.15.7:0/3469201068 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1ed0089070 con 0x7f1ed8154610 2023-11-28T18:23:51.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ed804e890 con 0x7f1ed8154610 2023-11-28T18:23:51.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1ec6ffd640 1 -- 172.21.15.7:0/3469201068 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ed0051ed0 con 0x7f1ed8154610 2023-11-28T18:23:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-11-28T18:23:51.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1eb8066900 msgr2=0x7f1eb8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1eb8066900 0x7f1eb8068dc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80af000 tx=0x7f1ec80b6040 comp rx=0 tx=0).stop 2023-11-28T18:23:51.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154610 msgr2=0x7f1ed80cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:51.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154610 0x7f1ed80cebd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0007d10 tx=0x7f1ed000e4b0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 shutdown_connections 2023-11-28T18:23:51.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1eb8066900 0x7f1eb8068dc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ed8154610 0x7f1ed80cebd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ed8154230 0x7f1ed80ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 --2- 172.21.15.7:0/3469201068 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ed80d0150 0x7f1ed80c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:51.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 >> 172.21.15.7:0/3469201068 conn(0x7f1ed80662e0 msgr2=0x7f1ed814e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:51.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 shutdown_connections 2023-11-28T18:23:51.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:51.899+0000 7f1edcbd8640 1 -- 172.21.15.7:0/3469201068 wait complete. 2023-11-28T18:23:51.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:23:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faebe757640 1 Processor -- start 2023-11-28T18:23:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faebe757640 1 -- start start 2023-11-28T18:23:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faebe757640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faeb8073070 0x7faeb8074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faebe757640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb80c8490 con 0x7faeb8072950 2023-11-28T18:23:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faebe757640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb80c81f0 con 0x7faeb806c920 2023-11-28T18:23:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faebe757640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb815ab00 con 0x7faeb8073070 2023-11-28T18:23:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faeb7fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faeb8072950 0x7faeb80c19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57488/0 (socket says 172.21.15.7:57488) 2023-11-28T18:23:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faeb7fff640 1 -- 172.21.15.7:0/185950839 learned_addr learned my addr 172.21.15.7:0/185950839 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:23:52.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faeb77fe640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faeb8073070 0x7faeb8074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502780330 0 0) 0x7faeb80c8490 con 0x7faeb8072950 2023-11-28T18:23:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.063+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea0003530 con 0x7faeb8072950 2023-11-28T18:23:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126487936 0 0) 0x7faea0003530 con 0x7faeb8072950 2023-11-28T18:23:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faeb8073070 msgr2=0x7faeb8074030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faeb8073070 0x7faeb8074030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 >> v1:172.21.15.193:6789/0 conn(0x7faeb806c920 legacy=0x7faeb806cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb815ade0 con 0x7faeb8072950 2023-11-28T18:23:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7faea80016a0 con 0x7faeb8072950 2023-11-28T18:23:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 >> v1:172.21.15.7:6789/0 conn(0x7faeb8072950 legacy=0x7faeb80c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faeb8073070 0x7faea00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faea0003ef0 0x7faea00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea0004d10 0x7faea0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb815ab00 con 0x7faea0003ef0 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea000bb50 con 0x7faea0004d10 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea000bd10 con 0x7faeb8073070 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb77fe640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faea0003ef0 0x7faea00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb7fff640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faeb8073070 0x7faea00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faebccf4640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea0004d10 0x7faea0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb77fe640 1 -- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faeb8073070 msgr2=0x7faea00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:52.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb77fe640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faeb8073070 0x7faea00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb77fe640 1 -- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea0004d10 msgr2=0x7faea0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:52.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb77fe640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea0004d10 0x7faea0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb77fe640 1 -- 172.21.15.7:0/185950839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0003530 con 0x7faea0003ef0 2023-11-28T18:23:52.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faebccf4640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea0004d10 0x7faea0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.067+0000 7faeb77fe640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faea0003ef0 0x7faea00047d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faeac009ab0 tx=0x7faeac01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edeae28af7d6635c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0a9400 con 0x7faea0003ef0 2023-11-28T18:23:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac0a9560 con 0x7faea0003ef0 2023-11-28T18:23:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faeb6ffd640 1 -- 172.21.15.7:0/185950839 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0a9710 con 0x7faea0003ef0 2023-11-28T18:23:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- 172.21.15.7:0/185950839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faea0003ef0 msgr2=0x7faea00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 --2- 172.21.15.7:0/185950839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faea0003ef0 0x7faea00047d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faeac009ab0 tx=0x7faeac01ca30 comp rx=0 tx=0).stop 2023-11-28T18:23:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebccf4640 1 -- 172.21.15.7:0/185950839 reap_dead start 2023-11-28T18:23:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- 172.21.15.7:0/185950839 shutdown_connections 2023-11-28T18:23:52.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- 172.21.15.7:0/185950839 >> 172.21.15.7:0/185950839 conn(0x7faeb80cbf80 msgr2=0x7faeb80cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:52.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- 172.21.15.7:0/185950839 shutdown_connections 2023-11-28T18:23:52.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- 172.21.15.7:0/185950839 wait complete. 2023-11-28T18:23:52.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 Processor -- start 2023-11-28T18:23:52.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- start start 2023-11-28T18:23:52.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faea0003ef0 0x7faeb80b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faeb8158b10 0x7faeb81f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faeb80ae8d0 0x7faeb80c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb81594c0 con 0x7faeb8158b10 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8159780 con 0x7faeb80ae8d0 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebe757640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8159a60 con 0x7faea0003ef0 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faeb77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faeb8158b10 0x7faeb81f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faeb77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faeb8158b10 0x7faeb81f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47628/0 (socket says 172.21.15.7:47628) 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faeb77fe640 1 -- 172.21.15.7:0/989630656 learned_addr learned my addr 172.21.15.7:0/989630656 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faebccf4640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faeb80ae8d0 0x7faeb80c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.071+0000 7faeb7fff640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faea0003ef0 0x7faeb80b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb77fe640 1 -- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faea0003ef0 msgr2=0x7faeb80b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb77fe640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faea0003ef0 0x7faeb80b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb77fe640 1 -- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faeb80ae8d0 msgr2=0x7faeb80c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb77fe640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faeb80ae8d0 0x7faeb80c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb77fe640 1 -- 172.21.15.7:0/989630656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb81f2880 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faebccf4640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faeb80ae8d0 0x7faeb80c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb77fe640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faeb8158b10 0x7faeb81f3080 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faeac009a00 tx=0x7faeac0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb4ff9640 1 -- 172.21.15.7:0/989630656 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0a9d70 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb4ff9640 1 -- 172.21.15.7:0/989630656 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac0bb440 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb4ff9640 1 -- 172.21.15.7:0/989630656 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0bb7f0 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb81f2b60 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb81f2570 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb804e890 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb4ff9640 1 -- 172.21.15.7:0/989630656 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7faeac0bb950 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb4ff9640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faea4066900 0x7faea4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb4ff9640 1 -- 172.21.15.7:0/989630656 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faeac11e0f0 con 0x7faeb8158b10 2023-11-28T18:23:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.075+0000 7faeb7fff640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faea4066900 0x7faea4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.079+0000 7faeb7fff640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faea4066900 0x7faea4068dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faea8000c10 tx=0x7faea8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:52.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.079+0000 7faeb4ff9640 1 -- 172.21.15.7:0/989630656 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faeac0eb210 con 0x7faeb8158b10 2023-11-28T18:23:52.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeb8159ba0 con 0x7faeb8158b10 2023-11-28T18:23:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faeb4ff9640 1 -- 172.21.15.7:0/989630656 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6550 (secure 0 0 0) 0x7faeb8159ba0 con 0x7faeb8158b10 2023-11-28T18:23:52.276 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:52.277 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:23:35.257171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:23:52.277 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2023-11-28T18:23:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faea4066900 msgr2=0x7faea4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faea4066900 0x7faea4068dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faea8000c10 tx=0x7faea8009040 comp rx=0 tx=0).stop 2023-11-28T18:23:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faeb8158b10 msgr2=0x7faeb81f3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faeb8158b10 0x7faeb81f3080 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faeac009a00 tx=0x7faeac0197d0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 shutdown_connections 2023-11-28T18:23:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faea4066900 0x7faea4068dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faeb80ae8d0 0x7faeb80c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faeb8158b10 0x7faeb81f3080 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 --2- 172.21.15.7:0/989630656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faea0003ef0 0x7faeb80b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 >> 172.21.15.7:0/989630656 conn(0x7faeb80cbf80 msgr2=0x7faeb80cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:52.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 shutdown_connections 2023-11-28T18:23:52.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.271+0000 7faebe757640 1 -- 172.21.15.7:0/989630656 wait complete. 2023-11-28T18:23:52.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:23:52.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.435+0000 7fda4e29c640 1 Processor -- start 2023-11-28T18:23:52.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.435+0000 7fda4e29c640 1 -- start start 2023-11-28T18:23:52.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4e29c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda48154040 0x7fda48159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4e29c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fda4806a8a0 0x7fda4815a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4e29c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda480662e0 0x7fda48066770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4e29c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda480c8430 con 0x7fda480662e0 2023-11-28T18:23:52.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4e29c640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4806ae50 con 0x7fda4806a8a0 2023-11-28T18:23:52.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4e29c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4806b110 con 0x7fda48154040 2023-11-28T18:23:52.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4c839640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda480662e0 0x7fda48066770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4c839640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda480662e0 0x7fda48066770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47630/0 (socket says 172.21.15.7:47630) 2023-11-28T18:23:52.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4c839640 1 -- 172.21.15.7:0/2299463292 learned_addr learned my addr 172.21.15.7:0/2299463292 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:52.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda477fe640 1 --2- 172.21.15.7:0/2299463292 >> v2:172.21.15.193:3300/0 conn(0x7fda4806a8a0 0x7fda4815a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda47fff640 1 --2- 172.21.15.7:0/2299463292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda48154040 0x7fda48159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4c839640 1 -- 172.21.15.7:0/2299463292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda48154040 msgr2=0x7fda48159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:52.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4c839640 1 --2- 172.21.15.7:0/2299463292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda48154040 0x7fda48159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4c839640 1 -- 172.21.15.7:0/2299463292 >> v2:172.21.15.193:3300/0 conn(0x7fda4806a8a0 msgr2=0x7fda4815a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:23:52.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4c839640 1 --2- 172.21.15.7:0/2299463292 >> v2:172.21.15.193:3300/0 conn(0x7fda4806a8a0 0x7fda4815a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda4c839640 1 -- 172.21.15.7:0/2299463292 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda480c7820 con 0x7fda480662e0 2023-11-28T18:23:52.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.439+0000 7fda47fff640 1 --2- 172.21.15.7:0/2299463292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda48154040 0x7fda48159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:23:52.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4c839640 1 --2- 172.21.15.7:0/2299463292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48066770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fda480c9d30 tx=0x7fda4001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a447fbc952a958cd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:52.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda46ffd640 1 -- 172.21.15.7:0/2299463292 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda400af400 con 0x7fda480662e0 2023-11-28T18:23:52.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda46ffd640 1 -- 172.21.15.7:0/2299463292 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda400ab050 con 0x7fda480662e0 2023-11-28T18:23:52.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda46ffd640 1 -- 172.21.15.7:0/2299463292 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda400af780 con 0x7fda480662e0 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 -- 172.21.15.7:0/2299463292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 msgr2=0x7fda48066770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2299463292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48066770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fda480c9d30 tx=0x7fda4001f1c0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 -- 172.21.15.7:0/2299463292 shutdown_connections 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2299463292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48066770 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2299463292 >> v2:172.21.15.193:3300/0 conn(0x7fda4806a8a0 0x7fda4815a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2299463292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda48154040 0x7fda48159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 -- 172.21.15.7:0/2299463292 >> 172.21.15.7:0/2299463292 conn(0x7fda4806d4d0 msgr2=0x7fda4806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 -- 172.21.15.7:0/2299463292 shutdown_connections 2023-11-28T18:23:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 -- 172.21.15.7:0/2299463292 wait complete. 2023-11-28T18:23:52.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 Processor -- start 2023-11-28T18:23:52.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.443+0000 7fda4e29c640 1 -- start start 2023-11-28T18:23:52.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda4806a8a0 0x7fda48150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda48154040 0x7fda480c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda480ba910 con 0x7fda480662e0 2023-11-28T18:23:52.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda481ee0d0 con 0x7fda48154040 2023-11-28T18:23:52.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda481f3910 con 0x7fda4806a8a0 2023-11-28T18:23:52.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47632/0 (socket says 172.21.15.7:47632) 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 -- 172.21.15.7:0/2921642503 learned_addr learned my addr 172.21.15.7:0/2921642503 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4c839640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda48154040 0x7fda480c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda477fe640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda4806a8a0 0x7fda48150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 -- 172.21.15.7:0/2921642503 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda4806a8a0 msgr2=0x7fda48150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda4806a8a0 0x7fda48150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 -- 172.21.15.7:0/2921642503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda48154040 msgr2=0x7fda480c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda48154040 0x7fda480c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 -- 172.21.15.7:0/2921642503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda40009b30 con 0x7fda480662e0 2023-11-28T18:23:52.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda47fff640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48156f30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d830 tx=0x7fda3800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:52.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda44ff9640 1 -- 172.21.15.7:0/2921642503 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380096c0 con 0x7fda480662e0 2023-11-28T18:23:52.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda44ff9640 1 -- 172.21.15.7:0/2921642503 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda38009820 con 0x7fda480662e0 2023-11-28T18:23:52.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda44ff9640 1 -- 172.21.15.7:0/2921642503 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38009bd0 con 0x7fda480662e0 2023-11-28T18:23:52.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda48157470 con 0x7fda480662e0 2023-11-28T18:23:52.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda481f3e70 con 0x7fda480662e0 2023-11-28T18:23:52.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4804e890 con 0x7fda480662e0 2023-11-28T18:23:52.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.447+0000 7fda44ff9640 1 -- 172.21.15.7:0/2921642503 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fda38009d30 con 0x7fda480662e0 2023-11-28T18:23:52.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.451+0000 7fda44ff9640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fda34066900 0x7fda34068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:23:52.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.451+0000 7fda44ff9640 1 -- 172.21.15.7:0/2921642503 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fda38084b00 con 0x7fda480662e0 2023-11-28T18:23:52.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.451+0000 7fda477fe640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fda34066900 0x7fda34068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:23:52.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.451+0000 7fda477fe640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fda34066900 0x7fda34068dc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda4806e410 tx=0x7fda3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:23:52.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.455+0000 7fda44ff9640 1 -- 172.21.15.7:0/2921642503 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fda3808a020 con 0x7fda480662e0 2023-11-28T18:23:52.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.615+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda48157800 con 0x7fda480662e0 2023-11-28T18:23:52.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.615+0000 7fda44ff9640 1 -- 172.21.15.7:0/2921642503 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16694 (secure 0 0 0) 0x7fda38051cd0 con 0x7fda480662e0 2023-11-28T18:23:52.622 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:23:52.622 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:23:49.291754+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:23:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fda34066900 msgr2=0x7fda34068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fda34066900 0x7fda34068dc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda4806e410 tx=0x7fda3c008040 comp rx=0 tx=0).stop 2023-11-28T18:23:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 msgr2=0x7fda48156f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:23:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48156f30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d830 tx=0x7fda3800dcf0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 shutdown_connections 2023-11-28T18:23:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fda34066900 0x7fda34068dc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda48154040 0x7fda480c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fda4806a8a0 0x7fda48150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 --2- 172.21.15.7:0/2921642503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda480662e0 0x7fda48156f30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:23:52.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 >> 172.21.15.7:0/2921642503 conn(0x7fda4806d4d0 msgr2=0x7fda48158400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:23:52.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 shutdown_connections 2023-11-28T18:23:52.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:23:52.619+0000 7fda4e29c640 1 -- 172.21.15.7:0/2921642503 wait complete. 2023-11-28T18:23:52.644 DEBUG:tasks.mds_thrash:Yielding 2023-11-28T18:23:52.644 INFO:teuthology.run_tasks:Running task workunit... 2023-11-28T18:23:52.656 INFO:tasks.workunit:Pulling workunits from ref 47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T18:23:52.657 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-28T18:23:52.657 INFO:tasks.workunit:timeout=3h 2023-11-28T18:23:52.657 INFO:tasks.workunit:cleanup=True 2023-11-28T18:23:52.657 DEBUG:teuthology.orchestra.run.smithi007:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:52.658 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-11-28T18:23:52.658 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2023-11-28T18:23:52.669 INFO:teuthology.orchestra.run.smithi007.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:52.669 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2023-11-28T18:23:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-11-28T18:23:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T18:23:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-11-28T18:23:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-28 18:23:13.659248574 +0000 2023-11-28T18:23:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-28 18:23:45.478517068 +0000 2023-11-28T18:23:52.669 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-11-28T18:23:52.669 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-11-28T18:23:52.670 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-11-28T18:23:52.744 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T18:23:52.802 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-28T18:23:57.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:23:57.134 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:24:27.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:24:27.399 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:24:57.451 INFO:tasks.workunit.client.0.smithi007.stderr:Updating files: 64% (7875/12117) Updating files: 65% (7877/12117) Updating files: 66% (7998/12117) Updating files: 67% (8119/12117) Updating files: 68% (8240/12117) Updating files: 69% (8361/12117) Updating files: 70% (8482/12117) Updating files: 71% (8604/12117) Updating files: 72% (8725/12117) Updating files: 73% (8846/12117) Updating files: 74% (8967/12117) Updating files: 75% (9088/12117) Updating files: 76% (9209/12117) Updating files: 77% (9331/12117) Updating files: 78% (9452/12117) Updating files: 79% (9573/12117) Updating files: 80% (9694/12117) Updating files: 81% (9815/12117) Updating files: 82% (9936/12117) Updating files: 83% (10058/12117) Updating files: 84% (10179/12117) Updating files: 85% (10300/12117) Updating files: 86% (10421/12117) Updating files: 87% (10542/12117) Updating files: 88% (10663/12117) Updating files: 89% (10785/12117) Updating files: 90% (10906/12117) Updating files: 91% (11027/12117) Updating files: 92% (11148/12117) Updating files: 93% (11269/12117) Updating files: 94% (11390/12117) Updating files: 95% (11512/12117) Updating files: 96% (11633/12117) Updating files: 97% (11754/12117) Updating files: 98% (11875/12117) Updating files: 99% (11996/12117) Updating files: 100% (12117/12117) Updating files: 100% (12117/12117), done. 2023-11-28T18:24:57.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:24:57.515 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:24:57.846 INFO:tasks.workunit.client.0.smithi007.stderr:Note: switching to '47c1c547372ed5bff7984647dc200bac0e019caf'. 2023-11-28T18:24:57.846 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-28T18:24:57.846 INFO:tasks.workunit.client.0.smithi007.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-28T18:24:57.846 INFO:tasks.workunit.client.0.smithi007.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr:state without impacting any branches by switching back to a branch. 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr: git switch -c 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr:Or undo this operation with: 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr: git switch - 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-28T18:24:57.847 INFO:tasks.workunit.client.0.smithi007.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-28T18:24:57.848 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-28T18:24:57.848 INFO:tasks.workunit.client.0.smithi007.stderr:HEAD is now at 47c1c547372 Merge PR #51931 into wip-vshankar-testing-20231127.102654 2023-11-28T18:24:57.850 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-28T18:24:57.859 INFO:tasks.workunit.client.0.smithi007.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-28T18:24:57.862 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-28T18:24:57.862 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-28T18:24:58.587 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-28T18:24:58.687 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-28T18:24:58.746 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-28T18:24:58.748 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-28T18:24:58.748 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-28T18:24:58.803 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-28T18:24:58.809 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:24:58.809 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-28T18:24:58.860 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2023-11-28T18:24:58.861 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2023-11-28T18:24:58.861 DEBUG:teuthology.orchestra.run.smithi007:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2023-11-28T18:24:58.927 INFO:tasks.workunit.client.0.smithi007.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2023-11-28T18:24:58.928 INFO:tasks.workunit.client.0.smithi007.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2023-11-28T18:24:58.928 INFO:tasks.workunit.client.0.smithi007.stderr:+ trap -- retry EXIT 2023-11-28T18:24:58.928 INFO:tasks.workunit.client.0.smithi007.stderr:+ rm -rf ffsb 2023-11-28T18:24:58.931 INFO:tasks.workunit.client.0.smithi007.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2023-11-28T18:24:58.938 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into 'ffsb'... 2023-11-28T18:25:09.518 INFO:tasks.workunit.client.0.smithi007.stderr:Updating files: 69% (41/59) Updating files: 71% (42/59) Updating files: 72% (43/59) Updating files: 74% (44/59) Updating files: 76% (45/59) Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-11-28T18:25:09.606 INFO:tasks.workunit.client.0.smithi007.stderr:+ trap - EXIT 2023-11-28T18:25:09.606 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd ffsb 2023-11-28T18:25:09.610 INFO:tasks.workunit.client.0.smithi007.stderr:+ ./configure 2023-11-28T18:25:10.254 INFO:tasks.workunit.client.0.smithi007.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-11-28T18:25:11.279 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether build environment is sane... yes 2023-11-28T18:25:11.318 INFO:tasks.workunit.client.0.smithi007.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2023-11-28T18:25:11.319 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gawk... gawk 2023-11-28T18:25:11.355 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether make sets $(MAKE)... yes 2023-11-28T18:25:11.458 INFO:tasks.workunit.client.0.smithi007.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-11-28T18:25:11.458 INFO:tasks.workunit.client.0.smithi007.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-11-28T18:25:11.487 INFO:tasks.workunit.client.0.smithi007.stdout:checking for style of include used by make... GNU 2023-11-28T18:25:11.499 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc... gcc 2023-11-28T18:25:11.639 INFO:tasks.workunit.client.0.smithi007.stdout:checking for C compiler default output file name... a.out 2023-11-28T18:25:11.646 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether the C compiler works... yes 2023-11-28T18:25:11.657 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are cross compiling... no 2023-11-28T18:25:11.752 INFO:tasks.workunit.client.0.smithi007.stdout:checking for suffix of executables... 2023-11-28T18:25:11.836 INFO:tasks.workunit.client.0.smithi007.stdout:checking for suffix of object files... o 2023-11-28T18:25:11.931 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are using the GNU C compiler... yes 2023-11-28T18:25:12.025 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether gcc accepts -g... yes 2023-11-28T18:25:12.180 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc option to accept ISO C89... none needed 2023-11-28T18:25:12.443 INFO:tasks.workunit.client.0.smithi007.stdout:checking dependency style of gcc... gcc3 2023-11-28T18:25:13.164 INFO:tasks.workunit.client.0.smithi007.stdout:checking how to run the C preprocessor... gcc -E 2023-11-28T18:25:13.308 INFO:tasks.workunit.client.0.smithi007.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-11-28T18:25:13.312 INFO:tasks.workunit.client.0.smithi007.stdout:checking for egrep... /usr/bin/grep -E 2023-11-28T18:25:13.338 INFO:tasks.workunit.client.0.smithi007.stdout:checking for AIX... no 2023-11-28T18:25:13.349 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc... (cached) gcc 2023-11-28T18:25:13.375 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are using the GNU C compiler... (cached) yes 2023-11-28T18:25:13.379 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether gcc accepts -g... (cached) yes 2023-11-28T18:25:13.379 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2023-11-28T18:25:13.379 INFO:tasks.workunit.client.0.smithi007.stdout:checking dependency style of gcc... (cached) gcc3 2023-11-28T18:25:13.543 INFO:tasks.workunit.client.0.smithi007.stdout:checking for main in -lm... yes 2023-11-28T18:25:13.709 INFO:tasks.workunit.client.0.smithi007.stdout:checking for main in -lpthread... yes 2023-11-28T18:25:16.084 INFO:tasks.workunit.client.0.smithi007.stdout:checking for ANSI C header files... yes 2023-11-28T18:25:16.227 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-11-28T18:25:16.344 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/types.h... yes 2023-11-28T18:25:16.470 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/stat.h... yes 2023-11-28T18:25:16.598 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdlib.h... yes 2023-11-28T18:25:16.724 INFO:tasks.workunit.client.0.smithi007.stdout:checking for string.h... yes 2023-11-28T18:25:16.857 INFO:tasks.workunit.client.0.smithi007.stdout:checking for memory.h... yes 2023-11-28T18:25:16.993 INFO:tasks.workunit.client.0.smithi007.stdout:checking for strings.h... yes 2023-11-28T18:25:17.120 INFO:tasks.workunit.client.0.smithi007.stdout:checking for inttypes.h... yes 2023-11-28T18:25:17.709 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdint.h... yes 2023-11-28T18:25:17.849 INFO:tasks.workunit.client.0.smithi007.stdout:checking for unistd.h... yes 2023-11-28T18:25:17.998 INFO:tasks.workunit.client.0.smithi007.stdout:checking pthread.h usability... yes 2023-11-28T18:25:18.066 INFO:tasks.workunit.client.0.smithi007.stdout:checking pthread.h presence... yes 2023-11-28T18:25:18.068 INFO:tasks.workunit.client.0.smithi007.stdout:checking for pthread.h... yes 2023-11-28T18:25:18.201 INFO:tasks.workunit.client.0.smithi007.stdout:checking fcntl.h usability... yes 2023-11-28T18:25:18.268 INFO:tasks.workunit.client.0.smithi007.stdout:checking fcntl.h presence... yes 2023-11-28T18:25:18.270 INFO:tasks.workunit.client.0.smithi007.stdout:checking for fcntl.h... yes 2023-11-28T18:25:18.402 INFO:tasks.workunit.client.0.smithi007.stdout:checking limits.h usability... yes 2023-11-28T18:25:18.469 INFO:tasks.workunit.client.0.smithi007.stdout:checking limits.h presence... yes 2023-11-28T18:25:18.471 INFO:tasks.workunit.client.0.smithi007.stdout:checking for limits.h... yes 2023-11-28T18:25:18.494 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdint.h... (cached) yes 2023-11-28T18:25:18.622 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/time.h usability... yes 2023-11-28T18:25:18.688 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/time.h presence... yes 2023-11-28T18:25:18.689 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/time.h... yes 2023-11-28T18:25:18.716 INFO:tasks.workunit.client.0.smithi007.stdout:checking for unistd.h... (cached) yes 2023-11-28T18:25:19.469 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/vfs.h usability... yes 2023-11-28T18:25:19.534 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/vfs.h presence... yes 2023-11-28T18:25:19.536 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/vfs.h... yes 2023-11-28T18:25:19.724 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/limits.h usability... no 2023-11-28T18:25:19.801 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/limits.h presence... no 2023-11-28T18:25:19.803 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/limits.h... no 2023-11-28T18:25:19.904 INFO:tasks.workunit.client.0.smithi007.stdout:checking for an ANSI C-conforming const... yes 2023-11-28T18:25:20.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:20.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa1281f8640 1 Processor -- start 2023-11-28T18:25:20.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa1281f8640 1 -- start start 2023-11-28T18:25:20.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa1281f8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa120150b30 0x7fa1200c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:20.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa1281f8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1200c7980 0x7fa1200c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa1281f8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1200c6140 con 0x7fa1201575c0 2023-11-28T18:25:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa1281f8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1200c72b0 con 0x7fa1200c7980 2023-11-28T18:25:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa1281f8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa120070c00 con 0x7fa120150b30 2023-11-28T18:25:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa126795640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1200c7980 0x7fa1200c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa126795640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1200c7980 0x7fa1200c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50552/0 (socket says 172.21.15.7:50552) 2023-11-28T18:25:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa126795640 1 -- 172.21.15.7:0/3118708817 learned_addr learned my addr 172.21.15.7:0/3118708817 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa125f94640 1 --2- 172.21.15.7:0/3118708817 >> v2:172.21.15.193:3300/0 conn(0x7fa120150b30 0x7fa1200c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa126795640 1 -- 172.21.15.7:0/3118708817 >> v2:172.21.15.193:3300/0 conn(0x7fa120150b30 msgr2=0x7fa1200c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa126795640 1 --2- 172.21.15.7:0/3118708817 >> v2:172.21.15.193:3300/0 conn(0x7fa120150b30 0x7fa1200c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa126795640 1 -- 172.21.15.7:0/3118708817 >> v1:172.21.15.193:6789/0 conn(0x7fa1201575c0 legacy=0x7fa1200c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.802+0000 7fa126795640 1 -- 172.21.15.7:0/3118708817 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1200c8310 con 0x7fa1200c7980 2023-11-28T18:25:20.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.806+0000 7fa126795640 1 --2- 172.21.15.7:0/3118708817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1200c7980 0x7fa1200c7dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa12006ec80 tx=0x7fa11801f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecdb9899d694af9c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:20.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.806+0000 7fa124f92640 1 -- 172.21.15.7:0/3118708817 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1180af400 con 0x7fa1200c7980 2023-11-28T18:25:20.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.806+0000 7fa124f92640 1 -- 172.21.15.7:0/3118708817 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1180ab050 con 0x7fa1200c7980 2023-11-28T18:25:20.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.806+0000 7fa124f92640 1 -- 172.21.15.7:0/3118708817 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1180af780 con 0x7fa1200c7980 2023-11-28T18:25:20.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 -- 172.21.15.7:0/3118708817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1200c7980 msgr2=0x7fa1200c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:20.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 --2- 172.21.15.7:0/3118708817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1200c7980 0x7fa1200c7dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa12006ec80 tx=0x7fa11801f180 comp rx=0 tx=0).stop 2023-11-28T18:25:20.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 -- 172.21.15.7:0/3118708817 shutdown_connections 2023-11-28T18:25:20.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 --2- 172.21.15.7:0/3118708817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1200c7980 0x7fa1200c7dd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:20.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 --2- 172.21.15.7:0/3118708817 >> v2:172.21.15.193:3300/0 conn(0x7fa120150b30 0x7fa1200c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:20.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 -- 172.21.15.7:0/3118708817 >> 172.21.15.7:0/3118708817 conn(0x7fa1200662e0 msgr2=0x7fa1200666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:20.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 -- 172.21.15.7:0/3118708817 shutdown_connections 2023-11-28T18:25:20.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 -- 172.21.15.7:0/3118708817 wait complete. 2023-11-28T18:25:20.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 Processor -- start 2023-11-28T18:25:20.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.810+0000 7fa1281f8640 1 -- start start 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1281f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1200c7980 0x7fa12006e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1281f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa120150b30 0x7fa12014a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1281f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1201575c0 0x7fa120153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1281f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1201f6f70 con 0x7fa1201575c0 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1281f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1201525a0 con 0x7fa120150b30 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1281f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa120153750 con 0x7fa1200c7980 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa120150b30 0x7fa12014a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa120150b30 0x7fa12014a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:55476/0 (socket says 172.21.15.7:55476) 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125f94640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1200c7980 0x7fa12006e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 -- 172.21.15.7:0/1029406239 learned_addr learned my addr 172.21.15.7:0/1029406239 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 -- 172.21.15.7:0/1029406239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1200c7980 msgr2=0x7fa12006e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1200c7980 0x7fa12006e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 -- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1201575c0 msgr2=0x7fa120153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1201575c0 0x7fa120153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 -- 172.21.15.7:0/1029406239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa118009b30 con 0x7fa120150b30 2023-11-28T18:25:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa125793640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa120150b30 0x7fa12014a2a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa11000b410 tx=0x7fa11000b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1167fc640 1 -- 172.21.15.7:0/1029406239 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110004020 con 0x7fa120150b30 2023-11-28T18:25:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1167fc640 1 -- 172.21.15.7:0/1029406239 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa110004180 con 0x7fa120150b30 2023-11-28T18:25:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1167fc640 1 -- 172.21.15.7:0/1029406239 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110011630 con 0x7fa120150b30 2023-11-28T18:25:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa120151a20 con 0x7fa120150b30 2023-11-28T18:25:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.814+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa120151fe0 con 0x7fa120150b30 2023-11-28T18:25:20.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.818+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12004e890 con 0x7fa120150b30 2023-11-28T18:25:20.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.818+0000 7fa1167fc640 1 -- 172.21.15.7:0/1029406239 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa110011790 con 0x7fa120150b30 2023-11-28T18:25:20.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.818+0000 7fa1167fc640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa104066920 0x7fa104068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:20.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.818+0000 7fa1167fc640 1 -- 172.21.15.7:0/1029406239 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa1100840c0 con 0x7fa120150b30 2023-11-28T18:25:20.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.818+0000 7fa125f94640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa104066920 0x7fa104068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:20.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.822+0000 7fa125f94640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa104066920 0x7fa104068de0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa11c007a60 tx=0x7fa11c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:20.826+0000 7fa1167fc640 1 -- 172.21.15.7:0/1029406239 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa110051360 con 0x7fa120150b30 2023-11-28T18:25:20.920 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether time.h and sys/time.h may both be included... yes 2023-11-28T18:25:20.924 INFO:tasks.workunit.client.0.smithi007.stdout:checking for function prototypes... yes 2023-11-28T18:25:21.029 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether setvbuf arguments are reversed... no 2023-11-28T18:25:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.042+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1200c2d80 con 0x7fa120150b30 2023-11-28T18:25:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.042+0000 7fa1167fc640 1 -- 172.21.15.7:0/1029406239 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6553 (secure 0 0 0) 0x7fa110054980 con 0x7fa120150b30 2023-11-28T18:25:21.046 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:21.046 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:18.705853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:21.046 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 18 2023-11-28T18:25:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa104066920 msgr2=0x7fa104068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa104066920 0x7fa104068de0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa11c007a60 tx=0x7fa11c008040 comp rx=0 tx=0).stop 2023-11-28T18:25:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa120150b30 msgr2=0x7fa12014a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa120150b30 0x7fa12014a2a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa11000b410 tx=0x7fa11000b8d0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 shutdown_connections 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa104066920 0x7fa104068de0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1201575c0 0x7fa120153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa120150b30 0x7fa12014a2a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 --2- 172.21.15.7:0/1029406239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1200c7980 0x7fa12006e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 >> 172.21.15.7:0/1029406239 conn(0x7fa1200662e0 msgr2=0x7fa1200c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 shutdown_connections 2023-11-28T18:25:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.046+0000 7fa1281f8640 1 -- 172.21.15.7:0/1029406239 wait complete. 2023-11-28T18:25:21.071 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2023-11-28T18:25:21.072 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T18:25:21.072 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:25:21.085 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T18:25:21.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:21.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.238+0000 7f7d00279640 1 Processor -- start 2023-11-28T18:25:21.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.238+0000 7f7d00279640 1 -- start start 2023-11-28T18:25:21.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7d00279640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:21.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7d00279640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8158000 con 0x7f7cf8157b90 2023-11-28T18:25:21.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7d00279640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8071e60 con 0x7f7cf806e560 2023-11-28T18:25:21.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7d00279640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8070930 con 0x7f7cf8151cd0 2023-11-28T18:25:21.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfe816640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7cf8157b90 0x7f7cf8162510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47618/0 (socket says 172.21.15.7:47618) 2023-11-28T18:25:21.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfe816640 1 -- 172.21.15.7:0/2862071468 learned_addr learned my addr 172.21.15.7:0/2862071468 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:25:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfe015640 1 --2- 172.21.15.7:0/2862071468 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfd013640 1 -- 172.21.15.7:0/2862071468 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455052806 0 0) 0x7f7cf8158000 con 0x7f7cf8157b90 2023-11-28T18:25:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfd013640 1 -- 172.21.15.7:0/2862071468 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce4003530 con 0x7f7cf8157b90 2023-11-28T18:25:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfd013640 1 -- 172.21.15.7:0/2862071468 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567857087 0 0) 0x7f7cf8071e60 con 0x7f7cf806e560 2023-11-28T18:25:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfd013640 1 -- 172.21.15.7:0/2862071468 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf8158000 con 0x7f7cf806e560 2023-11-28T18:25:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfe015640 1 -- 172.21.15.7:0/2862071468 >> v1:172.21.15.193:6789/0 conn(0x7f7cf806e560 legacy=0x7f7cf80c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfe015640 1 -- 172.21.15.7:0/2862071468 >> v1:172.21.15.7:6789/0 conn(0x7f7cf8157b90 legacy=0x7f7cf8162510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.242+0000 7f7cfe015640 1 -- 172.21.15.7:0/2862071468 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf8070bf0 con 0x7f7cf8151cd0 2023-11-28T18:25:21.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7cfe015640 1 --2- 172.21.15.7:0/2862071468 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf815bd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40097f0 tx=0x7f7cf401c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ed78aa3965829a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:21.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7cfd013640 1 -- 172.21.15.7:0/2862071468 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf40ab600 con 0x7f7cf8151cd0 2023-11-28T18:25:21.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7cfd013640 1 -- 172.21.15.7:0/2862071468 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf40a7050 con 0x7f7cf8151cd0 2023-11-28T18:25:21.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7cfd013640 1 -- 172.21.15.7:0/2862071468 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf40ab980 con 0x7f7cf8151cd0 2023-11-28T18:25:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 -- 172.21.15.7:0/2862071468 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 msgr2=0x7f7cf815bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 --2- 172.21.15.7:0/2862071468 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf815bd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40097f0 tx=0x7f7cf401c430 comp rx=0 tx=0).stop 2023-11-28T18:25:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 -- 172.21.15.7:0/2862071468 shutdown_connections 2023-11-28T18:25:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 --2- 172.21.15.7:0/2862071468 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf815bd00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 -- 172.21.15.7:0/2862071468 >> 172.21.15.7:0/2862071468 conn(0x7f7cf80662e0 msgr2=0x7f7cf80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 -- 172.21.15.7:0/2862071468 shutdown_connections 2023-11-28T18:25:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 -- 172.21.15.7:0/2862071468 wait complete. 2023-11-28T18:25:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 Processor -- start 2023-11-28T18:25:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.246+0000 7f7d00279640 1 -- start start 2023-11-28T18:25:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf806e560 0x7f7cf80727e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf81f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf8157b90 0x7f7cf81f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8146570 con 0x7f7cf806e560 2023-11-28T18:25:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8147040 con 0x7f7cf8157b90 2023-11-28T18:25:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8147590 con 0x7f7cf8151cd0 2023-11-28T18:25:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf806e560 0x7f7cf80727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf806e560 0x7f7cf80727e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50580/0 (socket says 172.21.15.7:50580) 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 -- 172.21.15.7:0/2554903813 learned_addr learned my addr 172.21.15.7:0/2554903813 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe816640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf8157b90 0x7f7cf81f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfd814640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf81f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 -- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 msgr2=0x7f7cf81f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf81f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 -- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf8157b90 msgr2=0x7f7cf81f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf8157b90 0x7f7cf81f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 -- 172.21.15.7:0/2554903813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4009420 con 0x7f7cf806e560 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfd814640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf81f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:25:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe816640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf8157b90 0x7f7cf81f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:25:21.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7cfe015640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf806e560 0x7f7cf80727e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009740 tx=0x7f7cf401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:21.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7ceeffd640 1 -- 172.21.15.7:0/2554903813 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf40b7450 con 0x7f7cf806e560 2023-11-28T18:25:21.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7ceeffd640 1 -- 172.21.15.7:0/2554903813 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf40b75d0 con 0x7f7cf806e560 2023-11-28T18:25:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf80c6fa0 con 0x7f7cf806e560 2023-11-28T18:25:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7ceeffd640 1 -- 172.21.15.7:0/2554903813 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf40b7980 con 0x7f7cf806e560 2023-11-28T18:25:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf80c7560 con 0x7f7cf806e560 2023-11-28T18:25:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.250+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf804e890 con 0x7f7cf806e560 2023-11-28T18:25:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.254+0000 7f7ceeffd640 1 -- 172.21.15.7:0/2554903813 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7cf40b7ae0 con 0x7f7cf806e560 2023-11-28T18:25:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.254+0000 7f7ceeffd640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7cdc066900 0x7f7cdc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.254+0000 7f7ceeffd640 1 -- 172.21.15.7:0/2554903813 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7cf411e920 con 0x7f7cf806e560 2023-11-28T18:25:21.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.254+0000 7f7cfd814640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7cdc066900 0x7f7cdc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:21.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.254+0000 7f7cfd814640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7cdc066900 0x7f7cdc068dc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80ca5f0 tx=0x7f7ce8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:21.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.258+0000 7f7ceeffd640 1 -- 172.21.15.7:0/2554903813 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7cf40ebad0 con 0x7f7cf806e560 2023-11-28T18:25:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf80c62e0 con 0x7f7cf806e560 2023-11-28T18:25:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7ceeffd640 1 -- 172.21.15.7:0/2554903813 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f7cf40ef0f0 con 0x7f7cf806e560 2023-11-28T18:25:21.449 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:21.449 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:18.705853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 18 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7cdc066900 msgr2=0x7f7cdc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7cdc066900 0x7f7cdc068dc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80ca5f0 tx=0x7f7ce8009040 comp rx=0 tx=0).stop 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf806e560 msgr2=0x7f7cf80727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf806e560 0x7f7cf80727e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009740 tx=0x7f7cf401ac70 comp rx=0 tx=0).stop 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 shutdown_connections 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7cdc066900 0x7f7cdc068dc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf8157b90 0x7f7cf81f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7cf8151cd0 0x7f7cf81f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 --2- 172.21.15.7:0/2554903813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf806e560 0x7f7cf80727e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 >> 172.21.15.7:0/2554903813 conn(0x7f7cf80662e0 msgr2=0x7f7cf80c9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 shutdown_connections 2023-11-28T18:25:21.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:21.446+0000 7f7d00279640 1 -- 172.21.15.7:0/2554903813 wait complete. 2023-11-28T18:25:21.470 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:25:21.487 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-11-28T18:25:21.993 INFO:tasks.workunit.client.0.smithi007.stdout:checking for system... yes 2023-11-28T18:25:22.510 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gettimeofday... yes 2023-11-28T18:25:22.838 INFO:tasks.workunit.client.0.smithi007.stdout:checking for mkdir... yes 2023-11-28T18:25:23.021 INFO:tasks.workunit.client.0.smithi007.stdout:checking for strerror... yes 2023-11-28T18:25:23.202 INFO:tasks.workunit.client.0.smithi007.stdout:checking for open64... yes 2023-11-28T18:25:23.381 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stat64... yes 2023-11-28T18:25:23.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:23.571 INFO:tasks.workunit.client.0.smithi007.stdout:checking for fseeko64... yes 2023-11-28T18:25:23.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed20417640 1 Processor -- start 2023-11-28T18:25:23.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed20417640 1 -- start start 2023-11-28T18:25:23.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed20417640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fed180c24f0 0x7fed181575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:23.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed20417640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fed180c7980 0x7fed180c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:23.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed20417640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed180c6140 con 0x7fed18150b30 2023-11-28T18:25:23.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed20417640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed180c72b0 con 0x7fed180c7980 2023-11-28T18:25:23.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed20417640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed18070c00 con 0x7fed180c24f0 2023-11-28T18:25:23.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1e9b4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fed180c7980 0x7fed180c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1e9b4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fed180c7980 0x7fed180c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39432/0 (socket says 172.21.15.7:39432) 2023-11-28T18:25:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1e1b3640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fed18150b30 0x7fed180c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50654/0 (socket says 172.21.15.7:50654) 2023-11-28T18:25:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1e9b4640 1 -- 172.21.15.7:0/1145829703 learned_addr learned my addr 172.21.15.7:0/1145829703 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1d9b2640 1 --2- 172.21.15.7:0/1145829703 >> v2:172.21.15.193:3300/0 conn(0x7fed180c24f0 0x7fed181575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1e9b4640 1 -- 172.21.15.7:0/1145829703 >> v2:172.21.15.193:3300/0 conn(0x7fed180c24f0 msgr2=0x7fed181575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1e9b4640 1 --2- 172.21.15.7:0/1145829703 >> v2:172.21.15.193:3300/0 conn(0x7fed180c24f0 0x7fed181575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1e9b4640 1 -- 172.21.15.7:0/1145829703 >> v1:172.21.15.7:6789/0 conn(0x7fed18150b30 legacy=0x7fed180c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.626+0000 7fed1e9b4640 1 -- 172.21.15.7:0/1145829703 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed18157b00 con 0x7fed180c7980 2023-11-28T18:25:23.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed1e9b4640 1 --2- 172.21.15.7:0/1145829703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c7980 0x7fed180c7dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fed1806fee0 tx=0x7fed1001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15114ff7f32378d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:23.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed1d1b1640 1 -- 172.21.15.7:0/1145829703 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100af400 con 0x7fed180c7980 2023-11-28T18:25:23.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed1d1b1640 1 -- 172.21.15.7:0/1145829703 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed100ab050 con 0x7fed180c7980 2023-11-28T18:25:23.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed1d1b1640 1 -- 172.21.15.7:0/1145829703 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100af780 con 0x7fed180c7980 2023-11-28T18:25:23.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 -- 172.21.15.7:0/1145829703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c7980 msgr2=0x7fed180c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:23.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 --2- 172.21.15.7:0/1145829703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c7980 0x7fed180c7dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fed1806fee0 tx=0x7fed1001f1c0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 -- 172.21.15.7:0/1145829703 shutdown_connections 2023-11-28T18:25:23.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 --2- 172.21.15.7:0/1145829703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c7980 0x7fed180c7dd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 --2- 172.21.15.7:0/1145829703 >> v2:172.21.15.193:3300/0 conn(0x7fed180c24f0 0x7fed181575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 -- 172.21.15.7:0/1145829703 >> 172.21.15.7:0/1145829703 conn(0x7fed180662e0 msgr2=0x7fed180666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:23.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 -- 172.21.15.7:0/1145829703 shutdown_connections 2023-11-28T18:25:23.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 -- 172.21.15.7:0/1145829703 wait complete. 2023-11-28T18:25:23.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.630+0000 7fed20417640 1 Processor -- start 2023-11-28T18:25:23.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 -- start start 2023-11-28T18:25:23.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c24f0 0x7fed181f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:23.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fed180c7980 0x7fed181e1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18150b30 0x7fed181e2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed181e1790 con 0x7fed180c24f0 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1814df50 con 0x7fed18150b30 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1814dc70 con 0x7fed180c7980 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c24f0 0x7fed181f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c24f0 0x7fed181f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39434/0 (socket says 172.21.15.7:39434) 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 -- 172.21.15.7:0/3397049415 learned_addr learned my addr 172.21.15.7:0/3397049415 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1d9b2640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fed180c7980 0x7fed181e1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e9b4640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18150b30 0x7fed181e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 -- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fed180c7980 msgr2=0x7fed181e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fed180c7980 0x7fed181e1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 -- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18150b30 msgr2=0x7fed181e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18150b30 0x7fed181e2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 -- 172.21.15.7:0/3397049415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10009b30 con 0x7fed180c24f0 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1d9b2640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fed180c7980 0x7fed181e1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed1e1b3640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c24f0 0x7fed181f0430 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fed1400c490 tx=0x7fed1400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed0effd640 1 -- 172.21.15.7:0/3397049415 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14007480 con 0x7fed180c24f0 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed0effd640 1 -- 172.21.15.7:0/3397049415 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed140075e0 con 0x7fed180c24f0 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed0effd640 1 -- 172.21.15.7:0/3397049415 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14004a10 con 0x7fed180c24f0 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed181e1140 con 0x7fed180c24f0 2023-11-28T18:25:23.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1814d970 con 0x7fed180c24f0 2023-11-28T18:25:23.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.634+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1804e890 con 0x7fed180c24f0 2023-11-28T18:25:23.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.638+0000 7fed0effd640 1 -- 172.21.15.7:0/3397049415 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fed14004b70 con 0x7fed180c24f0 2023-11-28T18:25:23.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.638+0000 7fed0effd640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fecfc0668d0 0x7fecfc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:23.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.638+0000 7fed0effd640 1 -- 172.21.15.7:0/3397049415 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fed140836b0 con 0x7fed180c24f0 2023-11-28T18:25:23.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.638+0000 7fed1d9b2640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fecfc0668d0 0x7fecfc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:23.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.638+0000 7fed1d9b2640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fecfc0668d0 0x7fecfc068d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fed181f0ae0 tx=0x7fed08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:23.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.642+0000 7fed0effd640 1 -- 172.21.15.7:0/3397049415 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fed14050860 con 0x7fed180c24f0 2023-11-28T18:25:23.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed1814e240 con 0x7fed180c24f0 2023-11-28T18:25:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed0effd640 1 -- 172.21.15.7:0/3397049415 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6553 (secure 0 0 0) 0x7fed14053e80 con 0x7fed180c24f0 2023-11-28T18:25:23.848 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:23.848 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:18.705853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:23.849 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 18 2023-11-28T18:25:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fecfc0668d0 msgr2=0x7fecfc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fecfc0668d0 0x7fecfc068d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fed181f0ae0 tx=0x7fed08008040 comp rx=0 tx=0).stop 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c24f0 msgr2=0x7fed181f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c24f0 0x7fed181f0430 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fed1400c490 tx=0x7fed1400c950 comp rx=0 tx=0).stop 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 shutdown_connections 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fecfc0668d0 0x7fecfc068d90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18150b30 0x7fed181e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fed180c7980 0x7fed181e1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 --2- 172.21.15.7:0/3397049415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed180c24f0 0x7fed181f0430 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 >> 172.21.15.7:0/3397049415 conn(0x7fed180662e0 msgr2=0x7fed180c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 shutdown_connections 2023-11-28T18:25:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:23.846+0000 7fed20417640 1 -- 172.21.15.7:0/3397049415 wait complete. 2023-11-28T18:25:23.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:25:23.917 INFO:tasks.workunit.client.0.smithi007.stdout:checking for lrand48_r... yes 2023-11-28T18:25:24.095 INFO:tasks.workunit.client.0.smithi007.stdout:checking for srand48_r... yes 2023-11-28T18:25:24.160 INFO:tasks.workunit.client.0.smithi007.stdout:configure: creating ./config.status 2023-11-28T18:25:24.508 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating Makefile 2023-11-28T18:25:24.555 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating config.h 2023-11-28T18:25:25.017 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: executing depfiles commands 2023-11-28T18:25:25.282 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: executing default commands 2023-11-28T18:25:25.374 INFO:tasks.workunit.client.0.smithi007.stderr:+ make 2023-11-28T18:25:25.434 INFO:tasks.workunit.client.0.smithi007.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2023-11-28T18:25:25.448 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2023-11-28T18:25:25.450 INFO:tasks.workunit.client.0.smithi007.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2023-11-28T18:25:25.450 INFO:tasks.workunit.client.0.smithi007.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2023-11-28T18:25:25.450 INFO:tasks.workunit.client.0.smithi007.stderr: to install the `Automake' and `Perl' packages. Grab them from 2023-11-28T18:25:25.450 INFO:tasks.workunit.client.0.smithi007.stderr: any GNU archive site. 2023-11-28T18:25:25.461 INFO:tasks.workunit.client.0.smithi007.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2023-11-28T18:25:25.471 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2023-11-28T18:25:25.474 INFO:tasks.workunit.client.0.smithi007.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2023-11-28T18:25:25.474 INFO:tasks.workunit.client.0.smithi007.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2023-11-28T18:25:25.474 INFO:tasks.workunit.client.0.smithi007.stderr: You might want to install the `Automake' and `Perl' packages. 2023-11-28T18:25:25.474 INFO:tasks.workunit.client.0.smithi007.stderr: Grab them from any GNU archive site. 2023-11-28T18:25:25.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:26.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.022+0000 7fe9dad93640 1 Processor -- start 2023-11-28T18:25:26.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9dad93640 1 -- start start 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9dad93640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe9d415ce10 0x7fe9d40ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9dad93640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9dad93640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d415a3a0 con 0x7fe9d415ca30 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9dad93640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d415a620 con 0x7fe9d415ce10 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9dad93640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d415a900 con 0x7fe9d40cafe0 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9da592640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9d9590640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe9d415ce10 0x7fe9d40ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9da592640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:56262/0 (socket says 172.21.15.7:56262) 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9da592640 1 -- 172.21.15.7:0/4119116493 learned_addr learned my addr 172.21.15.7:0/4119116493 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9d9d91640 1 --1- 172.21.15.7:0/4119116493 >> v1:172.21.15.193:6789/0 conn(0x7fe9d415ca30 0x7fe9d4159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:39262/0 (socket says 172.21.15.7:39262) 2023-11-28T18:25:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9da592640 1 -- 172.21.15.7:0/4119116493 >> v2:172.21.15.193:3300/0 conn(0x7fe9d415ce10 msgr2=0x7fe9d40ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9da592640 1 --2- 172.21.15.7:0/4119116493 >> v2:172.21.15.193:3300/0 conn(0x7fe9d415ce10 0x7fe9d40ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9da592640 1 -- 172.21.15.7:0/4119116493 >> v1:172.21.15.193:6789/0 conn(0x7fe9d415ca30 legacy=0x7fe9d4159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9da592640 1 -- 172.21.15.7:0/4119116493 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d415abe0 con 0x7fe9d40cafe0 2023-11-28T18:25:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9d8d8f640 1 -- 172.21.15.7:0/4119116493 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772620485 0 0) 0x7fe9d415a3a0 con 0x7fe9d415ca30 2023-11-28T18:25:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.026+0000 7fe9d9590640 1 --2- 172.21.15.7:0/4119116493 >> v2:172.21.15.193:3300/0 conn(0x7fe9d415ce10 0x7fe9d40ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:25:26.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9da592640 1 --2- 172.21.15.7:0/4119116493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40ca710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc009e50 tx=0x7fe9cc01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=949c56597563265a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:26.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9d8d8f640 1 -- 172.21.15.7:0/4119116493 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0af400 con 0x7fe9d40cafe0 2023-11-28T18:25:26.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9d8d8f640 1 -- 172.21.15.7:0/4119116493 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc0ab050 con 0x7fe9d40cafe0 2023-11-28T18:25:26.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9d8d8f640 1 -- 172.21.15.7:0/4119116493 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0af780 con 0x7fe9d40cafe0 2023-11-28T18:25:26.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9dad93640 1 -- 172.21.15.7:0/4119116493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 msgr2=0x7fe9d40ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:26.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9dad93640 1 --2- 172.21.15.7:0/4119116493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40ca710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc009e50 tx=0x7fe9cc01f1f0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9dad93640 1 -- 172.21.15.7:0/4119116493 shutdown_connections 2023-11-28T18:25:26.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9dad93640 1 --2- 172.21.15.7:0/4119116493 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40ca710 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9dad93640 1 --2- 172.21.15.7:0/4119116493 >> v2:172.21.15.193:3300/0 conn(0x7fe9d415ce10 0x7fe9d40ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9dad93640 1 -- 172.21.15.7:0/4119116493 >> 172.21.15.7:0/4119116493 conn(0x7fe9d406c060 msgr2=0x7fe9d40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:26.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9dad93640 1 -- 172.21.15.7:0/4119116493 shutdown_connections 2023-11-28T18:25:26.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.030+0000 7fe9dad93640 1 -- 172.21.15.7:0/4119116493 wait complete. 2023-11-28T18:25:26.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9dad93640 1 Processor -- start 2023-11-28T18:25:26.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9dad93640 1 -- start start 2023-11-28T18:25:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9dad93640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9dad93640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d415ca30 0x7fe9d41e6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9dad93640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9d415ce10 0x7fe9d414eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9dad93640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4155ac0 con 0x7fe9d415ca30 2023-11-28T18:25:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9dad93640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4154690 con 0x7fe9d415ce10 2023-11-28T18:25:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9dad93640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4156fb0 con 0x7fe9d40cafe0 2023-11-28T18:25:26.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d415ca30 0x7fe9d41e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:26.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d415ca30 0x7fe9d41e6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39444/0 (socket says 172.21.15.7:39444) 2023-11-28T18:25:26.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9d91640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:26.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 -- 172.21.15.7:0/3881142377 learned_addr learned my addr 172.21.15.7:0/3881142377 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:26.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9da592640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9d415ce10 0x7fe9d414eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 -- 172.21.15.7:0/3881142377 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 msgr2=0x7fe9d40c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 -- 172.21.15.7:0/3881142377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9d415ce10 msgr2=0x7fe9d414eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9d415ce10 0x7fe9d414eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 -- 172.21.15.7:0/3881142377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c4009420 con 0x7fe9d415ca30 2023-11-28T18:25:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.034+0000 7fe9d9590640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d415ca30 0x7fe9d41e6840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400e9b0 tx=0x7fe9c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.038+0000 7fe9ca7fc640 1 -- 172.21.15.7:0/3881142377 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c400c9a0 con 0x7fe9d415ca30 2023-11-28T18:25:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.038+0000 7fe9ca7fc640 1 -- 172.21.15.7:0/3881142377 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c400cb00 con 0x7fe9d415ca30 2023-11-28T18:25:26.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.038+0000 7fe9ca7fc640 1 -- 172.21.15.7:0/3881142377 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c400ce90 con 0x7fe9d415ca30 2023-11-28T18:25:26.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.038+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc009b30 con 0x7fe9d415ca30 2023-11-28T18:25:26.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.038+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d4157220 con 0x7fe9d415ca30 2023-11-28T18:25:26.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.038+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d404e890 con 0x7fe9d415ca30 2023-11-28T18:25:26.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.038+0000 7fe9ca7fc640 1 -- 172.21.15.7:0/3881142377 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe9c4013470 con 0x7fe9d415ca30 2023-11-28T18:25:26.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.042+0000 7fe9ca7fc640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe9b8066900 0x7fe9b8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:26.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.042+0000 7fe9ca7fc640 1 -- 172.21.15.7:0/3881142377 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe9c40849c0 con 0x7fe9d415ca30 2023-11-28T18:25:26.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.042+0000 7fe9d9d91640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe9b8066900 0x7fe9b8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:26.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.042+0000 7fe9d9d91640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe9b8066900 0x7fe9b8068dc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0007fd0 tx=0x7fe9d0007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:26.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.046+0000 7fe9ca7fc640 1 -- 172.21.15.7:0/3881142377 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe9c4051c20 con 0x7fe9d415ca30 2023-11-28T18:25:26.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d41547d0 con 0x7fe9d415ca30 2023-11-28T18:25:26.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9ca7fc640 1 -- 172.21.15.7:0/3881142377 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fe9c4017090 con 0x7fe9d415ca30 2023-11-28T18:25:26.244 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:26.244 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:23.706668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:26.244 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 19 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe9b8066900 msgr2=0x7fe9b8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe9b8066900 0x7fe9b8068dc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0007fd0 tx=0x7fe9d0007d60 comp rx=0 tx=0).stop 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d415ca30 msgr2=0x7fe9d41e6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d415ca30 0x7fe9d41e6840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400e9b0 tx=0x7fe9c400ee70 comp rx=0 tx=0).stop 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 shutdown_connections 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe9b8066900 0x7fe9b8068dc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9d415ce10 0x7fe9d414eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d415ca30 0x7fe9d41e6840 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 --2- 172.21.15.7:0/3881142377 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe9d40cafe0 0x7fe9d40c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:26.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 >> 172.21.15.7:0/3881142377 conn(0x7fe9d406c060 msgr2=0x7fe9d40ce0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:26.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 shutdown_connections 2023-11-28T18:25:26.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:26.242+0000 7fe9dad93640 1 -- 172.21.15.7:0/3881142377 wait complete. 2023-11-28T18:25:26.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:25:26.532 INFO:tasks.workunit.client.0.smithi007.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2023-11-28T18:25:27.357 INFO:tasks.workunit.client.0.smithi007.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-11-28T18:25:27.357 INFO:tasks.workunit.client.0.smithi007.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-11-28T18:25:27.357 INFO:tasks.workunit.client.0.smithi007.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-11-28T18:25:27.357 INFO:tasks.workunit.client.0.smithi007.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-11-28T18:25:27.474 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2023-11-28T18:25:27.474 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:4: You should run autoupdate. 2023-11-28T18:25:27.474 INFO:tasks.workunit.client.0.smithi007.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2023-11-28T18:25:27.474 INFO:tasks.workunit.client.0.smithi007.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2023-11-28T18:25:27.474 INFO:tasks.workunit.client.0.smithi007.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:4: the top level 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:6: You should run autoupdate. 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:6: the top level 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:10: You should run autoupdate. 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:10: the top level 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:36: You should run autoupdate. 2023-11-28T18:25:27.475 INFO:tasks.workunit.client.0.smithi007.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:36: the top level 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:42: You should run autoupdate. 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:42: the top level 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:45: the top level 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2023-11-28T18:25:27.476 INFO:tasks.workunit.client.0.smithi007.stderr:configure.in:50: You should run autoupdate. 2023-11-28T18:25:27.561 INFO:tasks.workunit.client.0.smithi007.stdout:/bin/bash ./config.status --recheck 2023-11-28T18:25:27.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:25:27.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:25:27.729 INFO:tasks.workunit.client.0.smithi007.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2023-11-28T18:25:28.186 INFO:tasks.workunit.client.0.smithi007.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-11-28T18:25:28.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:28.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2023-11-28T18:25:28.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.406+0000 7f88f472a640 1 Processor -- start 2023-11-28T18:25:28.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.406+0000 7f88f472a640 1 -- start start 2023-11-28T18:25:28.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f472a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88ec151cd0 0x7f88ec15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:28.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f472a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec06e560 0x7f88ec157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:28.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f472a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f88ec0701f0 0x7f88ec0c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:28.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f472a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec071e60 con 0x7f88ec151cd0 2023-11-28T18:25:28.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f472a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec070930 con 0x7f88ec0701f0 2023-11-28T18:25:28.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f472a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec070660 con 0x7f88ec06e560 2023-11-28T18:25:28.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f24c6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88ec151cd0 0x7f88ec15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:28.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f24c6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88ec151cd0 0x7f88ec15bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39454/0 (socket says 172.21.15.7:39454) 2023-11-28T18:25:28.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f24c6640 1 -- 172.21.15.7:0/877360229 learned_addr learned my addr 172.21.15.7:0/877360229 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:28.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f1cc5640 1 --2- 172.21.15.7:0/877360229 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec06e560 0x7f88ec157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f2cc7640 1 --2- 172.21.15.7:0/877360229 >> v2:172.21.15.193:3300/0 conn(0x7f88ec0701f0 0x7f88ec0c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f24c6640 1 -- 172.21.15.7:0/877360229 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec06e560 msgr2=0x7f88ec157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f24c6640 1 --2- 172.21.15.7:0/877360229 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec06e560 0x7f88ec157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f24c6640 1 -- 172.21.15.7:0/877360229 >> v2:172.21.15.193:3300/0 conn(0x7f88ec0701f0 msgr2=0x7f88ec0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:28.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f24c6640 1 --2- 172.21.15.7:0/877360229 >> v2:172.21.15.193:3300/0 conn(0x7f88ec0701f0 0x7f88ec0c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.410+0000 7f88f24c6640 1 -- 172.21.15.7:0/877360229 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec070bf0 con 0x7f88ec151cd0 2023-11-28T18:25:28.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f24c6640 1 --2- 172.21.15.7:0/877360229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec151cd0 0x7f88ec15bd00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0c95f0 tx=0x7f88e801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd7a2c5c4a37f2b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:28.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f14c4640 1 -- 172.21.15.7:0/877360229 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e80ab600 con 0x7f88ec151cd0 2023-11-28T18:25:28.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f14c4640 1 -- 172.21.15.7:0/877360229 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e80a7050 con 0x7f88ec151cd0 2023-11-28T18:25:28.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f14c4640 1 -- 172.21.15.7:0/877360229 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e80ab980 con 0x7f88ec151cd0 2023-11-28T18:25:28.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 -- 172.21.15.7:0/877360229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec151cd0 msgr2=0x7f88ec15bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:28.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 --2- 172.21.15.7:0/877360229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec151cd0 0x7f88ec15bd00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0c95f0 tx=0x7f88e801c470 comp rx=0 tx=0).stop 2023-11-28T18:25:28.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 -- 172.21.15.7:0/877360229 shutdown_connections 2023-11-28T18:25:28.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 --2- 172.21.15.7:0/877360229 >> v2:172.21.15.193:3300/0 conn(0x7f88ec0701f0 0x7f88ec0c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 --2- 172.21.15.7:0/877360229 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec06e560 0x7f88ec157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 --2- 172.21.15.7:0/877360229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec151cd0 0x7f88ec15bd00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 -- 172.21.15.7:0/877360229 >> 172.21.15.7:0/877360229 conn(0x7f88ec0662e0 msgr2=0x7f88ec0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:28.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 -- 172.21.15.7:0/877360229 shutdown_connections 2023-11-28T18:25:28.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 -- 172.21.15.7:0/877360229 wait complete. 2023-11-28T18:25:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 Processor -- start 2023-11-28T18:25:28.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.414+0000 7f88f472a640 1 -- start start 2023-11-28T18:25:28.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f472a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec06e560 0x7f88ec154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:28.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f472a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ec0701f0 0x7f88ec150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:28.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f472a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec151cd0 0x7f88ec14c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:28.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f472a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec14ae60 con 0x7f88ec06e560 2023-11-28T18:25:28.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f472a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec14b2f0 con 0x7f88ec0701f0 2023-11-28T18:25:28.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f472a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1f1730 con 0x7f88ec151cd0 2023-11-28T18:25:28.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec06e560 0x7f88ec154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:28.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec06e560 0x7f88ec154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39462/0 (socket says 172.21.15.7:39462) 2023-11-28T18:25:28.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 -- 172.21.15.7:0/4157911949 learned_addr learned my addr 172.21.15.7:0/4157911949 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:28.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f2cc7640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec151cd0 0x7f88ec14c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:28.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f1cc5640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ec0701f0 0x7f88ec150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:28.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 -- 172.21.15.7:0/4157911949 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec151cd0 msgr2=0x7f88ec14c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec151cd0 0x7f88ec14c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 -- 172.21.15.7:0/4157911949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ec0701f0 msgr2=0x7f88ec150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ec0701f0 0x7f88ec150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 -- 172.21.15.7:0/4157911949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e80094d0 con 0x7f88ec06e560 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f24c6640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec06e560 0x7f88ec154f40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f88e801cd10 tx=0x7f88e8019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88e2ffd640 1 -- 172.21.15.7:0/4157911949 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e80b4620 con 0x7f88ec06e560 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88e2ffd640 1 -- 172.21.15.7:0/4157911949 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e80b47a0 con 0x7f88ec06e560 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88e2ffd640 1 -- 172.21.15.7:0/4157911949 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e80b4b50 con 0x7f88ec06e560 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f472a640 1 -- 172.21.15.7:0/4157911949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec14a250 con 0x7f88ec06e560 2023-11-28T18:25:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88f472a640 1 -- 172.21.15.7:0/4157911949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec14a810 con 0x7f88ec06e560 2023-11-28T18:25:28.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88e0ff9640 1 -- 172.21.15.7:0/4157911949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c8005260 con 0x7f88ec06e560 2023-11-28T18:25:28.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.418+0000 7f88e2ffd640 1 -- 172.21.15.7:0/4157911949 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f88e80b2070 con 0x7f88ec06e560 2023-11-28T18:25:28.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.422+0000 7f88e2ffd640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88d00668d0 0x7f88d0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:28.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.422+0000 7f88e2ffd640 1 -- 172.21.15.7:0/4157911949 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f88e811d720 con 0x7f88ec06e560 2023-11-28T18:25:28.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.422+0000 7f88f1cc5640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88d00668d0 0x7f88d0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:28.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.422+0000 7f88f1cc5640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88d00668d0 0x7f88d0068d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88ec1593b0 tx=0x7f88dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:28.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.426+0000 7f88e2ffd640 1 -- 172.21.15.7:0/4157911949 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f88e80ea860 con 0x7f88ec06e560 2023-11-28T18:25:28.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.614+0000 7f88e0ff9640 1 -- 172.21.15.7:0/4157911949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88c8005a90 con 0x7f88ec06e560 2023-11-28T18:25:28.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e2ffd640 1 -- 172.21.15.7:0/4157911949 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f88e80b0090 con 0x7f88ec06e560 2023-11-28T18:25:28.620 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:28.620 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:23.706668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:28.620 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 19 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 -- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88d00668d0 msgr2=0x7f88d0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88d00668d0 0x7f88d0068d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88ec1593b0 tx=0x7f88dc008040 comp rx=0 tx=0).stop 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 -- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec06e560 msgr2=0x7f88ec154f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec06e560 0x7f88ec154f40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f88e801cd10 tx=0x7f88e8019210 comp rx=0 tx=0).stop 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 -- 172.21.15.7:0/4157911949 shutdown_connections 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88d00668d0 0x7f88d0068d90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88ec151cd0 0x7f88ec14c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ec0701f0 0x7f88ec150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 --2- 172.21.15.7:0/4157911949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88ec06e560 0x7f88ec154f40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 -- 172.21.15.7:0/4157911949 >> 172.21.15.7:0/4157911949 conn(0x7f88ec0662e0 msgr2=0x7f88ec1577e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 -- 172.21.15.7:0/4157911949 shutdown_connections 2023-11-28T18:25:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:28.618+0000 7f88e0ff9640 1 -- 172.21.15.7:0/4157911949 wait complete. 2023-11-28T18:25:28.641 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:25:29.209 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether build environment is sane... yes 2023-11-28T18:25:29.230 INFO:tasks.workunit.client.0.smithi007.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2023-11-28T18:25:29.231 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gawk... gawk 2023-11-28T18:25:29.265 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether make sets $(MAKE)... yes 2023-11-28T18:25:29.365 INFO:tasks.workunit.client.0.smithi007.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-11-28T18:25:29.365 INFO:tasks.workunit.client.0.smithi007.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-11-28T18:25:29.395 INFO:tasks.workunit.client.0.smithi007.stdout:checking for style of include used by make... GNU 2023-11-28T18:25:29.409 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc... gcc 2023-11-28T18:25:30.142 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether the C compiler works... yes 2023-11-28T18:25:30.142 INFO:tasks.workunit.client.0.smithi007.stdout:checking for C compiler default output file name... a.out 2023-11-28T18:25:30.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:30.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65eb7d8640 1 Processor -- start 2023-11-28T18:25:30.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65eb7d8640 1 -- start start 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65eb7d8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 0x7f65e415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65eb7d8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65e40cbb70 0x7f65e40cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65eb7d8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f65e40cc810 0x7f65e4072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65eb7d8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e4071d50 con 0x7f65e40cbb70 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65eb7d8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e4072030 con 0x7f65e40cc810 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65eb7d8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40ca920 con 0x7f65e41542e0 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e8d73640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65e40cbb70 0x7f65e40cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e8d73640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65e40cbb70 0x7f65e40cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39476/0 (socket says 172.21.15.7:39476) 2023-11-28T18:25:30.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e9d75640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f65e40cc810 0x7f65e4072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:30.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e8d73640 1 -- 172.21.15.7:0/3365534094 learned_addr learned my addr 172.21.15.7:0/3365534094 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:30.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e9574640 1 --2- 172.21.15.7:0/3365534094 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 0x7f65e415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:30.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e8d73640 1 -- 172.21.15.7:0/3365534094 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 msgr2=0x7f65e415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:30.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e8d73640 1 --2- 172.21.15.7:0/3365534094 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 0x7f65e415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:30.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e8d73640 1 -- 172.21.15.7:0/3365534094 >> v2:172.21.15.193:3300/0 conn(0x7f65e40cc810 msgr2=0x7f65e4072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:30.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e8d73640 1 --2- 172.21.15.7:0/3365534094 >> v2:172.21.15.193:3300/0 conn(0x7f65e40cc810 0x7f65e4072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:30.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.794+0000 7f65e8d73640 1 -- 172.21.15.7:0/3365534094 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e415e550 con 0x7f65e40cbb70 2023-11-28T18:25:30.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65e8d73640 1 --2- 172.21.15.7:0/3365534094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cbb70 0x7f65e40cc2d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f65e4066fc0 tx=0x7f65d401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a44f1ea1fcc64c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:30.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65dbfff640 1 -- 172.21.15.7:0/3365534094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d40ac630 con 0x7f65e40cbb70 2023-11-28T18:25:30.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65dbfff640 1 -- 172.21.15.7:0/3365534094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d40ac790 con 0x7f65e40cbb70 2023-11-28T18:25:30.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65dbfff640 1 -- 172.21.15.7:0/3365534094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d40acb20 con 0x7f65e40cbb70 2023-11-28T18:25:30.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 -- 172.21.15.7:0/3365534094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cbb70 msgr2=0x7f65e40cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:30.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/3365534094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cbb70 0x7f65e40cc2d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f65e4066fc0 tx=0x7f65d401c500 comp rx=0 tx=0).stop 2023-11-28T18:25:30.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 -- 172.21.15.7:0/3365534094 shutdown_connections 2023-11-28T18:25:30.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/3365534094 >> v2:172.21.15.193:3300/0 conn(0x7f65e40cc810 0x7f65e4072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:30.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/3365534094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cbb70 0x7f65e40cc2d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:30.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/3365534094 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 0x7f65e415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:30.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 -- 172.21.15.7:0/3365534094 >> 172.21.15.7:0/3365534094 conn(0x7f65e40717e0 msgr2=0x7f65e40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:30.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 -- 172.21.15.7:0/3365534094 shutdown_connections 2023-11-28T18:25:30.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.798+0000 7f65eb7d8640 1 -- 172.21.15.7:0/3365534094 wait complete. 2023-11-28T18:25:30.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 Processor -- start 2023-11-28T18:25:30.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 -- start start 2023-11-28T18:25:30.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e40cbb70 0x7f65e40d0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:30.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cc810 0x7f65e40c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 0x7f65e415d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40c9b10 con 0x7f65e40cc810 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40cfac0 con 0x7f65e40cbb70 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40cfda0 con 0x7f65e41542e0 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cc810 0x7f65e40c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cc810 0x7f65e40c95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39480/0 (socket says 172.21.15.7:39480) 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 -- 172.21.15.7:0/356486560 learned_addr learned my addr 172.21.15.7:0/356486560 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e9574640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e40cbb70 0x7f65e40d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e9d75640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 0x7f65e415d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 -- 172.21.15.7:0/356486560 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 msgr2=0x7f65e415d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 0x7f65e415d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 -- 172.21.15.7:0/356486560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e40cbb70 msgr2=0x7f65e40d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:30.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e40cbb70 0x7f65e40d0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:30.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 -- 172.21.15.7:0/356486560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40094d0 con 0x7f65e40cc810 2023-11-28T18:25:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65e8d73640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cc810 0x7f65e40c95d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f65d40af040 tx=0x7f65e415b080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65d9ffb640 1 -- 172.21.15.7:0/356486560 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d40b7670 con 0x7f65e40cc810 2023-11-28T18:25:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65d9ffb640 1 -- 172.21.15.7:0/356486560 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d40b77d0 con 0x7f65e40cc810 2023-11-28T18:25:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65d9ffb640 1 -- 172.21.15.7:0/356486560 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d40b7b80 con 0x7f65e40cc810 2023-11-28T18:25:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e41f3ca0 con 0x7f65e40cc810 2023-11-28T18:25:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.802+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e40cc2d0 con 0x7f65e40cc810 2023-11-28T18:25:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.806+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e406c670 con 0x7f65e40cc810 2023-11-28T18:25:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.806+0000 7f65d9ffb640 1 -- 172.21.15.7:0/356486560 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f65d40b3070 con 0x7f65e40cc810 2023-11-28T18:25:30.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.806+0000 7f65d9ffb640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f65c80668d0 0x7f65c8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:30.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.806+0000 7f65d9ffb640 1 -- 172.21.15.7:0/356486560 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f65d411e9e0 con 0x7f65e40cc810 2023-11-28T18:25:30.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.806+0000 7f65e9574640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f65c80668d0 0x7f65c8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:30.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.806+0000 7f65e9574640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f65c80668d0 0x7f65c8068d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009ac0 tx=0x7f65e0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:30.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:30.810+0000 7f65d9ffb640 1 -- 172.21.15.7:0/356486560 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f65d40ebcd0 con 0x7f65e40cc810 2023-11-28T18:25:31.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65e40cff60 con 0x7f65e40cc810 2023-11-28T18:25:31.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65d9ffb640 1 -- 172.21.15.7:0/356486560 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6557 (secure 0 0 0) 0x7f65e40cff60 con 0x7f65e40cc810 2023-11-28T18:25:31.012 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:31.012 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:28.706376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:31.012 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 20 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f65c80668d0 msgr2=0x7f65c8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f65c80668d0 0x7f65c8068d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009ac0 tx=0x7f65e0009190 comp rx=0 tx=0).stop 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cc810 msgr2=0x7f65e40c95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cc810 0x7f65e40c95d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f65d40af040 tx=0x7f65e415b080 comp rx=0 tx=0).stop 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 shutdown_connections 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f65c80668d0 0x7f65c8068d90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f65e41542e0 0x7f65e415d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e40cc810 0x7f65e40c95d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 --2- 172.21.15.7:0/356486560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e40cbb70 0x7f65e40d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 >> 172.21.15.7:0/356486560 conn(0x7f65e40717e0 msgr2=0x7f65e40b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:31.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 shutdown_connections 2023-11-28T18:25:31.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:31.010+0000 7f65eb7d8640 1 -- 172.21.15.7:0/356486560 wait complete. 2023-11-28T18:25:31.032 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:25:31.097 INFO:tasks.workunit.client.0.smithi007.stdout:checking for suffix of executables... 2023-11-28T18:25:31.207 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are cross compiling... no 2023-11-28T18:25:31.288 INFO:tasks.workunit.client.0.smithi007.stdout:checking for suffix of object files... o 2023-11-28T18:25:31.362 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether the compiler supports GNU C... yes 2023-11-28T18:25:31.856 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether gcc accepts -g... yes 2023-11-28T18:25:31.986 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc option to enable C11 features... none needed 2023-11-28T18:25:33.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1b6417640 1 Processor -- start 2023-11-28T18:25:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1b6417640 1 -- start start 2023-11-28T18:25:33.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1b6417640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1b0072950 0x7fe1b0074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:33.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1b6417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:33.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1b6417640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b00c8490 con 0x7fe1b0073070 2023-11-28T18:25:33.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1b6417640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b006d2e0 con 0x7fe1b0072950 2023-11-28T18:25:33.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1b6417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00c81f0 con 0x7fe1b006c920 2023-11-28T18:25:33.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1affff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1b0072950 0x7fe1b0074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:33.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1affff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1b0072950 0x7fe1b0074030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39490/0 (socket says 172.21.15.7:39490) 2023-11-28T18:25:33.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.194+0000 7fe1affff640 1 -- 172.21.15.7:0/2438299197 learned_addr learned my addr 172.21.15.7:0/2438299197 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:33.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1b49b4640 1 --2- 172.21.15.7:0/2438299197 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:33.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1affff640 1 -- 172.21.15.7:0/2438299197 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 msgr2=0x7fe1b006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:33.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1affff640 1 --2- 172.21.15.7:0/2438299197 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1affff640 1 -- 172.21.15.7:0/2438299197 >> v1:172.21.15.193:6789/0 conn(0x7fe1b0073070 legacy=0x7fe1b00c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:33.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1affff640 1 -- 172.21.15.7:0/2438299197 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b015ab00 con 0x7fe1b0072950 2023-11-28T18:25:33.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1affff640 1 --2- 172.21.15.7:0/2438299197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b0074030 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009870 tx=0x7fe1a001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3973816c369ead18 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1aeffd640 1 -- 172.21.15.7:0/2438299197 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00ab600 con 0x7fe1b0072950 2023-11-28T18:25:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1aeffd640 1 -- 172.21.15.7:0/2438299197 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a00a7050 con 0x7fe1b0072950 2023-11-28T18:25:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.198+0000 7fe1aeffd640 1 -- 172.21.15.7:0/2438299197 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00ab980 con 0x7fe1b0072950 2023-11-28T18:25:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- 172.21.15.7:0/2438299197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 msgr2=0x7fe1b0074030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 --2- 172.21.15.7:0/2438299197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b0074030 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009870 tx=0x7fe1a001c420 comp rx=0 tx=0).stop 2023-11-28T18:25:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- 172.21.15.7:0/2438299197 shutdown_connections 2023-11-28T18:25:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 --2- 172.21.15.7:0/2438299197 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 --2- 172.21.15.7:0/2438299197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b0074030 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- 172.21.15.7:0/2438299197 >> 172.21.15.7:0/2438299197 conn(0x7fe1b00cbf80 msgr2=0x7fe1b00cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- 172.21.15.7:0/2438299197 shutdown_connections 2023-11-28T18:25:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- 172.21.15.7:0/2438299197 wait complete. 2023-11-28T18:25:33.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 Processor -- start 2023-11-28T18:25:33.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- start start 2023-11-28T18:25:33.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b0149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b014efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1b0073070 0x7fe1b0147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01f2500 con 0x7fe1b0072950 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01f3380 con 0x7fe1b0073070 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b6417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01f2770 con 0x7fe1b006c920 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1af7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b014efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1af7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b014efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39500/0 (socket says 172.21.15.7:39500) 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1affff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b0149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1af7fe640 1 -- 172.21.15.7:0/3093729176 learned_addr learned my addr 172.21.15.7:0/3093729176 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.202+0000 7fe1b49b4640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1b0073070 0x7fe1b0147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:33.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1af7fe640 1 -- 172.21.15.7:0/3093729176 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 msgr2=0x7fe1b0149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1af7fe640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b0149c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1af7fe640 1 -- 172.21.15.7:0/3093729176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1b0073070 msgr2=0x7fe1b0147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1af7fe640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1b0073070 0x7fe1b0147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1af7fe640 1 -- 172.21.15.7:0/3093729176 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00094d0 con 0x7fe1b0072950 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1affff640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b0149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1af7fe640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b014efb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe1a400c3c0 tx=0x7fe1a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1acff9640 1 -- 172.21.15.7:0/3093729176 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40052d0 con 0x7fe1b0072950 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1acff9640 1 -- 172.21.15.7:0/3093729176 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a4005430 con 0x7fe1b0072950 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b01f2a50 con 0x7fe1b0072950 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1acff9640 1 -- 172.21.15.7:0/3093729176 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4003820 con 0x7fe1b0072950 2023-11-28T18:25:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b01f3010 con 0x7fe1b0072950 2023-11-28T18:25:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b004e890 con 0x7fe1b0072950 2023-11-28T18:25:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1acff9640 1 -- 172.21.15.7:0/3093729176 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe1a4003980 con 0x7fe1b0072950 2023-11-28T18:25:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1acff9640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe19c0668d0 0x7fe19c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:33.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.206+0000 7fe1acff9640 1 -- 172.21.15.7:0/3093729176 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe1a4083a30 con 0x7fe1b0072950 2023-11-28T18:25:33.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.210+0000 7fe1affff640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe19c0668d0 0x7fe19c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:33.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.210+0000 7fe1affff640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe19c0668d0 0x7fe19c068d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00ad000 tx=0x7fe1a00a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:33.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.210+0000 7fe1acff9640 1 -- 172.21.15.7:0/3093729176 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe1a4050b70 con 0x7fe1b0072950 2023-11-28T18:25:33.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.402+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1b014dd20 con 0x7fe1b0072950 2023-11-28T18:25:33.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1acff9640 1 -- 172.21.15.7:0/3093729176 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6557 (secure 0 0 0) 0x7fe1a4017020 con 0x7fe1b0072950 2023-11-28T18:25:33.407 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:33.407 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:28.706376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:33.408 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 20 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe19c0668d0 msgr2=0x7fe19c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe19c0668d0 0x7fe19c068d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00ad000 tx=0x7fe1a00a6000 comp rx=0 tx=0).stop 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 msgr2=0x7fe1b014efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b014efb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe1a400c3c0 tx=0x7fe1a400c880 comp rx=0 tx=0).stop 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 shutdown_connections 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe19c0668d0 0x7fe19c068d90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1b0073070 0x7fe1b0147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1b0072950 0x7fe1b014efb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 --2- 172.21.15.7:0/3093729176 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1b006c920 0x7fe1b0149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:33.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 >> 172.21.15.7:0/3093729176 conn(0x7fe1b00cbf80 msgr2=0x7fe1b00c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:33.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 shutdown_connections 2023-11-28T18:25:33.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:33.406+0000 7fe1b6417640 1 -- 172.21.15.7:0/3093729176 wait complete. 2023-11-28T18:25:33.428 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:25:33.594 INFO:tasks.workunit.client.0.smithi007.stdout:checking dependency style of gcc... gcc3 2023-11-28T18:25:33.667 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdio.h... yes 2023-11-28T18:25:33.748 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdlib.h... yes 2023-11-28T18:25:34.423 INFO:tasks.workunit.client.0.smithi007.stdout:checking for string.h... yes 2023-11-28T18:25:34.663 INFO:tasks.workunit.client.0.smithi007.stdout:checking for inttypes.h... yes 2023-11-28T18:25:34.746 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdint.h... yes 2023-11-28T18:25:34.835 INFO:tasks.workunit.client.0.smithi007.stdout:checking for strings.h... yes 2023-11-28T18:25:34.919 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/stat.h... yes 2023-11-28T18:25:35.002 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/types.h... yes 2023-11-28T18:25:35.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.574+0000 7fa1eec17640 1 Processor -- start 2023-11-28T18:25:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.574+0000 7fa1eec17640 1 -- start start 2023-11-28T18:25:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1eec17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1eec17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa1e806e560 0x7fa1e8157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1eec17640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e8071e60 con 0x7fa1e80701f0 2023-11-28T18:25:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1eec17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e8070930 con 0x7fa1e806e560 2023-11-28T18:25:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1eec17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e8070660 con 0x7fa1e8151cd0 2023-11-28T18:25:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1ec9b3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1ed1b4640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa1e80701f0 0x7fa1e80c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:42536/0 (socket says 172.21.15.7:42536) 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1ed1b4640 1 -- 172.21.15.7:0/1986244380 learned_addr learned my addr 172.21.15.7:0/1986244380 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1e7fff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa1e806e560 0x7fa1e8157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1e77fe640 1 -- 172.21.15.7:0/1986244380 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559901124 0 0) 0x7fa1e8071e60 con 0x7fa1e80701f0 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1e77fe640 1 -- 172.21.15.7:0/1986244380 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8003530 con 0x7fa1e80701f0 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1e7fff640 1 -- 172.21.15.7:0/1986244380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 msgr2=0x7fa1e815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1e7fff640 1 --2- 172.21.15.7:0/1986244380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1e7fff640 1 -- 172.21.15.7:0/1986244380 >> v1:172.21.15.193:6789/0 conn(0x7fa1e80701f0 legacy=0x7fa1e80c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1e7fff640 1 -- 172.21.15.7:0/1986244380 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e8070bf0 con 0x7fa1e806e560 2023-11-28T18:25:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.578+0000 7fa1ec9b3640 1 --2- 172.21.15.7:0/1986244380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:25:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1e7fff640 1 --2- 172.21.15.7:0/1986244380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e8157b90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40097c0 tx=0x7fa1d401c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81ed90cc1fc68fc9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1e77fe640 1 -- 172.21.15.7:0/1986244380 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d40ab600 con 0x7fa1e806e560 2023-11-28T18:25:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1e77fe640 1 -- 172.21.15.7:0/1986244380 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d40a7050 con 0x7fa1e806e560 2023-11-28T18:25:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1e77fe640 1 -- 172.21.15.7:0/1986244380 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d40ab980 con 0x7fa1e806e560 2023-11-28T18:25:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 -- 172.21.15.7:0/1986244380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 msgr2=0x7fa1e8157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 --2- 172.21.15.7:0/1986244380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e8157b90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40097c0 tx=0x7fa1d401c3f0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 -- 172.21.15.7:0/1986244380 shutdown_connections 2023-11-28T18:25:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 --2- 172.21.15.7:0/1986244380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e8157b90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 --2- 172.21.15.7:0/1986244380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 -- 172.21.15.7:0/1986244380 >> 172.21.15.7:0/1986244380 conn(0x7fa1e80662e0 msgr2=0x7fa1e80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 -- 172.21.15.7:0/1986244380 shutdown_connections 2023-11-28T18:25:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 -- 172.21.15.7:0/1986244380 wait complete. 2023-11-28T18:25:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 Processor -- start 2023-11-28T18:25:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 -- start start 2023-11-28T18:25:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e80c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e80701f0 0x7fa1e80c4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.582+0000 7fa1eec17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e81506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1eec17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80c8d90 con 0x7fa1e80701f0 2023-11-28T18:25:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1eec17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e814fa00 con 0x7fa1e806e560 2023-11-28T18:25:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1eec17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80c86d0 con 0x7fa1e8151cd0 2023-11-28T18:25:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1ec9b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e80c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e80701f0 0x7fa1e80c4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1ec9b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e80c7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:45014/0 (socket says 172.21.15.7:45014) 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1ec9b3640 1 -- 172.21.15.7:0/851080991 learned_addr learned my addr 172.21.15.7:0/851080991 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1ed1b4640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e81506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e7fff640 1 -- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 msgr2=0x7fa1e81506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e7fff640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e81506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e7fff640 1 -- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 msgr2=0x7fa1e80c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e7fff640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e80c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e7fff640 1 -- 172.21.15.7:0/851080991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc009420 con 0x7fa1e80701f0 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1ed1b4640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e81506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:25:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1ec9b3640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e80c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:25:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e7fff640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e80701f0 0x7fa1e80c4020 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40ab000 tx=0x7fa1d4019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e57fa640 1 -- 172.21.15.7:0/851080991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d40b4410 con 0x7fa1e80701f0 2023-11-28T18:25:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e57fa640 1 -- 172.21.15.7:0/851080991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d40b4570 con 0x7fa1e80701f0 2023-11-28T18:25:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d4009420 con 0x7fa1e80701f0 2023-11-28T18:25:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e57fa640 1 -- 172.21.15.7:0/851080991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d40b4920 con 0x7fa1e80701f0 2023-11-28T18:25:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e80c7e30 con 0x7fa1e80701f0 2023-11-28T18:25:35.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e804e890 con 0x7fa1e80701f0 2023-11-28T18:25:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.586+0000 7fa1e57fa640 1 -- 172.21.15.7:0/851080991 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa1d40b2040 con 0x7fa1e80701f0 2023-11-28T18:25:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.590+0000 7fa1e57fa640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1cc066900 0x7fa1cc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.590+0000 7fa1e57fa640 1 -- 172.21.15.7:0/851080991 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa1d411e360 con 0x7fa1e80701f0 2023-11-28T18:25:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.590+0000 7fa1ec9b3640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1cc066900 0x7fa1cc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.590+0000 7fa1ec9b3640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1cc066900 0x7fa1cc068dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0114b0 tx=0x7fa1dc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:35.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.594+0000 7fa1e57fa640 1 -- 172.21.15.7:0/851080991 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa1d40eb650 con 0x7fa1e80701f0 2023-11-28T18:25:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.778+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1e81555b0 con 0x7fa1e80701f0 2023-11-28T18:25:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1e57fa640 1 -- 172.21.15.7:0/851080991 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6557 (secure 0 0 0) 0x7fa1d40eec70 con 0x7fa1e80701f0 2023-11-28T18:25:35.783 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:35.784 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:28.706376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.193:6836/1372595381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1372595381},{"type":"v1","addr":"172.21.15.193:6836","nonce":1372595381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:35.784 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 20 2023-11-28T18:25:35.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1cc066900 msgr2=0x7fa1cc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:35.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1cc066900 0x7fa1cc068dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0114b0 tx=0x7fa1dc009190 comp rx=0 tx=0).stop 2023-11-28T18:25:35.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e80701f0 msgr2=0x7fa1e80c4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:35.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e80701f0 0x7fa1e80c4020 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40ab000 tx=0x7fa1d4019190 comp rx=0 tx=0).stop 2023-11-28T18:25:35.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 shutdown_connections 2023-11-28T18:25:35.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1cc066900 0x7fa1cc068dc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa1e8151cd0 0x7fa1e81506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e80701f0 0x7fa1e80c4020 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 --2- 172.21.15.7:0/851080991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e806e560 0x7fa1e80c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:35.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 >> 172.21.15.7:0/851080991 conn(0x7fa1e80662e0 msgr2=0x7fa1e80cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:35.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 shutdown_connections 2023-11-28T18:25:35.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:35.782+0000 7fa1eec17640 1 -- 172.21.15.7:0/851080991 wait complete. 2023-11-28T18:25:35.804 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:25:35.880 INFO:tasks.workunit.client.0.smithi007.stdout:checking for unistd.h... yes 2023-11-28T18:25:35.998 INFO:tasks.workunit.client.0.smithi007.stdout:checking for wchar.h... yes 2023-11-28T18:25:36.098 INFO:tasks.workunit.client.0.smithi007.stdout:checking for minix/config.h... no 2023-11-28T18:25:36.182 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/time.h... yes 2023-11-28T18:25:36.277 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2023-11-28T18:25:36.357 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether _XOPEN_SOURCE should be defined... no 2023-11-28T18:25:36.375 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc... (cached) gcc 2023-11-28T18:25:37.295 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether the compiler supports GNU C... (cached) yes 2023-11-28T18:25:37.295 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether gcc accepts -g... (cached) yes 2023-11-28T18:25:37.295 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc option to enable C11 features... (cached) none needed 2023-11-28T18:25:37.296 INFO:tasks.workunit.client.0.smithi007.stdout:checking dependency style of gcc... (cached) gcc3 2023-11-28T18:25:37.771 INFO:tasks.workunit.client.0.smithi007.stdout:checking for main in -lm... yes 2023-11-28T18:25:37.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:37.920 INFO:tasks.workunit.client.0.smithi007.stdout:checking for main in -lpthread... yes 2023-11-28T18:25:37.926 INFO:tasks.workunit.client.0.smithi007.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-11-28T18:25:37.930 INFO:tasks.workunit.client.0.smithi007.stdout:checking for egrep... /usr/bin/grep -E 2023-11-28T18:25:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f7d89a640 1 Processor -- start 2023-11-28T18:25:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f7d89a640 1 -- start start 2023-11-28T18:25:37.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f7d89a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9f780cb020 0x7f9f780ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:37.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f7d89a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f780cb5a0 0x7f9f780cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:37.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f7d89a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7815d420 con 0x7f9f78153f00 2023-11-28T18:25:37.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f7d89a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780ca900 con 0x7f9f780cb020 2023-11-28T18:25:37.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f7d89a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7815d700 con 0x7f9f780cb5a0 2023-11-28T18:25:37.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f76ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9f78153f00 0x7f9f780b1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51666/0 (socket says 172.21.15.7:51666) 2023-11-28T18:25:37.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f76ffd640 1 -- 172.21.15.7:0/2497409560 learned_addr learned my addr 172.21.15.7:0/2497409560 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:25:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f777fe640 1 --2- 172.21.15.7:0/2497409560 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f780cb5a0 0x7f9f780cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f767fc640 1 --2- 172.21.15.7:0/2497409560 >> v2:172.21.15.193:3300/0 conn(0x7f9f780cb020 0x7f9f780ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f75ffb640 1 -- 172.21.15.7:0/2497409560 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213725710 0 0) 0x7f9f7815d420 con 0x7f9f78153f00 2023-11-28T18:25:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f75ffb640 1 -- 172.21.15.7:0/2497409560 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f60003530 con 0x7f9f78153f00 2023-11-28T18:25:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f767fc640 1 -- 172.21.15.7:0/2497409560 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f780cb5a0 msgr2=0x7f9f780cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f767fc640 1 --2- 172.21.15.7:0/2497409560 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f780cb5a0 0x7f9f780cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f767fc640 1 -- 172.21.15.7:0/2497409560 >> v1:172.21.15.7:6789/0 conn(0x7f9f78153f00 legacy=0x7f9f780b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.970+0000 7f9f767fc640 1 -- 172.21.15.7:0/2497409560 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7815d9e0 con 0x7f9f780cb020 2023-11-28T18:25:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f767fc640 1 --2- 172.21.15.7:0/2497409560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 0x7f9f780ca250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c0097c0 tx=0x7f9f6c01c290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d8e84fa193f104b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f75ffb640 1 -- 172.21.15.7:0/2497409560 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0a4ea0 con 0x7f9f780cb020 2023-11-28T18:25:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f75ffb640 1 -- 172.21.15.7:0/2497409560 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c0a7050 con 0x7f9f780cb020 2023-11-28T18:25:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f75ffb640 1 -- 172.21.15.7:0/2497409560 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f680016a0 con 0x7f9f780cb020 2023-11-28T18:25:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f7d89a640 1 -- 172.21.15.7:0/2497409560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 msgr2=0x7f9f780ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/2497409560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 0x7f9f780ca250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c0097c0 tx=0x7f9f6c01c290 comp rx=0 tx=0).stop 2023-11-28T18:25:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f7d89a640 1 -- 172.21.15.7:0/2497409560 shutdown_connections 2023-11-28T18:25:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/2497409560 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f780cb5a0 0x7f9f780cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:37.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/2497409560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 0x7f9f780ca250 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:37.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f7d89a640 1 -- 172.21.15.7:0/2497409560 >> 172.21.15.7:0/2497409560 conn(0x7f9f7806c060 msgr2=0x7f9f780662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:37.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f7d89a640 1 -- 172.21.15.7:0/2497409560 shutdown_connections 2023-11-28T18:25:37.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.974+0000 7f9f7d89a640 1 -- 172.21.15.7:0/2497409560 wait complete. 2023-11-28T18:25:37.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 Processor -- start 2023-11-28T18:25:37.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 -- start start 2023-11-28T18:25:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 0x7f9f7806ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f780cb5a0 0x7f9f781fa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f78153f00 0x7f9f781fa560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f780c8aa0 con 0x7f9f780cb5a0 2023-11-28T18:25:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f780709e0 con 0x7f9f780cb020 2023-11-28T18:25:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78070330 con 0x7f9f78153f00 2023-11-28T18:25:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f780cb5a0 0x7f9f781fa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f780cb5a0 0x7f9f781fa020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38686/0 (socket says 172.21.15.7:38686) 2023-11-28T18:25:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 -- 172.21.15.7:0/3657431446 learned_addr learned my addr 172.21.15.7:0/3657431446 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f76ffd640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 0x7f9f7806ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f777fe640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f78153f00 0x7f9f781fa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 -- 172.21.15.7:0/3657431446 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f78153f00 msgr2=0x7f9f781fa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f78153f00 0x7f9f781fa560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 -- 172.21.15.7:0/3657431446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 msgr2=0x7f9f7806ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 0x7f9f7806ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 -- 172.21.15.7:0/3657431446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c009420 con 0x7f9f780cb5a0 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f767fc640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f780cb5a0 0x7f9f781fa020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c01cb30 tx=0x7f9f6c019030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f5ffff640 1 -- 172.21.15.7:0/3657431446 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0ba070 con 0x7f9f780cb5a0 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f5ffff640 1 -- 172.21.15.7:0/3657431446 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c0a4ea0 con 0x7f9f780cb5a0 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f780c7e60 con 0x7f9f780cb5a0 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f5ffff640 1 -- 172.21.15.7:0/3657431446 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c006e20 con 0x7f9f780cb5a0 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f780c83e0 con 0x7f9f780cb5a0 2023-11-28T18:25:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.978+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7804e890 con 0x7f9f780cb5a0 2023-11-28T18:25:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.982+0000 7f9f5ffff640 1 -- 172.21.15.7:0/3657431446 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9f6c0b2040 con 0x7f9f780cb5a0 2023-11-28T18:25:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.982+0000 7f9f5ffff640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9f640668d0 0x7f9f64068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.982+0000 7f9f5ffff640 1 -- 172.21.15.7:0/3657431446 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f9f6c11dab0 con 0x7f9f780cb5a0 2023-11-28T18:25:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.982+0000 7f9f76ffd640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9f640668d0 0x7f9f64068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.982+0000 7f9f76ffd640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9f640668d0 0x7f9f64068d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9f68000fc0 tx=0x7f9f68001ce0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:37.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:37.986+0000 7f9f5ffff640 1 -- 172.21.15.7:0/3657431446 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9f6c0eade0 con 0x7f9f780cb5a0 2023-11-28T18:25:38.030 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-11-28T18:25:38.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.190+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f780cae00 con 0x7f9f780cb5a0 2023-11-28T18:25:38.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.190+0000 7f9f5ffff640 1 -- 172.21.15.7:0/3657431446 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f9f6c0ee400 con 0x7f9f780cb5a0 2023-11-28T18:25:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:38.193 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:37.139805+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2023-11-28T18:25:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.190+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9f640668d0 msgr2=0x7f9f64068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.190+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9f640668d0 0x7f9f64068d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9f68000fc0 tx=0x7f9f68001ce0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.190+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f780cb5a0 msgr2=0x7f9f781fa020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.190+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f780cb5a0 0x7f9f781fa020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c01cb30 tx=0x7f9f6c019030 comp rx=0 tx=0).stop 2023-11-28T18:25:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.194+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 shutdown_connections 2023-11-28T18:25:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.194+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9f640668d0 0x7f9f64068d90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.194+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9f78153f00 0x7f9f781fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.194+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f780cb5a0 0x7f9f781fa020 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.194+0000 7f9f7d89a640 1 --2- 172.21.15.7:0/3657431446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f780cb020 0x7f9f7806ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.194+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 >> 172.21.15.7:0/3657431446 conn(0x7f9f7806c060 msgr2=0x7f9f780ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.194+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 shutdown_connections 2023-11-28T18:25:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.194+0000 7f9f7d89a640 1 -- 172.21.15.7:0/3657431446 wait complete. 2023-11-28T18:25:38.214 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-11-28T18:25:38.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:25:38.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:38.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.358+0000 7f2a22a6b640 1 Processor -- start 2023-11-28T18:25:38.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.358+0000 7f2a22a6b640 1 -- start start 2023-11-28T18:25:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.358+0000 7f2a22a6b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a1c0c8090 0x7f2a1c1575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.358+0000 7f2a22a6b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0c6140 con 0x7f2a1c0c24f0 2023-11-28T18:25:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.358+0000 7f2a22a6b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c157b00 con 0x7f2a1c150b30 2023-11-28T18:25:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.358+0000 7f2a22a6b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c0c72b0 con 0x7f2a1c0c8090 2023-11-28T18:25:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.358+0000 7f2a21008640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a1c0c8090 0x7f2a1c1575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:38.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a1bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2a1c0c24f0 0x7f2a1c0c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51670/0 (socket says 172.21.15.7:51670) 2023-11-28T18:25:38.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a1bfff640 1 -- 172.21.15.7:0/797011696 learned_addr learned my addr 172.21.15.7:0/797011696 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:25:38.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a21008640 1 -- 172.21.15.7:0/797011696 >> v1:172.21.15.193:6789/0 conn(0x7f2a1c150b30 legacy=0x7f2a1c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a21008640 1 -- 172.21.15.7:0/797011696 >> v1:172.21.15.7:6789/0 conn(0x7f2a1c0c24f0 legacy=0x7f2a1c0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a21008640 1 -- 172.21.15.7:0/797011696 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0c2960 con 0x7f2a1c0c8090 2023-11-28T18:25:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a1b7fe640 1 -- 172.21.15.7:0/797011696 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998336161 0 0) 0x7f2a1c157b00 con 0x7f2a1c150b30 2023-11-28T18:25:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a21008640 1 --2- 172.21.15.7:0/797011696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c0c8090 0x7f2a1c1575c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009d90 tx=0x7f2a1401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25088f0215d190f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a1b7fe640 1 -- 172.21.15.7:0/797011696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140a58d0 con 0x7f2a1c0c8090 2023-11-28T18:25:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a1b7fe640 1 -- 172.21.15.7:0/797011696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a140a5a30 con 0x7f2a1c0c8090 2023-11-28T18:25:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.362+0000 7f2a1b7fe640 1 -- 172.21.15.7:0/797011696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140a5dc0 con 0x7f2a1c0c8090 2023-11-28T18:25:38.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- 172.21.15.7:0/797011696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c0c8090 msgr2=0x7f2a1c1575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 --2- 172.21.15.7:0/797011696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c0c8090 0x7f2a1c1575c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009d90 tx=0x7f2a1401c870 comp rx=0 tx=0).stop 2023-11-28T18:25:38.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- 172.21.15.7:0/797011696 shutdown_connections 2023-11-28T18:25:38.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 --2- 172.21.15.7:0/797011696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c0c8090 0x7f2a1c1575c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- 172.21.15.7:0/797011696 >> 172.21.15.7:0/797011696 conn(0x7f2a1c0662e0 msgr2=0x7f2a1c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:38.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- 172.21.15.7:0/797011696 shutdown_connections 2023-11-28T18:25:38.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- 172.21.15.7:0/797011696 wait complete. 2023-11-28T18:25:38.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 Processor -- start 2023-11-28T18:25:38.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- start start 2023-11-28T18:25:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a1c0c24f0 0x7f2a1c1564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a1c0c8090 0x7f2a1c149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c150b30 0x7f2a1c144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c143430 con 0x7f2a1c150b30 2023-11-28T18:25:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c156a10 con 0x7f2a1c0c24f0 2023-11-28T18:25:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a22a6b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c1436f0 con 0x7f2a1c0c8090 2023-11-28T18:25:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a21008640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c150b30 0x7f2a1c144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a21008640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c150b30 0x7f2a1c144440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38694/0 (socket says 172.21.15.7:38694) 2023-11-28T18:25:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a21008640 1 -- 172.21.15.7:0/1178629181 learned_addr learned my addr 172.21.15.7:0/1178629181 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a20807640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a1c0c24f0 0x7f2a1c1564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a1bfff640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a1c0c8090 0x7f2a1c149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a21008640 1 -- 172.21.15.7:0/1178629181 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a1c0c8090 msgr2=0x7f2a1c149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a21008640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a1c0c8090 0x7f2a1c149de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a21008640 1 -- 172.21.15.7:0/1178629181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a1c0c24f0 msgr2=0x7f2a1c1564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a21008640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a1c0c24f0 0x7f2a1c1564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.366+0000 7f2a21008640 1 -- 172.21.15.7:0/1178629181 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a14009a70 con 0x7f2a1c150b30 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a21008640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c150b30 0x7f2a1c144440 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009d90 tx=0x7f2a14019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a197fa640 1 -- 172.21.15.7:0/1178629181 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140aed50 con 0x7f2a1c150b30 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a197fa640 1 -- 172.21.15.7:0/1178629181 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a140039a0 con 0x7f2a1c150b30 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a197fa640 1 -- 172.21.15.7:0/1178629181 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14003d10 con 0x7f2a1c150b30 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c154df0 con 0x7f2a1c150b30 2023-11-28T18:25:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c143960 con 0x7f2a1c150b30 2023-11-28T18:25:38.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c04e890 con 0x7f2a1c150b30 2023-11-28T18:25:38.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a197fa640 1 -- 172.21.15.7:0/1178629181 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2a140b4050 con 0x7f2a1c150b30 2023-11-28T18:25:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a197fa640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a00066900 0x7f2a00068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a197fa640 1 -- 172.21.15.7:0/1178629181 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f2a1411ed10 con 0x7f2a1c150b30 2023-11-28T18:25:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a20807640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a00066900 0x7f2a00068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.370+0000 7f2a20807640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a00066900 0x7f2a00068dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2a10007fd0 tx=0x7f2a10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:38.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.374+0000 7f2a197fa640 1 -- 172.21.15.7:0/1178629181 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2a140ebe10 con 0x7f2a1c150b30 2023-11-28T18:25:38.519 INFO:tasks.workunit.client.0.smithi007.stdout:checking for pthread.h... yes 2023-11-28T18:25:38.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.578+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a1c0c2d80 con 0x7f2a1c150b30 2023-11-28T18:25:38.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.578+0000 7f2a197fa640 1 -- 172.21.15.7:0/1178629181 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f2a140b2090 con 0x7f2a1c150b30 2023-11-28T18:25:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:38.582 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:37.139805+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:38.584 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2023-11-28T18:25:38.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a00066900 msgr2=0x7f2a00068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a00066900 0x7f2a00068dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2a10007fd0 tx=0x7f2a10009040 comp rx=0 tx=0).stop 2023-11-28T18:25:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c150b30 msgr2=0x7f2a1c144440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:38.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c150b30 0x7f2a1c144440 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009d90 tx=0x7f2a14019610 comp rx=0 tx=0).stop 2023-11-28T18:25:38.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 shutdown_connections 2023-11-28T18:25:38.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a00066900 0x7f2a00068dc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a1c150b30 0x7f2a1c144440 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a1c0c8090 0x7f2a1c149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 --2- 172.21.15.7:0/1178629181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a1c0c24f0 0x7f2a1c1564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 >> 172.21.15.7:0/1178629181 conn(0x7f2a1c0662e0 msgr2=0x7f2a1c14fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 shutdown_connections 2023-11-28T18:25:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:38.582+0000 7f2a22a6b640 1 -- 172.21.15.7:0/1178629181 wait complete. 2023-11-28T18:25:38.607 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4279": { "addr": "172.21.15.193:6835/4143265275", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 4143265275, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 4143265275, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 21, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4284": { "addr": "172.21.15.7:6835/1419029356", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1419029356, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1419029356, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4284, "incarnation": 13, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4285": { "addr": "172.21.15.193:6837/3911527535", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 3911527535, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 3911527535, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 11, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.7:6838/2690556675", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 2690556675, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 2690556675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 23, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:25:37.139805+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 4288, "mds_2": 4285, "mds_3": 4284, "mds_4": 4279 } } } ], "standbys": [] } 2023-11-28T18:25:38.618 INFO:tasks.workunit.client.0.smithi007.stdout:checking for fcntl.h... yes 2023-11-28T18:25:38.931 INFO:tasks.workunit.client.0.smithi007.stdout:checking for limits.h... yes 2023-11-28T18:25:38.933 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdint.h... (cached) yes 2023-11-28T18:25:38.935 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/time.h... (cached) yes 2023-11-28T18:25:38.937 INFO:tasks.workunit.client.0.smithi007.stdout:checking for unistd.h... (cached) yes 2023-11-28T18:25:39.631 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/vfs.h... yes 2023-11-28T18:25:39.733 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/limits.h... no 2023-11-28T18:25:40.550 INFO:tasks.workunit.client.0.smithi007.stdout:checking for an ANSI C-conforming const... yes 2023-11-28T18:25:40.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:40.698 INFO:tasks.workunit.client.0.smithi007.stdout:checking for system... yes 2023-11-28T18:25:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.758+0000 7fc63a0c8640 1 Processor -- start 2023-11-28T18:25:40.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.758+0000 7fc63a0c8640 1 -- start start 2023-11-28T18:25:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc63a0c8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6340ca330 0x7fc6340ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc63a0c8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63415d240 con 0x7fc634159c90 2023-11-28T18:25:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc63a0c8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63415d520 con 0x7fc634153ee0 2023-11-28T18:25:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc63a0c8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6340cbd10 con 0x7fc6340ca330 2023-11-28T18:25:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc6398c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6340ca330 0x7fc6340ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc6398c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6340ca330 0x7fc6340ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38700/0 (socket says 172.21.15.7:38700) 2023-11-28T18:25:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc6398c7640 1 -- 172.21.15.7:0/3899522840 learned_addr learned my addr 172.21.15.7:0/3899522840 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:40.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc6388c5640 1 --1- 172.21.15.7:0/3899522840 >> v1:172.21.15.7:6789/0 conn(0x7fc634159c90 0x7fc63415d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51684/0 (socket says 172.21.15.7:51684) 2023-11-28T18:25:40.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc6398c7640 1 -- 172.21.15.7:0/3899522840 >> v1:172.21.15.193:6789/0 conn(0x7fc634153ee0 legacy=0x7fc6340cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:40.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc6398c7640 1 -- 172.21.15.7:0/3899522840 >> v1:172.21.15.7:6789/0 conn(0x7fc634159c90 legacy=0x7fc63415d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:40.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.762+0000 7fc6398c7640 1 -- 172.21.15.7:0/3899522840 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6340c4bc0 con 0x7fc6340ca330 2023-11-28T18:25:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc6398c7640 1 --2- 172.21.15.7:0/3899522840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6340ca330 0x7fc6340ca7a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009e50 tx=0x7fc62c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffeddefab3e0be36 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc62bfff640 1 -- 172.21.15.7:0/3899522840 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0af400 con 0x7fc6340ca330 2023-11-28T18:25:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc62bfff640 1 -- 172.21.15.7:0/3899522840 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c0ab050 con 0x7fc6340ca330 2023-11-28T18:25:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc62bfff640 1 -- 172.21.15.7:0/3899522840 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0af780 con 0x7fc6340ca330 2023-11-28T18:25:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc63a0c8640 1 -- 172.21.15.7:0/3899522840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6340ca330 msgr2=0x7fc6340ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc63a0c8640 1 --2- 172.21.15.7:0/3899522840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6340ca330 0x7fc6340ca7a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009e50 tx=0x7fc62c01f1f0 comp rx=0 tx=0).stop 2023-11-28T18:25:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc63a0c8640 1 -- 172.21.15.7:0/3899522840 shutdown_connections 2023-11-28T18:25:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc63a0c8640 1 --2- 172.21.15.7:0/3899522840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6340ca330 0x7fc6340ca7a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc63a0c8640 1 -- 172.21.15.7:0/3899522840 >> 172.21.15.7:0/3899522840 conn(0x7fc634071e20 msgr2=0x7fc634072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc63a0c8640 1 -- 172.21.15.7:0/3899522840 shutdown_connections 2023-11-28T18:25:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.766+0000 7fc63a0c8640 1 -- 172.21.15.7:0/3899522840 wait complete. 2023-11-28T18:25:40.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 Processor -- start 2023-11-28T18:25:40.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 -- start start 2023-11-28T18:25:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6340ca330 0x7fc63415b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634153ee0 0x7fc634157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc634159c90 0x7fc63413e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6341e3930 con 0x7fc634159c90 2023-11-28T18:25:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63415cec0 con 0x7fc634153ee0 2023-11-28T18:25:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63415af40 con 0x7fc6340ca330 2023-11-28T18:25:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc634159c90 0x7fc63413e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc634159c90 0x7fc63413e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38702/0 (socket says 172.21.15.7:38702) 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 -- 172.21.15.7:0/361165039 learned_addr learned my addr 172.21.15.7:0/361165039 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6390c6640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6340ca330 0x7fc63415b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6388c5640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634153ee0 0x7fc634157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 -- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6340ca330 msgr2=0x7fc63415b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6340ca330 0x7fc63415b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 -- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634153ee0 msgr2=0x7fc634157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634153ee0 0x7fc634157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 -- 172.21.15.7:0/361165039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62c009b30 con 0x7fc634159c90 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6388c5640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634153ee0 0x7fc634157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:25:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc6398c7640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc634159c90 0x7fc63413e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc62c01be00 tx=0x7fc62c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc629ffb640 1 -- 172.21.15.7:0/361165039 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0afde0 con 0x7fc634159c90 2023-11-28T18:25:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc629ffb640 1 -- 172.21.15.7:0/361165039 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c009510 con 0x7fc634159c90 2023-11-28T18:25:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc629ffb640 1 -- 172.21.15.7:0/361165039 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0ba660 con 0x7fc634159c90 2023-11-28T18:25:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc634149510 con 0x7fc634159c90 2023-11-28T18:25:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63415cbe0 con 0x7fc634159c90 2023-11-28T18:25:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.770+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63415b0b0 con 0x7fc634159c90 2023-11-28T18:25:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.774+0000 7fc629ffb640 1 -- 172.21.15.7:0/361165039 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc62c0b6040 con 0x7fc634159c90 2023-11-28T18:25:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.774+0000 7fc629ffb640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6180668d0 0x7fc618068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.774+0000 7fc629ffb640 1 -- 172.21.15.7:0/361165039 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fc62c122740 con 0x7fc634159c90 2023-11-28T18:25:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.774+0000 7fc6390c6640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6180668d0 0x7fc618068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.774+0000 7fc6390c6640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6180668d0 0x7fc618068d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc630007fd0 tx=0x7fc630009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:40.778+0000 7fc629ffb640 1 -- 172.21.15.7:0/361165039 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc62c0ef960 con 0x7fc634159c90 2023-11-28T18:25:40.854 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gettimeofday... yes 2023-11-28T18:25:41.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.006+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc63404e890 con 0x7fc634159c90 2023-11-28T18:25:41.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc629ffb640 1 -- 172.21.15.7:0/361165039 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5765 (secure 0 0 0) 0x7fc62c0f2f80 con 0x7fc634159c90 2023-11-28T18:25:41.011 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:41.011 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:37.139805+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2023-11-28T18:25:41.012 INFO:tasks.workunit.client.0.smithi007.stdout:checking for mkdir... yes 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6180668d0 msgr2=0x7fc618068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6180668d0 0x7fc618068d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc630007fd0 tx=0x7fc630009040 comp rx=0 tx=0).stop 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc634159c90 msgr2=0x7fc63413e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc634159c90 0x7fc63413e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc62c01be00 tx=0x7fc62c01bf90 comp rx=0 tx=0).stop 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 shutdown_connections 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6180668d0 0x7fc618068d90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc634159c90 0x7fc63413e090 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634153ee0 0x7fc634157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 --2- 172.21.15.7:0/361165039 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6340ca330 0x7fc63415b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:41.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 >> 172.21.15.7:0/361165039 conn(0x7fc634071e20 msgr2=0x7fc6341e5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:41.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 shutdown_connections 2023-11-28T18:25:41.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:41.010+0000 7fc63a0c8640 1 -- 172.21.15.7:0/361165039 wait complete. 2023-11-28T18:25:41.039 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:25:42.151 INFO:tasks.workunit.client.0.smithi007.stdout:checking for strerror... yes 2023-11-28T18:25:42.298 INFO:tasks.workunit.client.0.smithi007.stdout:checking for open64... yes 2023-11-28T18:25:42.441 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stat64... yes 2023-11-28T18:25:42.586 INFO:tasks.workunit.client.0.smithi007.stdout:checking for fseeko64... yes 2023-11-28T18:25:42.728 INFO:tasks.workunit.client.0.smithi007.stdout:checking for lrand48_r... yes 2023-11-28T18:25:42.872 INFO:tasks.workunit.client.0.smithi007.stdout:checking for srand48_r... yes 2023-11-28T18:25:47.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:47.760 INFO:tasks.workunit.client.0.smithi007.stdout:configure: creating ./config.status 2023-11-28T18:25:47.760 INFO:tasks.workunit.client.0.smithi007.stdout: /bin/bash ./config.status 2023-11-28T18:25:47.760 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating Makefile 2023-11-28T18:25:47.760 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating config.h 2023-11-28T18:25:47.760 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: executing depfiles commands 2023-11-28T18:25:47.760 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: executing default commands 2023-11-28T18:25:47.761 INFO:tasks.workunit.client.0.smithi007.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2023-11-28T18:25:47.761 INFO:tasks.workunit.client.0.smithi007.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2023-11-28T18:25:47.761 INFO:tasks.workunit.client.0.smithi007.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-11-28T18:25:47.762 INFO:tasks.workunit.client.0.smithi007.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-11-28T18:25:47.762 INFO:tasks.workunit.client.0.smithi007.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-11-28T18:25:47.762 INFO:tasks.workunit.client.0.smithi007.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:rm -f stamp-h1 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:touch config.h.in 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:cd . && /bin/bash ./config.status config.h 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating config.h 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:make all-am 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2023-11-28T18:25:47.763 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2023-11-28T18:25:47.764 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2023-11-28T18:25:47.764 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2023-11-28T18:25:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f95357640 1 Processor -- start 2023-11-28T18:25:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f95357640 1 -- start start 2023-11-28T18:25:47.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f95357640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f9015a5b0 0x7f6f900c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:47.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f95357640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f900c8430 con 0x7f6f9006a8a0 2023-11-28T18:25:47.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f95357640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9006ae50 con 0x7f6f90154040 2023-11-28T18:25:47.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f95357640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9006b110 con 0x7f6f9015a5b0 2023-11-28T18:25:47.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f8f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6f9006a8a0 0x7f6f900c8ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33212/0 (socket says 172.21.15.7:33212) 2023-11-28T18:25:47.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f8f7fe640 1 -- 172.21.15.7:0/4282693308 learned_addr learned my addr 172.21.15.7:0/4282693308 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:25:47.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f87fff640 1 --2- 172.21.15.7:0/4282693308 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f9015a5b0 0x7f6f900c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:47.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f8e7fc640 1 -- 172.21.15.7:0/4282693308 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508917066 0 0) 0x7f6f900c8430 con 0x7f6f9006a8a0 2023-11-28T18:25:47.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f8e7fc640 1 -- 172.21.15.7:0/4282693308 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f7c003530 con 0x7f6f9006a8a0 2023-11-28T18:25:47.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f8e7fc640 1 -- 172.21.15.7:0/4282693308 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744241280 0 0) 0x7f6f9006ae50 con 0x7f6f90154040 2023-11-28T18:25:47.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f8e7fc640 1 -- 172.21.15.7:0/4282693308 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f900c8430 con 0x7f6f90154040 2023-11-28T18:25:47.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f87fff640 1 -- 172.21.15.7:0/4282693308 >> v1:172.21.15.193:6789/0 conn(0x7f6f90154040 legacy=0x7f6f90159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:47.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f87fff640 1 -- 172.21.15.7:0/4282693308 >> v1:172.21.15.7:6789/0 conn(0x7f6f9006a8a0 legacy=0x7f6f900c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:47.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.910+0000 7f6f87fff640 1 -- 172.21.15.7:0/4282693308 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900c7820 con 0x7f6f9015a5b0 2023-11-28T18:25:47.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f87fff640 1 --2- 172.21.15.7:0/4282693308 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f9015a5b0 0x7f6f900c89b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f780097f0 tx=0x7f6f7801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=307c7e3fb897b1e3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:47.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f8e7fc640 1 -- 172.21.15.7:0/4282693308 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f780a89b0 con 0x7f6f9015a5b0 2023-11-28T18:25:47.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f8e7fc640 1 -- 172.21.15.7:0/4282693308 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f780a8b10 con 0x7f6f9015a5b0 2023-11-28T18:25:47.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f8e7fc640 1 -- 172.21.15.7:0/4282693308 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f780a8e80 con 0x7f6f9015a5b0 2023-11-28T18:25:47.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f95357640 1 -- 172.21.15.7:0/4282693308 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f9015a5b0 msgr2=0x7f6f900c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:47.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f95357640 1 --2- 172.21.15.7:0/4282693308 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f9015a5b0 0x7f6f900c89b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f780097f0 tx=0x7f6f7801ecb0 comp rx=0 tx=0).stop 2023-11-28T18:25:47.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f95357640 1 -- 172.21.15.7:0/4282693308 shutdown_connections 2023-11-28T18:25:47.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f95357640 1 --2- 172.21.15.7:0/4282693308 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f9015a5b0 0x7f6f900c89b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:47.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f95357640 1 -- 172.21.15.7:0/4282693308 >> 172.21.15.7:0/4282693308 conn(0x7f6f9006d4d0 msgr2=0x7f6f9006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:47.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f95357640 1 -- 172.21.15.7:0/4282693308 shutdown_connections 2023-11-28T18:25:47.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f95357640 1 -- 172.21.15.7:0/4282693308 wait complete. 2023-11-28T18:25:47.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.914+0000 7f6f95357640 1 Processor -- start 2023-11-28T18:25:47.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 -- start start 2023-11-28T18:25:47.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f9006a8a0 0x7f6f90157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:47.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f90154040 0x7f6f90150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:47.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f9015a5b0 0x7f6f90158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:47.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90150d80 con 0x7f6f9006a8a0 2023-11-28T18:25:47.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9015c6a0 con 0x7f6f9015a5b0 2023-11-28T18:25:47.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9015d740 con 0x7f6f90154040 2023-11-28T18:25:47.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f9006a8a0 0x7f6f90157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:47.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f9006a8a0 0x7f6f90157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39446/0 (socket says 172.21.15.7:39446) 2023-11-28T18:25:47.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 -- 172.21.15.7:0/1852137139 learned_addr learned my addr 172.21.15.7:0/1852137139 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:47.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 -- 172.21.15.7:0/1852137139 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f90154040 msgr2=0x7f6f90150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:47.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f90154040 0x7f6f90150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:47.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 -- 172.21.15.7:0/1852137139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f9015a5b0 msgr2=0x7f6f90158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:47.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f9015a5b0 0x7f6f90158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:47.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 -- 172.21.15.7:0/1852137139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f78009420 con 0x7f6f9006a8a0 2023-11-28T18:25:47.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f8effd640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f9006a8a0 0x7f6f90157380 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000c500 tx=0x7f6f8000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:47.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f877fe640 1 -- 172.21.15.7:0/1852137139 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f800052d0 con 0x7f6f9006a8a0 2023-11-28T18:25:47.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f877fe640 1 -- 172.21.15.7:0/1852137139 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f80005450 con 0x7f6f9006a8a0 2023-11-28T18:25:47.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f877fe640 1 -- 172.21.15.7:0/1852137139 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80012870 con 0x7f6f9006a8a0 2023-11-28T18:25:47.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9015d9e0 con 0x7f6f9006a8a0 2023-11-28T18:25:47.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9015dfa0 con 0x7f6f9006a8a0 2023-11-28T18:25:47.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.918+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9015e190 con 0x7f6f9006a8a0 2023-11-28T18:25:47.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.922+0000 7f6f877fe640 1 -- 172.21.15.7:0/1852137139 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6f800129d0 con 0x7f6f9006a8a0 2023-11-28T18:25:47.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.922+0000 7f6f877fe640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f700668d0 0x7f6f70068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:47.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.922+0000 7f6f877fe640 1 -- 172.21.15.7:0/1852137139 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f6f80084200 con 0x7f6f9006a8a0 2023-11-28T18:25:47.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.922+0000 7f6f87fff640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f700668d0 0x7f6f70068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:47.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.922+0000 7f6f87fff640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f700668d0 0x7f6f70068d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6f780b2000 tx=0x7f6f78004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:47.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:47.926+0000 7f6f877fe640 1 -- 172.21.15.7:0/1852137139 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6f9015e190 con 0x7f6f9006a8a0 2023-11-28T18:25:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.118+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f9015e190 con 0x7f6f9006a8a0 2023-11-28T18:25:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.118+0000 7f6f877fe640 1 -- 172.21.15.7:0/1852137139 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5767 (secure 0 0 0) 0x7f6f9015e190 con 0x7f6f9006a8a0 2023-11-28T18:25:48.121 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:48.122 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:42.154092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:resolve","state_seq":42,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 22 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f700668d0 msgr2=0x7f6f70068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f700668d0 0x7f6f70068d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6f780b2000 tx=0x7f6f78004740 comp rx=0 tx=0).stop 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f9006a8a0 msgr2=0x7f6f90157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f9006a8a0 0x7f6f90157380 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000c500 tx=0x7f6f8000c9c0 comp rx=0 tx=0).stop 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 shutdown_connections 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f700668d0 0x7f6f70068d90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f9015a5b0 0x7f6f90158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f90154040 0x7f6f90150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 --2- 172.21.15.7:0/1852137139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f9006a8a0 0x7f6f90157380 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 >> 172.21.15.7:0/1852137139 conn(0x7f6f9006d4d0 msgr2=0x7f6f901596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 shutdown_connections 2023-11-28T18:25:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:48.122+0000 7f6f95357640 1 -- 172.21.15.7:0/1852137139 wait complete. 2023-11-28T18:25:48.144 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:25:48.463 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/main.Tpo .deps/main.Po 2023-11-28T18:25:48.630 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2023-11-28T18:25:48.718 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2023-11-28T18:25:48.735 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2023-11-28T18:25:48.995 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2023-11-28T18:25:49.012 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2023-11-28T18:25:49.866 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2023-11-28T18:25:49.883 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2023-11-28T18:25:50.040 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2023-11-28T18:25:50.057 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2023-11-28T18:25:50.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.286+0000 7fdcfca17640 1 Processor -- start 2023-11-28T18:25:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.286+0000 7fdcfca17640 1 -- start start 2023-11-28T18:25:50.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcfca17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fdcf8154000 0x7fdcf80c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcfca17640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdcf80c6ba0 0x7fdcf80c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcfca17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf8159e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcfca17640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf815e0f0 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcfca17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf815d520 con 0x7fdcf8154000 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcfca17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf80c35d0 con 0x7fdcf80c3180 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdcf80c6ba0 0x7fdcf80c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdcf80c6ba0 0x7fdcf80c2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39456/0 (socket says 172.21.15.7:39456) 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 -- 172.21.15.7:0/2296977993 learned_addr learned my addr 172.21.15.7:0/2296977993 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcf659c640 1 --2- 172.21.15.7:0/2296977993 >> v2:172.21.15.193:3300/0 conn(0x7fdcf8154000 0x7fdcf80c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcf6d9d640 1 --2- 172.21.15.7:0/2296977993 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf8159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 -- 172.21.15.7:0/2296977993 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 msgr2=0x7fdcf8159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 --2- 172.21.15.7:0/2296977993 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf8159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 -- 172.21.15.7:0/2296977993 >> v2:172.21.15.193:3300/0 conn(0x7fdcf8154000 msgr2=0x7fdcf80c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 --2- 172.21.15.7:0/2296977993 >> v2:172.21.15.193:3300/0 conn(0x7fdcf8154000 0x7fdcf80c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 -- 172.21.15.7:0/2296977993 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf80c7c70 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdcf6d9d640 1 --2- 172.21.15.7:0/2296977993 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf8159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:25:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.290+0000 7fdceffff640 1 --2- 172.21.15.7:0/2296977993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf80c2c40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80c8120 tx=0x7fdce001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e315a241b6633ae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcf5d9b640 1 -- 172.21.15.7:0/2296977993 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce00a89b0 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcf5d9b640 1 -- 172.21.15.7:0/2296977993 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce00a8b10 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcf5d9b640 1 -- 172.21.15.7:0/2296977993 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce00a8ea0 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- 172.21.15.7:0/2296977993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 msgr2=0x7fdcf80c2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 --2- 172.21.15.7:0/2296977993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf80c2c40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80c8120 tx=0x7fdce001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- 172.21.15.7:0/2296977993 shutdown_connections 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 --2- 172.21.15.7:0/2296977993 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf8159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 --2- 172.21.15.7:0/2296977993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf80c2c40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 --2- 172.21.15.7:0/2296977993 >> v2:172.21.15.193:3300/0 conn(0x7fdcf8154000 0x7fdcf80c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- 172.21.15.7:0/2296977993 >> 172.21.15.7:0/2296977993 conn(0x7fdcf80662e0 msgr2=0x7fdcf80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- 172.21.15.7:0/2296977993 shutdown_connections 2023-11-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- 172.21.15.7:0/2296977993 wait complete. 2023-11-28T18:25:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 Processor -- start 2023-11-28T18:25:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- start start 2023-11-28T18:25:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf81f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf8154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcf8154000 0x7fdcf80ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf81f08f0 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf815d280 con 0x7fdcf8154000 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.294+0000 7fdcfca17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf815cfa0 con 0x7fdcf80c3180 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf8154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf8154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39458/0 (socket says 172.21.15.7:39458) 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 -- 172.21.15.7:0/4008749307 learned_addr learned my addr 172.21.15.7:0/4008749307 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdcf659c640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf81f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdcf6d9d640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcf8154000 0x7fdcf80ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 -- 172.21.15.7:0/4008749307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 msgr2=0x7fdcf81f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf81f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 -- 172.21.15.7:0/4008749307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcf8154000 msgr2=0x7fdcf80ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:50.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcf8154000 0x7fdcf80ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 -- 172.21.15.7:0/4008749307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce00094d0 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceffff640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf8154780 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdce001d3a0 tx=0x7fdce001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceeffd640 1 -- 172.21.15.7:0/4008749307 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce00ba600 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceeffd640 1 -- 172.21.15.7:0/4008749307 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce00ba760 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceeffd640 1 -- 172.21.15.7:0/4008749307 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce00bab10 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf8156160 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf8156720 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf80c2990 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.298+0000 7fdceeffd640 1 -- 172.21.15.7:0/4008749307 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fdce00bac70 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.302+0000 7fdceeffd640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdcd80668d0 0x7fdcd8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.302+0000 7fdceeffd640 1 -- 172.21.15.7:0/4008749307 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fdce0122b00 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.302+0000 7fdcf659c640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdcd80668d0 0x7fdcd8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.302+0000 7fdcf659c640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdcd80668d0 0x7fdcd8068d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009b60 tx=0x7fdce8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:50.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.302+0000 7fdceeffd640 1 -- 172.21.15.7:0/4008749307 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdce0124e60 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.494+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcf804e890 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.494+0000 7fdceeffd640 1 -- 172.21.15.7:0/4008749307 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fdce00b5020 con 0x7fdcf80c6ba0 2023-11-28T18:25:50.498 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:50.498 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:48.148836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:reconnect","state_seq":44,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:50.498 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2023-11-28T18:25:50.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdcd80668d0 msgr2=0x7fdcd8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:50.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdcd80668d0 0x7fdcd8068d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009b60 tx=0x7fdce8009190 comp rx=0 tx=0).stop 2023-11-28T18:25:50.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 msgr2=0x7fdcf8154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:50.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf8154780 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdce001d3a0 tx=0x7fdce001ba90 comp rx=0 tx=0).stop 2023-11-28T18:25:50.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 shutdown_connections 2023-11-28T18:25:50.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdcd80668d0 0x7fdcd8068d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcf8154000 0x7fdcf80ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcf80c6ba0 0x7fdcf8154780 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 --2- 172.21.15.7:0/4008749307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdcf80c3180 0x7fdcf81f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:50.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 >> 172.21.15.7:0/4008749307 conn(0x7fdcf80662e0 msgr2=0x7fdcf8151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:50.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 shutdown_connections 2023-11-28T18:25:50.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:50.498+0000 7fdcfca17640 1 -- 172.21.15.7:0/4008749307 wait complete. 2023-11-28T18:25:50.518 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:25:51.121 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2023-11-28T18:25:51.139 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2023-11-28T18:25:51.438 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2023-11-28T18:25:51.456 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2023-11-28T18:25:52.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b66ac9640 1 Processor -- start 2023-11-28T18:25:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b66ac9640 1 -- start start 2023-11-28T18:25:52.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b66ac9640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6b600cbb70 0x7f6b600cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b66ac9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b600cc810 0x7f6b60072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b66ac9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b600ca920 con 0x7f6b601542e0 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b66ac9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b60071d50 con 0x7f6b600cc810 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b66ac9640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b60072030 con 0x7f6b600cbb70 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b65066640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b600cc810 0x7f6b60072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b65066640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b600cc810 0x7f6b60072450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39460/0 (socket says 172.21.15.7:39460) 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b65066640 1 -- 172.21.15.7:0/3661428690 learned_addr learned my addr 172.21.15.7:0/3661428690 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b5ffff640 1 --2- 172.21.15.7:0/3661428690 >> v2:172.21.15.193:3300/0 conn(0x7f6b600cbb70 0x7f6b600cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b65066640 1 -- 172.21.15.7:0/3661428690 >> v2:172.21.15.193:3300/0 conn(0x7f6b600cbb70 msgr2=0x7f6b600cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b65066640 1 --2- 172.21.15.7:0/3661428690 >> v2:172.21.15.193:3300/0 conn(0x7f6b600cbb70 0x7f6b600cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b65066640 1 -- 172.21.15.7:0/3661428690 >> v1:172.21.15.193:6789/0 conn(0x7f6b601542e0 legacy=0x7f6b6015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:52.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.666+0000 7f6b65066640 1 -- 172.21.15.7:0/3661428690 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b600c4cd0 con 0x7f6b600cc810 2023-11-28T18:25:52.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b65066640 1 --2- 172.21.15.7:0/3661428690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60072450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6b600ca8b0 tx=0x7f6b5801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c608343f740766 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:52.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b5f7fe640 1 -- 172.21.15.7:0/3661428690 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b580a57a0 con 0x7f6b600cc810 2023-11-28T18:25:52.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b5f7fe640 1 -- 172.21.15.7:0/3661428690 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b580a5900 con 0x7f6b600cc810 2023-11-28T18:25:52.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b5f7fe640 1 -- 172.21.15.7:0/3661428690 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b580a5c90 con 0x7f6b600cc810 2023-11-28T18:25:52.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 -- 172.21.15.7:0/3661428690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 msgr2=0x7f6b60072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:52.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/3661428690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60072450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6b600ca8b0 tx=0x7f6b5801c9d0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 -- 172.21.15.7:0/3661428690 shutdown_connections 2023-11-28T18:25:52.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/3661428690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60072450 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/3661428690 >> v2:172.21.15.193:3300/0 conn(0x7f6b600cbb70 0x7f6b600cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 -- 172.21.15.7:0/3661428690 >> 172.21.15.7:0/3661428690 conn(0x7f6b600717e0 msgr2=0x7f6b600662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:52.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 -- 172.21.15.7:0/3661428690 shutdown_connections 2023-11-28T18:25:52.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 -- 172.21.15.7:0/3661428690 wait complete. 2023-11-28T18:25:52.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.670+0000 7f6b66ac9640 1 Processor -- start 2023-11-28T18:25:52.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 -- start start 2023-11-28T18:25:52.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b600cbb70 0x7f6b600c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:52.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:52.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b601542e0 0x7f6b601461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:52.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b60157450 con 0x7f6b600cc810 2023-11-28T18:25:52.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b601579d0 con 0x7f6b601542e0 2023-11-28T18:25:52.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b601576f0 con 0x7f6b600cbb70 2023-11-28T18:25:52.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:52.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60147140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39464/0 (socket says 172.21.15.7:39464) 2023-11-28T18:25:52.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 -- 172.21.15.7:0/4202079735 learned_addr learned my addr 172.21.15.7:0/4202079735 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:52.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b65066640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b601542e0 0x7f6b601461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:52.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b64865640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b600cbb70 0x7f6b600c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:52.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 -- 172.21.15.7:0/4202079735 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b600cbb70 msgr2=0x7f6b600c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:52.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b600cbb70 0x7f6b600c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 -- 172.21.15.7:0/4202079735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b601542e0 msgr2=0x7f6b601461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:52.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b601542e0 0x7f6b601461a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 -- 172.21.15.7:0/4202079735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b58009b30 con 0x7f6b600cc810 2023-11-28T18:25:52.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5ffff640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60147140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00d800 tx=0x7f6b4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:52.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5d7fa640 1 -- 172.21.15.7:0/4202079735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c00b6c0 con 0x7f6b600cc810 2023-11-28T18:25:52.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5d7fa640 1 -- 172.21.15.7:0/4202079735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4c00b820 con 0x7f6b600cc810 2023-11-28T18:25:52.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b5d7fa640 1 -- 172.21.15.7:0/4202079735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c00bb90 con 0x7f6b600cc810 2023-11-28T18:25:52.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b60155850 con 0x7f6b600cc810 2023-11-28T18:25:52.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b60155ea0 con 0x7f6b600cc810 2023-11-28T18:25:52.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.674+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6006c670 con 0x7f6b600cc810 2023-11-28T18:25:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.678+0000 7f6b5d7fa640 1 -- 172.21.15.7:0/4202079735 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6b4c00bcf0 con 0x7f6b600cc810 2023-11-28T18:25:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.678+0000 7f6b5d7fa640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b44066920 0x7f6b44068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.678+0000 7f6b5d7fa640 1 -- 172.21.15.7:0/4202079735 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f6b4c084850 con 0x7f6b600cc810 2023-11-28T18:25:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.678+0000 7f6b64865640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b44066920 0x7f6b44068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.678+0000 7f6b64865640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b44066920 0x7f6b44068de0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6b54007fd0 tx=0x7f6b54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.682+0000 7f6b5d7fa640 1 -- 172.21.15.7:0/4202079735 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6b4c051b50 con 0x7f6b600cc810 2023-11-28T18:25:52.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.870+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b60145ec0 con 0x7f6b600cc810 2023-11-28T18:25:52.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b5d7fa640 1 -- 172.21.15.7:0/4202079735 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f6b60145ec0 con 0x7f6b600cc810 2023-11-28T18:25:52.876 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:52.877 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:48.148836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:reconnect","state_seq":44,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:52.878 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2023-11-28T18:25:52.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b44066920 msgr2=0x7f6b44068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:52.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b44066920 0x7f6b44068de0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6b54007fd0 tx=0x7f6b54009040 comp rx=0 tx=0).stop 2023-11-28T18:25:52.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 msgr2=0x7f6b60147140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:52.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60147140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00d800 tx=0x7f6b4c00dcc0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 shutdown_connections 2023-11-28T18:25:52.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b44066920 0x7f6b44068de0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b601542e0 0x7f6b601461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b600cc810 0x7f6b60147140 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 --2- 172.21.15.7:0/4202079735 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b600cbb70 0x7f6b600c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:52.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 >> 172.21.15.7:0/4202079735 conn(0x7f6b600717e0 msgr2=0x7f6b60153210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:52.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 shutdown_connections 2023-11-28T18:25:52.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:52.874+0000 7f6b66ac9640 1 -- 172.21.15.7:0/4202079735 wait complete. 2023-11-28T18:25:52.896 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:25:53.328 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2023-11-28T18:25:53.346 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2023-11-28T18:25:53.959 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2023-11-28T18:25:53.978 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2023-11-28T18:25:54.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:55.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2bb232640 1 Processor -- start 2023-11-28T18:25:55.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2bb232640 1 -- start start 2023-11-28T18:25:55.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2bb232640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2b41542e0 0x7ff2b415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:55.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2bb232640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 0x7ff2b40728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:55.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2bb232640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b40ca920 con 0x7ff2b40cbb70 2023-11-28T18:25:55.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2bb232640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b4071d50 con 0x7ff2b41542e0 2023-11-28T18:25:55.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2bb232640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b4072030 con 0x7ff2b4072450 2023-11-28T18:25:55.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2b8fce640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2b41542e0 0x7ff2b415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:55.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2b3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff2b40cbb70 0x7ff2b40cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53526/0 (socket says 172.21.15.7:53526) 2023-11-28T18:25:55.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2b3fff640 1 -- 172.21.15.7:0/2797464338 learned_addr learned my addr 172.21.15.7:0/2797464338 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:25:55.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2b97cf640 1 --2- 172.21.15.7:0/2797464338 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 0x7ff2b40728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:55.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2b8fce640 1 -- 172.21.15.7:0/2797464338 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 msgr2=0x7ff2b40728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:55.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2b8fce640 1 --2- 172.21.15.7:0/2797464338 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 0x7ff2b40728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2b8fce640 1 -- 172.21.15.7:0/2797464338 >> v1:172.21.15.7:6789/0 conn(0x7ff2b40cbb70 legacy=0x7ff2b40cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:55.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.050+0000 7ff2b8fce640 1 -- 172.21.15.7:0/2797464338 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b415e550 con 0x7ff2b41542e0 2023-11-28T18:25:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2b8fce640 1 --2- 172.21.15.7:0/2797464338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b41542e0 0x7ff2b415e010 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80097f0 tx=0x7ff2a801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59dd069e1448268 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2b37fe640 1 -- 172.21.15.7:0/2797464338 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80ad640 con 0x7ff2b41542e0 2023-11-28T18:25:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2b37fe640 1 -- 172.21.15.7:0/2797464338 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a80ad7a0 con 0x7ff2b41542e0 2023-11-28T18:25:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2b37fe640 1 -- 172.21.15.7:0/2797464338 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80adb30 con 0x7ff2b41542e0 2023-11-28T18:25:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2bb232640 1 -- 172.21.15.7:0/2797464338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b41542e0 msgr2=0x7ff2b415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2bb232640 1 --2- 172.21.15.7:0/2797464338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b41542e0 0x7ff2b415e010 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80097f0 tx=0x7ff2a801c480 comp rx=0 tx=0).stop 2023-11-28T18:25:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2bb232640 1 -- 172.21.15.7:0/2797464338 shutdown_connections 2023-11-28T18:25:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2bb232640 1 --2- 172.21.15.7:0/2797464338 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 0x7ff2b40728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2bb232640 1 --2- 172.21.15.7:0/2797464338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b41542e0 0x7ff2b415e010 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2bb232640 1 -- 172.21.15.7:0/2797464338 >> 172.21.15.7:0/2797464338 conn(0x7ff2b40717e0 msgr2=0x7ff2b40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2bb232640 1 -- 172.21.15.7:0/2797464338 shutdown_connections 2023-11-28T18:25:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.054+0000 7ff2bb232640 1 -- 172.21.15.7:0/2797464338 wait complete. 2023-11-28T18:25:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 Processor -- start 2023-11-28T18:25:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 -- start start 2023-11-28T18:25:55.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 0x7ff2b41f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b40cbb70 0x7ff2b41ec910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b41542e0 0x7ff2b41451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b415d0a0 con 0x7ff2b40cbb70 2023-11-28T18:25:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b41ebec0 con 0x7ff2b41542e0 2023-11-28T18:25:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b41ec480 con 0x7ff2b4072450 2023-11-28T18:25:55.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b40cbb70 0x7ff2b41ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:55.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b40cbb70 0x7ff2b41ec910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41258/0 (socket says 172.21.15.7:41258) 2023-11-28T18:25:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 -- 172.21.15.7:0/4101636655 learned_addr learned my addr 172.21.15.7:0/4101636655 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b8fce640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 0x7ff2b41f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b97cf640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b41542e0 0x7ff2b41451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 -- 172.21.15.7:0/4101636655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 msgr2=0x7ff2b41f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 0x7ff2b41f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 -- 172.21.15.7:0/4101636655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b41542e0 msgr2=0x7ff2b41451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b41542e0 0x7ff2b41451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 -- 172.21.15.7:0/4101636655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a8009420 con 0x7ff2b40cbb70 2023-11-28T18:25:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b3fff640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b40cbb70 0x7ff2b41ec910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000c500 tx=0x7ff2a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:55.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b17fa640 1 -- 172.21.15.7:0/4101636655 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0005cf0 con 0x7ff2b40cbb70 2023-11-28T18:25:55.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b17fa640 1 -- 172.21.15.7:0/4101636655 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a0005e50 con 0x7ff2b40cbb70 2023-11-28T18:25:55.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2b17fa640 1 -- 172.21.15.7:0/4101636655 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a00072f0 con 0x7ff2b40cbb70 2023-11-28T18:25:55.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b41f2fb0 con 0x7ff2b40cbb70 2023-11-28T18:25:55.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.058+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b41f3570 con 0x7ff2b40cbb70 2023-11-28T18:25:55.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.062+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b406c670 con 0x7ff2b40cbb70 2023-11-28T18:25:55.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.062+0000 7ff2b17fa640 1 -- 172.21.15.7:0/4101636655 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff2a0007450 con 0x7ff2b40cbb70 2023-11-28T18:25:55.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.062+0000 7ff2b17fa640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff298066900 0x7ff298068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:55.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.062+0000 7ff2b17fa640 1 -- 172.21.15.7:0/4101636655 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7ff2a00839f0 con 0x7ff2b40cbb70 2023-11-28T18:25:55.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.062+0000 7ff2b8fce640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff298066900 0x7ff298068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:55.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.062+0000 7ff2b8fce640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff298066900 0x7ff298068dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80af000 tx=0x7ff2a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:55.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.066+0000 7ff2b17fa640 1 -- 172.21.15.7:0/4101636655 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2a0050c00 con 0x7ff2b40cbb70 2023-11-28T18:25:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2b41ec5f0 con 0x7ff2b40cbb70 2023-11-28T18:25:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2b17fa640 1 -- 172.21.15.7:0/4101636655 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5766 (secure 0 0 0) 0x7ff2a0017020 con 0x7ff2b40cbb70 2023-11-28T18:25:55.261 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:55.261 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:53.338806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:rejoin","state_seq":46,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 24 2023-11-28T18:25:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff298066900 msgr2=0x7ff298068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff298066900 0x7ff298068dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80af000 tx=0x7ff2a8008040 comp rx=0 tx=0).stop 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b40cbb70 msgr2=0x7ff2b41ec910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b40cbb70 0x7ff2b41ec910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000c500 tx=0x7ff2a000c9c0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 shutdown_connections 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff298066900 0x7ff298068dc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b41542e0 0x7ff2b41451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b40cbb70 0x7ff2b41ec910 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 --2- 172.21.15.7:0/4101636655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4072450 0x7ff2b41f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.258+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 >> 172.21.15.7:0/4101636655 conn(0x7ff2b40717e0 msgr2=0x7ff2b415a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.262+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 shutdown_connections 2023-11-28T18:25:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:55.262+0000 7ff2bb232640 1 -- 172.21.15.7:0/4101636655 wait complete. 2023-11-28T18:25:55.283 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4279": { "addr": "172.21.15.193:6835/4143265275", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 4143265275, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 4143265275, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 21, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:rejoin", "state_seq": 46 }, "gid_4284": { "addr": "172.21.15.7:6835/1419029356", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1419029356, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1419029356, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4284, "incarnation": 13, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4285": { "addr": "172.21.15.193:6837/3911527535", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 3911527535, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 3911527535, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 11, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.7:6838/2690556675", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 2690556675, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 2690556675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 23, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:25:53.338806+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 4288, "mds_2": 4285, "mds_3": 4284, "mds_4": 4279 } } } ], "standbys": [] } 2023-11-28T18:25:55.479 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2023-11-28T18:25:55.498 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2023-11-28T18:25:56.804 INFO:tasks.workunit.client.0.smithi007.stderr:ffsb_op.c: In function 'print_op_results': 2023-11-28T18:25:56.804 INFO:tasks.workunit.client.0.smithi007.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2023-11-28T18:25:56.804 INFO:tasks.workunit.client.0.smithi007.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2023-11-28T18:25:56.804 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~ 2023-11-28T18:25:56.804 INFO:tasks.workunit.client.0.smithi007.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2023-11-28T18:25:56.804 INFO:tasks.workunit.client.0.smithi007.stderr: 86 | sprintf(buf, "NA\0"); 2023-11-28T18:25:56.804 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~ 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr:ffsb_op.c: In function 'print_results': 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~ 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr:In file included from /usr/include/stdio.h:894, 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr: from ffsb_op.c:20: 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2023-11-28T18:25:56.821 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:56.822 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | __glibc_objsize (__s), __fmt, 2023-11-28T18:25:56.822 INFO:tasks.workunit.client.0.smithi007.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:56.822 INFO:tasks.workunit.client.0.smithi007.stderr: 40 | __va_arg_pack ()); 2023-11-28T18:25:56.822 INFO:tasks.workunit.client.0.smithi007.stderr: | ~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:56.876 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2023-11-28T18:25:56.894 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2023-11-28T18:25:57.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:57.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa74432640 1 Processor -- start 2023-11-28T18:25:57.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa74432640 1 -- start start 2023-11-28T18:25:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa74432640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c0c3bc0 0x7ffa6c0c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa74432640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ffa6c0c3490 0x7ffa6c0c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa74432640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c0c3920 con 0x7ffa6c1503c0 2023-11-28T18:25:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa74432640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6c157240 con 0x7ffa6c0c3490 2023-11-28T18:25:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa74432640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0c2ce0 con 0x7ffa6c0c3bc0 2023-11-28T18:25:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa721ce640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ffa6c1503c0 0x7ffa6c06d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53532/0 (socket says 172.21.15.7:53532) 2023-11-28T18:25:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa721ce640 1 -- 172.21.15.7:0/3695079011 learned_addr learned my addr 172.21.15.7:0/3695079011 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:25:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa729cf640 1 --2- 172.21.15.7:0/3695079011 >> v2:172.21.15.193:3300/0 conn(0x7ffa6c0c3490 0x7ffa6c0c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950331953 0 0) 0x7ffa68001120 con 0x7ffa6c1503c0 2023-11-28T18:25:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa719cd640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c0c3bc0 0x7ffa6c0c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.426+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa58003530 con 0x7ffa6c1503c0 2023-11-28T18:25:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 226907196 0 0) 0x7ffa58003530 con 0x7ffa6c1503c0 2023-11-28T18:25:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c0c3bc0 msgr2=0x7ffa6c0c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c0c3bc0 0x7ffa6c0c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 >> v2:172.21.15.193:3300/0 conn(0x7ffa6c0c3490 msgr2=0x7ffa6c0c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 --2- 172.21.15.7:0/3695079011 >> v2:172.21.15.193:3300/0 conn(0x7ffa6c0c3490 0x7ffa6c0c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c0c2f80 con 0x7ffa6c1503c0 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7ffa68001810 con 0x7ffa6c1503c0 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 >> v1:172.21.15.7:6789/0 conn(0x7ffa6c1503c0 legacy=0x7ffa6c06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58003980 0x7ffa58003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c0c3bc0 0x7ffa580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa58004d10 0x7ffa58005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c157240 con 0x7ffa58004d10 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0c2ce0 con 0x7ffa58003980 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5800b990 con 0x7ffa6c0c3bc0 2023-11-28T18:25:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa729cf640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa58004d10 0x7ffa58005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa719cd640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c0c3bc0 0x7ffa580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa721ce640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58003980 0x7ffa58003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa729cf640 1 -- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c0c3bc0 msgr2=0x7ffa580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa729cf640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c0c3bc0 0x7ffa580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa729cf640 1 -- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58003980 msgr2=0x7ffa58003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa729cf640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58003980 0x7ffa58003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa729cf640 1 -- 172.21.15.7:0/3695079011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa58003530 con 0x7ffa58004d10 2023-11-28T18:25:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.430+0000 7ffa729cf640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa58004d10 0x7ffa58005190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009fc0 tx=0x7ffa6401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff926957cc7806a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa640a5cb0 con 0x7ffa58004d10 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6400f040 con 0x7ffa58004d10 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa711cc640 1 -- 172.21.15.7:0/3695079011 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa640ad680 con 0x7ffa58004d10 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- 172.21.15.7:0/3695079011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa58004d10 msgr2=0x7ffa58005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 --2- 172.21.15.7:0/3695079011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa58004d10 0x7ffa58005190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009fc0 tx=0x7ffa6401cca0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa729cf640 1 -- 172.21.15.7:0/3695079011 reap_dead start 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- 172.21.15.7:0/3695079011 shutdown_connections 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- 172.21.15.7:0/3695079011 >> 172.21.15.7:0/3695079011 conn(0x7ffa6c0662e0 msgr2=0x7ffa6c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- 172.21.15.7:0/3695079011 shutdown_connections 2023-11-28T18:25:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- 172.21.15.7:0/3695079011 wait complete. 2023-11-28T18:25:57.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 Processor -- start 2023-11-28T18:25:57.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- start start 2023-11-28T18:25:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58004d10 0x7ffa6c14b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c1f3130 0x7ffa6c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa6c1ef5c0 0x7ffa6c1efa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0c4eb0 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c14b360 con 0x7ffa58004d10 2023-11-28T18:25:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.434+0000 7ffa74432640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c1eff80 con 0x7ffa6c1f3130 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa6c1ef5c0 0x7ffa6c1efa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa6c1ef5c0 0x7ffa6c1efa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41276/0 (socket says 172.21.15.7:41276) 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 -- 172.21.15.7:0/662885032 learned_addr learned my addr 172.21.15.7:0/662885032 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa719cd640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c1f3130 0x7ffa6c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa721ce640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58004d10 0x7ffa6c14b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 -- 172.21.15.7:0/662885032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c1f3130 msgr2=0x7ffa6c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c1f3130 0x7ffa6c0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 -- 172.21.15.7:0/662885032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58004d10 msgr2=0x7ffa6c14b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58004d10 0x7ffa6c14b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 -- 172.21.15.7:0/662885032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c14b5e0 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa729cf640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa6c1ef5c0 0x7ffa6c1efa40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffa6401c710 tx=0x7ffa64009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa62ffd640 1 -- 172.21.15.7:0/662885032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa640039a0 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa62ffd640 1 -- 172.21.15.7:0/662885032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa64003b20 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa62ffd640 1 -- 172.21.15.7:0/662885032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa640ae890 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c155eb0 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c1f3550 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6c04e890 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.438+0000 7ffa62ffd640 1 -- 172.21.15.7:0/662885032 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ffa640b4040 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.442+0000 7ffa62ffd640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffa500668d0 0x7ffa50068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:57.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.442+0000 7ffa62ffd640 1 -- 172.21.15.7:0/662885032 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7ffa6411e0b0 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.442+0000 7ffa721ce640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffa500668d0 0x7ffa50068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:57.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.442+0000 7ffa721ce640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffa500668d0 0x7ffa50068d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffa68000c10 tx=0x7ffa68007a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:57.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.446+0000 7ffa62ffd640 1 -- 172.21.15.7:0/662885032 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffa640eb2c0 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.634+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa6c06e240 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.634+0000 7ffa62ffd640 1 -- 172.21.15.7:0/662885032 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5766 (secure 0 0 0) 0x7ffa640ee8e0 con 0x7ffa6c1ef5c0 2023-11-28T18:25:57.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:25:57.638 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:53.338806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:rejoin","state_seq":46,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:25:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 24 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffa500668d0 msgr2=0x7ffa50068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffa500668d0 0x7ffa50068d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffa68000c10 tx=0x7ffa68007a80 comp rx=0 tx=0).stop 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa6c1ef5c0 msgr2=0x7ffa6c1efa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa6c1ef5c0 0x7ffa6c1efa40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffa6401c710 tx=0x7ffa64009750 comp rx=0 tx=0).stop 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 shutdown_connections 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffa500668d0 0x7ffa50068d90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa6c1ef5c0 0x7ffa6c1efa40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffa6c1f3130 0x7ffa6c0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 --2- 172.21.15.7:0/662885032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa58004d10 0x7ffa6c14b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 >> 172.21.15.7:0/662885032 conn(0x7ffa6c0662e0 msgr2=0x7ffa6c070260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 shutdown_connections 2023-11-28T18:25:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:57.638+0000 7ffa74432640 1 -- 172.21.15.7:0/662885032 wait complete. 2023-11-28T18:25:57.659 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:25:58.158 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/util.Tpo .deps/util.Po 2023-11-28T18:25:58.175 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2023-11-28T18:25:58.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:25:58.478 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:25:59.439 INFO:tasks.workunit.client.0.smithi007.stderr:In file included from parser.h:22, 2023-11-28T18:25:59.440 INFO:tasks.workunit.client.0.smithi007.stderr: from parser.c:25: 2023-11-28T18:25:59.440 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c: In function 'init_filesys': 2023-11-28T18:25:59.442 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-11-28T18:25:59.442 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-11-28T18:25:59.442 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2023-11-28T18:25:59.442 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-11-28T18:25:59.442 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-11-28T18:25:59.443 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-11-28T18:25:59.444 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2023-11-28T18:25:59.445 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-11-28T18:25:59.446 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-11-28T18:25:59.446 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2023-11-28T18:25:59.446 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2023-11-28T18:25:59.446 INFO:tasks.workunit.client.0.smithi007.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2023-11-28T18:25:59.446 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.449 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-11-28T18:25:59.449 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-11-28T18:25:59.449 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2023-11-28T18:25:59.449 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-11-28T18:25:59.449 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-11-28T18:25:59.449 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c: In function 'init_tg_stats': 2023-11-28T18:25:59.450 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2023-11-28T18:25:59.451 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: 971 | list_for_each_entry(tmp_list, 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-11-28T18:25:59.452 INFO:tasks.workunit.client.0.smithi007.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: 971 | list_for_each_entry(tmp_list, 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2023-11-28T18:25:59.453 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr: 980 | list_for_each_entry(tmp_list, 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-11-28T18:25:59.454 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr: 980 | list_for_each_entry(tmp_list, 2023-11-28T18:25:59.455 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.498 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c: In function 'get_optstr': 2023-11-28T18:25:59.498 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2023-11-28T18:25:59.498 INFO:tasks.workunit.client.0.smithi007.stderr: 206 | len = strnlen(temp, 4095) + 1; 2023-11-28T18:25:59.499 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.499 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:200:14: note: source object declared here 2023-11-28T18:25:59.499 INFO:tasks.workunit.client.0.smithi007.stderr: 200 | char temp[BUFSIZE]; 2023-11-28T18:25:59.499 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~ 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr:In file included from /usr/include/string.h:535, 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr: from parser.c:19: 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr:In function 'memcpy', 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr: inlined from 'size64_convert' at parser.c:119:2: 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr: 30 | __glibc_objsize0 (__dest)); 2023-11-28T18:25:59.517 INFO:tasks.workunit.client.0.smithi007.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-11-28T18:25:59.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:25:59.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.802+0000 7faff292a640 1 Processor -- start 2023-11-28T18:25:59.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.802+0000 7faff292a640 1 -- start start 2023-11-28T18:25:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff292a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec154300 0x7fafec159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff292a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fafec15e090 0x7fafec0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff292a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafec15d490 con 0x7fafec15dc50 2023-11-28T18:25:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff292a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafec15a480 con 0x7fafec15e090 2023-11-28T18:25:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff292a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec15a760 con 0x7fafec154300 2023-11-28T18:25:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff0ec7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fafec15e090 0x7fafec0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff0ec7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fafec15e090 0x7fafec0ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41292/0 (socket says 172.21.15.7:41292) 2023-11-28T18:25:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff0ec7640 1 -- 172.21.15.7:0/1187915184 learned_addr learned my addr 172.21.15.7:0/1187915184 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7fafebfff640 1 --2- 172.21.15.7:0/1187915184 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec154300 0x7fafec159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff0ec7640 1 -- 172.21.15.7:0/1187915184 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec154300 msgr2=0x7fafec159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff0ec7640 1 --2- 172.21.15.7:0/1187915184 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec154300 0x7fafec159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff0ec7640 1 -- 172.21.15.7:0/1187915184 >> v1:172.21.15.193:6789/0 conn(0x7fafec15dc50 legacy=0x7fafec0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:25:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff0ec7640 1 -- 172.21.15.7:0/1187915184 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec15aa40 con 0x7fafec15e090 2023-11-28T18:25:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.806+0000 7faff0ec7640 1 --2- 172.21.15.7:0/1187915184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec15e090 0x7fafec0ca410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009e50 tx=0x7fafe401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9215c3da5d85ce0d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7fafeaffd640 1 -- 172.21.15.7:0/1187915184 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe40a5730 con 0x7fafec15e090 2023-11-28T18:25:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7fafeaffd640 1 -- 172.21.15.7:0/1187915184 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe40a5890 con 0x7fafec15e090 2023-11-28T18:25:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7fafeaffd640 1 -- 172.21.15.7:0/1187915184 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe40a5c20 con 0x7fafec15e090 2023-11-28T18:25:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- 172.21.15.7:0/1187915184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec15e090 msgr2=0x7fafec0ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 --2- 172.21.15.7:0/1187915184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec15e090 0x7fafec0ca410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009e50 tx=0x7fafe401c9a0 comp rx=0 tx=0).stop 2023-11-28T18:25:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- 172.21.15.7:0/1187915184 shutdown_connections 2023-11-28T18:25:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 --2- 172.21.15.7:0/1187915184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec15e090 0x7fafec0ca410 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 --2- 172.21.15.7:0/1187915184 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec154300 0x7fafec159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- 172.21.15.7:0/1187915184 >> 172.21.15.7:0/1187915184 conn(0x7fafec0662e0 msgr2=0x7fafec0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:25:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- 172.21.15.7:0/1187915184 shutdown_connections 2023-11-28T18:25:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- 172.21.15.7:0/1187915184 wait complete. 2023-11-28T18:25:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 Processor -- start 2023-11-28T18:25:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- start start 2023-11-28T18:25:59.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec154300 0x7fafec15b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:59.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafec15dc50 0x7fafec1529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:59.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec15e090 0x7fafec144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:59.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec143c20 con 0x7fafec154300 2023-11-28T18:25:59.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec15d080 con 0x7fafec15dc50 2023-11-28T18:25:59.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7faff292a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec143280 con 0x7fafec15e090 2023-11-28T18:25:59.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7fafebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec154300 0x7fafec15b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7fafebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec154300 0x7fafec15b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41296/0 (socket says 172.21.15.7:41296) 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7fafebfff640 1 -- 172.21.15.7:0/2427253579 learned_addr learned my addr 172.21.15.7:0/2427253579 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.810+0000 7fafeb7fe640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafec15dc50 0x7fafec1529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7faff0ec7640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec15e090 0x7fafec144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafebfff640 1 -- 172.21.15.7:0/2427253579 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec15e090 msgr2=0x7fafec144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafebfff640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec15e090 0x7fafec144040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafebfff640 1 -- 172.21.15.7:0/2427253579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafec15dc50 msgr2=0x7fafec1529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafebfff640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafec15dc50 0x7fafec1529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:25:59.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafebfff640 1 -- 172.21.15.7:0/2427253579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe4009b30 con 0x7fafec154300 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafebfff640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec154300 0x7fafec15b6c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00d800 tx=0x7fafdc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafe8ff9640 1 -- 172.21.15.7:0/2427253579 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc00b6c0 con 0x7fafec154300 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafe8ff9640 1 -- 172.21.15.7:0/2427253579 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc00b820 con 0x7fafec154300 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafe8ff9640 1 -- 172.21.15.7:0/2427253579 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc00bb90 con 0x7fafec154300 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec143560 con 0x7fafec154300 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec145080 con 0x7fafec154300 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec04e890 con 0x7fafec154300 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.814+0000 7fafe8ff9640 1 -- 172.21.15.7:0/2427253579 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fafdc00bcf0 con 0x7fafec154300 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.818+0000 7fafe8ff9640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fafd8066900 0x7fafd8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:25:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.818+0000 7fafe8ff9640 1 -- 172.21.15.7:0/2427253579 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fafdc084840 con 0x7fafec154300 2023-11-28T18:25:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.818+0000 7fafeb7fe640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fafd8066900 0x7fafd8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:25:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.818+0000 7fafeb7fe640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fafd8066900 0x7fafd8068dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fafec14ee70 tx=0x7fafe0007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:25:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:25:59.822+0000 7fafe8ff9640 1 -- 172.21.15.7:0/2427253579 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fafdc051af0 con 0x7fafec154300 2023-11-28T18:26:00.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.018+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafec0caf50 con 0x7fafec154300 2023-11-28T18:26:00.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.018+0000 7fafe8ff9640 1 -- 172.21.15.7:0/2427253579 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5766 (secure 0 0 0) 0x7fafdc055110 con 0x7fafec154300 2023-11-28T18:26:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:26:00.021 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:58.349371+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:26:00.021 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 25 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.018+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fafd8066900 msgr2=0x7fafd8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.018+0000 7faff292a640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fafd8066900 0x7fafd8068dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fafec14ee70 tx=0x7fafe0007e40 comp rx=0 tx=0).stop 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.018+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec154300 msgr2=0x7fafec15b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.018+0000 7faff292a640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec154300 0x7fafec15b6c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00d800 tx=0x7fafdc00dcc0 comp rx=0 tx=0).stop 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.022+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 shutdown_connections 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.022+0000 7faff292a640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fafd8066900 0x7fafd8068dc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.022+0000 7faff292a640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fafec15e090 0x7fafec144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.022+0000 7faff292a640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafec15dc50 0x7fafec1529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.022+0000 7faff292a640 1 --2- 172.21.15.7:0/2427253579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafec154300 0x7fafec15b6c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.022+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 >> 172.21.15.7:0/2427253579 conn(0x7fafec0662e0 msgr2=0x7fafec0666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:26:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.022+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 shutdown_connections 2023-11-28T18:26:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:00.022+0000 7faff292a640 1 -- 172.21.15.7:0/2427253579 wait complete. 2023-11-28T18:26:00.042 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=4, replacing gid=4282 2023-11-28T18:26:00.043 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2023-11-28T18:26:00.816 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2023-11-28T18:26:00.836 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2023-11-28T18:26:01.010 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2023-11-28T18:26:01.027 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2023-11-28T18:26:01.223 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2023-11-28T18:26:01.242 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2023-11-28T18:26:01.328 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/list.Tpo .deps/list.Po 2023-11-28T18:26:01.351 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2023-11-28T18:26:02.356 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2023-11-28T18:26:02.358 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd .. 2023-11-28T18:26:02.361 INFO:tasks.workunit.client.0.smithi007.stderr:+ mkdir tmp 2023-11-28T18:26:02.366 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd tmp 2023-11-28T18:26:02.367 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in $mydir/*.ffsb 2023-11-28T18:26:02.367 INFO:tasks.workunit.client.0.smithi007.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2023-11-28T18:26:02.371 INFO:tasks.workunit.client.0.smithi007.stdout:FFSB version 6.0-RC2 started 2023-11-28T18:26:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:26:02.377 INFO:tasks.workunit.client.0.smithi007.stdout:benchmark time = 300 2023-11-28T18:26:02.377 INFO:tasks.workunit.client.0.smithi007.stdout:ThreadGroup 0 2023-11-28T18:26:02.377 INFO:tasks.workunit.client.0.smithi007.stdout:================ 2023-11-28T18:26:02.377 INFO:tasks.workunit.client.0.smithi007.stdout: num_threads = 32 2023-11-28T18:26:02.377 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:26:02.377 INFO:tasks.workunit.client.0.smithi007.stdout: read_random = off 2023-11-28T18:26:02.377 INFO:tasks.workunit.client.0.smithi007.stdout: read_size = 0 (0B) 2023-11-28T18:26:02.377 INFO:tasks.workunit.client.0.smithi007.stdout: read_blocksize = 0 (0B) 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: read_skip = off 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: read_skipsize = 0 (0B) 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: write_random = on 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: write_size = 5242880 (5MB) 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: fsync_file = 0 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: write_blocksize = 4096 (4KB) 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: wait time = 0 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: op weights 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: read = 0 (0.00%) 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: readall = 0 (0.00%) 2023-11-28T18:26:02.378 INFO:tasks.workunit.client.0.smithi007.stdout: write = 1 (100.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: create = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: append = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: delete = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: metaop = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: createdir = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: stat = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: writeall = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: writeall_fsync = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: open_close = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: write_fsync = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: create_fsync = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: append_fsync = 0 (0.00%) 2023-11-28T18:26:02.379 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout:FileSystem . 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout:========== 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: num_dirs = 0 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: starting files = 128 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: min file size = 104857600 (100MB) 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: max file size = 104857600 (100MB) 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: directio = off 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: alignedio = on 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: bufferedio = off 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:26:02.380 INFO:tasks.workunit.client.0.smithi007.stdout: aging is off 2023-11-28T18:26:02.381 INFO:tasks.workunit.client.0.smithi007.stdout: current utilization = 0.00% 2023-11-28T18:26:02.381 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:26:02.381 INFO:tasks.workunit.client.0.smithi007.stdout:checking existing fs: . 2023-11-28T18:26:02.381 INFO:tasks.workunit.client.0.smithi007.stdout:recreating new fileset 2023-11-28T18:26:02.382 INFO:tasks.workunit.client.0.smithi007.stderr:opendir: No such file or directory 2023-11-28T18:26:09.044 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-11-28T18:26:09.044 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T18:26:09.044 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T18:26:09.048 INFO:tasks.ceph.mds.b:Started 2023-11-28T18:26:09.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:26:09.132 INFO:tasks.ceph.mds.b.smithi193.stdout:starting mds.b at 2023-11-28T18:26:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.193+0000 7f6363e17640 1 Processor -- start 2023-11-28T18:26:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.193+0000 7f6363e17640 1 -- start start 2023-11-28T18:26:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6363e17640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f635c1542e0 0x7f635c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:26:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6363e17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cbb70 0x7f635c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:26:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6363e17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635c0ca920 con 0x7f635c0cc810 2023-11-28T18:26:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6363e17640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635c071d50 con 0x7f635c1542e0 2023-11-28T18:26:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6363e17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c072030 con 0x7f635c0cbb70 2023-11-28T18:26:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6361bb3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f635c1542e0 0x7f635c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:26:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f63623b4640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f635c0cc810 0x7f635c072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37904/0 (socket says 172.21.15.7:37904) 2023-11-28T18:26:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f63623b4640 1 -- 172.21.15.7:0/4106673681 learned_addr learned my addr 172.21.15.7:0/4106673681 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:26:09.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f63613b2640 1 --2- 172.21.15.7:0/4106673681 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cbb70 0x7f635c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:26:09.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6361bb3640 1 -- 172.21.15.7:0/4106673681 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cbb70 msgr2=0x7f635c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:26:09.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6361bb3640 1 --2- 172.21.15.7:0/4106673681 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cbb70 0x7f635c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6361bb3640 1 -- 172.21.15.7:0/4106673681 >> v1:172.21.15.7:6789/0 conn(0x7f635c0cc810 legacy=0x7f635c072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:26:09.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6361bb3640 1 -- 172.21.15.7:0/4106673681 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c15e550 con 0x7f635c1542e0 2023-11-28T18:26:09.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.197+0000 7f6360bb1640 1 -- 172.21.15.7:0/4106673681 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221298225 0 0) 0x7f635c0ca920 con 0x7f635c0cc810 2023-11-28T18:26:09.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6361bb3640 1 --2- 172.21.15.7:0/4106673681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c1542e0 0x7f635c15e010 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63580097c0 tx=0x7f635801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a965d3201519fb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:26:09.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6360bb1640 1 -- 172.21.15.7:0/4106673681 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63580ad640 con 0x7f635c1542e0 2023-11-28T18:26:09.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6360bb1640 1 -- 172.21.15.7:0/4106673681 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63580ad7a0 con 0x7f635c1542e0 2023-11-28T18:26:09.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6360bb1640 1 -- 172.21.15.7:0/4106673681 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63580adb30 con 0x7f635c1542e0 2023-11-28T18:26:09.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6363e17640 1 -- 172.21.15.7:0/4106673681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c1542e0 msgr2=0x7f635c15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:26:09.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6363e17640 1 --2- 172.21.15.7:0/4106673681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c1542e0 0x7f635c15e010 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63580097c0 tx=0x7f635801c480 comp rx=0 tx=0).stop 2023-11-28T18:26:09.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6363e17640 1 -- 172.21.15.7:0/4106673681 shutdown_connections 2023-11-28T18:26:09.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6363e17640 1 --2- 172.21.15.7:0/4106673681 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cbb70 0x7f635c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6363e17640 1 --2- 172.21.15.7:0/4106673681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c1542e0 0x7f635c15e010 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6363e17640 1 -- 172.21.15.7:0/4106673681 >> 172.21.15.7:0/4106673681 conn(0x7f635c0717e0 msgr2=0x7f635c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:26:09.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6363e17640 1 -- 172.21.15.7:0/4106673681 shutdown_connections 2023-11-28T18:26:09.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.201+0000 7f6363e17640 1 -- 172.21.15.7:0/4106673681 wait complete. 2023-11-28T18:26:09.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 Processor -- start 2023-11-28T18:26:09.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 -- start start 2023-11-28T18:26:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c0cbb70 0x7f635c0c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:26:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cc810 0x7f635c140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:26:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635c1542e0 0x7f635c1405d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:26:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c15b270 con 0x7f635c0cbb70 2023-11-28T18:26:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c146810 con 0x7f635c1542e0 2023-11-28T18:26:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c0cf770 con 0x7f635c0cc810 2023-11-28T18:26:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c0cbb70 0x7f635c0c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:26:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c0cbb70 0x7f635c0c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39970/0 (socket says 172.21.15.7:39970) 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 -- 172.21.15.7:0/3323850003 learned_addr learned my addr 172.21.15.7:0/3323850003 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f63623b4640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635c1542e0 0x7f635c1405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f63613b2640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cc810 0x7f635c140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 -- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cc810 msgr2=0x7f635c140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cc810 0x7f635c140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 -- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635c1542e0 msgr2=0x7f635c1405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635c1542e0 0x7f635c1405d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 -- 172.21.15.7:0/3323850003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358009420 con 0x7f635c0cbb70 2023-11-28T18:26:09.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f63623b4640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635c1542e0 0x7f635c1405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:26:09.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6361bb3640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c0cbb70 0x7f635c0c7540 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f635801a0b0 tx=0x7f635801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:26:09.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f63527fc640 1 -- 172.21.15.7:0/3323850003 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358004230 con 0x7f635c0cbb70 2023-11-28T18:26:09.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f63527fc640 1 -- 172.21.15.7:0/3323850003 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63580a6070 con 0x7f635c0cbb70 2023-11-28T18:26:09.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f63527fc640 1 -- 172.21.15.7:0/3323850003 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358003620 con 0x7f635c0cbb70 2023-11-28T18:26:09.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c146a80 con 0x7f635c0cbb70 2023-11-28T18:26:09.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.205+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c147040 con 0x7f635c0cbb70 2023-11-28T18:26:09.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.209+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c06c670 con 0x7f635c0cbb70 2023-11-28T18:26:09.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.209+0000 7f63527fc640 1 -- 172.21.15.7:0/3323850003 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6358003780 con 0x7f635c0cbb70 2023-11-28T18:26:09.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.209+0000 7f63527fc640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63400668d0 0x7f6340068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:26:09.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.209+0000 7f63527fc640 1 -- 172.21.15.7:0/3323850003 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f635811dd50 con 0x7f635c0cbb70 2023-11-28T18:26:09.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.209+0000 7f63613b2640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63400668d0 0x7f6340068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:26:09.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.213+0000 7f63613b2640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63400668d0 0x7f6340068d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f635c072190 tx=0x7f634c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:26:09.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.217+0000 7f63527fc640 1 -- 172.21.15.7:0/3323850003 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6358121030 con 0x7f635c0cbb70 2023-11-28T18:26:09.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.449+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f635c04e890 con 0x7f635c0cbb70 2023-11-28T18:26:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.449+0000 7f63527fc640 1 -- 172.21.15.7:0/3323850003 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6559 (secure 0 0 0) 0x7f63580b2090 con 0x7f635c0cbb70 2023-11-28T18:26:09.454 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:26:09.454 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:25:58.349371+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:26:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 26 2023-11-28T18:26:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63400668d0 msgr2=0x7f6340068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:26:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63400668d0 0x7f6340068d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f635c072190 tx=0x7f634c009190 comp rx=0 tx=0).stop 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c0cbb70 msgr2=0x7f635c0c7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c0cbb70 0x7f635c0c7540 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f635801a0b0 tx=0x7f635801acc0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 shutdown_connections 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63400668d0 0x7f6340068d90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635c1542e0 0x7f635c1405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f635c0cc810 0x7f635c140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 --2- 172.21.15.7:0/3323850003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c0cbb70 0x7f635c0c7540 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 >> 172.21.15.7:0/3323850003 conn(0x7f635c0717e0 msgr2=0x7f635c15b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 shutdown_connections 2023-11-28T18:26:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:26:09.453+0000 7f6363e17640 1 -- 172.21.15.7:0/3323850003 wait complete. 2023-11-28T18:26:09.475 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-11-28T18:26:09.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 106 secs before thrashing 2023-11-28T18:26:28.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:26:28.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:26:58.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:26:58.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:27:28.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:27:28.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:27:55.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:27:55.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3ca609640 1 Processor -- start 2023-11-28T18:27:55.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3ca609640 1 -- start start 2023-11-28T18:27:55.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3ca609640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa3c406e560 0x7fa3c415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:55.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3ca609640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c4071e60 con 0x7fa3c4151cd0 2023-11-28T18:27:55.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3ca609640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c4070930 con 0x7fa3c4157b90 2023-11-28T18:27:55.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3ca609640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c4157fa0 con 0x7fa3c406e560 2023-11-28T18:27:55.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3c37fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa3c406e560 0x7fa3c415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:55.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3c3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa3c4151cd0 0x7fa3c40c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51056/0 (socket says 172.21.15.7:51056) 2023-11-28T18:27:55.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3c3fff640 1 -- 172.21.15.7:0/353867792 learned_addr learned my addr 172.21.15.7:0/353867792 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:27:55.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3c37fe640 1 -- 172.21.15.7:0/353867792 >> v1:172.21.15.193:6789/0 conn(0x7fa3c4157b90 legacy=0x7fa3c4162510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:55.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3c37fe640 1 -- 172.21.15.7:0/353867792 >> v1:172.21.15.7:6789/0 conn(0x7fa3c4151cd0 legacy=0x7fa3c40c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:55.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.628+0000 7fa3c37fe640 1 -- 172.21.15.7:0/353867792 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c4070bf0 con 0x7fa3c406e560 2023-11-28T18:27:55.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3c37fe640 1 --2- 172.21.15.7:0/353867792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c415bd00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80097c0 tx=0x7fa3b801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f83e510da58ae5d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:55.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3c2ffd640 1 -- 172.21.15.7:0/353867792 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b80ab600 con 0x7fa3c406e560 2023-11-28T18:27:55.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3c2ffd640 1 -- 172.21.15.7:0/353867792 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b80a7050 con 0x7fa3c406e560 2023-11-28T18:27:55.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3c2ffd640 1 -- 172.21.15.7:0/353867792 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b80ab980 con 0x7fa3c406e560 2023-11-28T18:27:55.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3ca609640 1 -- 172.21.15.7:0/353867792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 msgr2=0x7fa3c415bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:55.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3ca609640 1 --2- 172.21.15.7:0/353867792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c415bd00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80097c0 tx=0x7fa3b801c3f0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3ca609640 1 -- 172.21.15.7:0/353867792 shutdown_connections 2023-11-28T18:27:55.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3ca609640 1 --2- 172.21.15.7:0/353867792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c415bd00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3ca609640 1 -- 172.21.15.7:0/353867792 >> 172.21.15.7:0/353867792 conn(0x7fa3c40662e0 msgr2=0x7fa3c40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:27:55.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3ca609640 1 -- 172.21.15.7:0/353867792 shutdown_connections 2023-11-28T18:27:55.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.632+0000 7fa3ca609640 1 -- 172.21.15.7:0/353867792 wait complete. 2023-11-28T18:27:55.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 Processor -- start 2023-11-28T18:27:55.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 -- start start 2023-11-28T18:27:55.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c4162e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:55.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3c4151cd0 0x7fa3c41f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:55.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa3c4157b90 0x7fa3c41f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:55.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c414d060 con 0x7fa3c406e560 2023-11-28T18:27:55.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c414d930 con 0x7fa3c4151cd0 2023-11-28T18:27:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c4150510 con 0x7fa3c4157b90 2023-11-28T18:27:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c4162e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c4162e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51732/0 (socket says 172.21.15.7:51732) 2023-11-28T18:27:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 -- 172.21.15.7:0/4053267243 learned_addr learned my addr 172.21.15.7:0/4053267243 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:27:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c37fe640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3c4151cd0 0x7fa3c41f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c8ba6640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa3c4157b90 0x7fa3c41f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:55.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 -- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa3c4157b90 msgr2=0x7fa3c41f61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:55.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa3c4157b90 0x7fa3c41f61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 -- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3c4151cd0 msgr2=0x7fa3c41f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:55.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3c4151cd0 0x7fa3c41f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 -- 172.21.15.7:0/4053267243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8009420 con 0x7fa3c406e560 2023-11-28T18:27:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c8ba6640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa3c4157b90 0x7fa3c41f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:27:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c3fff640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c4162e20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400c500 tx=0x7fa3b400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c0ff9640 1 -- 172.21.15.7:0/4053267243 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4005cf0 con 0x7fa3c406e560 2023-11-28T18:27:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c0ff9640 1 -- 172.21.15.7:0/4053267243 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b4005e50 con 0x7fa3c406e560 2023-11-28T18:27:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c414bfd0 con 0x7fa3c406e560 2023-11-28T18:27:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3c0ff9640 1 -- 172.21.15.7:0/4053267243 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b40072f0 con 0x7fa3c406e560 2023-11-28T18:27:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.636+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c414c270 con 0x7fa3c406e560 2023-11-28T18:27:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.640+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c404e890 con 0x7fa3c406e560 2023-11-28T18:27:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.640+0000 7fa3c0ff9640 1 -- 172.21.15.7:0/4053267243 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa3b4007450 con 0x7fa3c406e560 2023-11-28T18:27:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.640+0000 7fa3c0ff9640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa3b00668d0 0x7fa3b0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.640+0000 7fa3c0ff9640 1 -- 172.21.15.7:0/4053267243 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fa3b40839c0 con 0x7fa3c406e560 2023-11-28T18:27:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.640+0000 7fa3c37fe640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa3b00668d0 0x7fa3b0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.640+0000 7fa3c37fe640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa3b00668d0 0x7fa3b0068d90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80ad000 tx=0x7fa3b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:55.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.648+0000 7fa3c0ff9640 1 -- 172.21.15.7:0/4053267243 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa3b4050bd0 con 0x7fa3c406e560 2023-11-28T18:27:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.856+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3c4155d70 con 0x7fa3c406e560 2023-11-28T18:27:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.856+0000 7fa3c0ff9640 1 -- 172.21.15.7:0/4053267243 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6557 (secure 0 0 0) 0x7fa3b4017020 con 0x7fa3c406e560 2023-11-28T18:27:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:27:55.861 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:27:18.707264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:27:55.861 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-11-28T18:27:55.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa3b00668d0 msgr2=0x7fa3b0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:55.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa3b00668d0 0x7fa3b0068d90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80ad000 tx=0x7fa3b8008040 comp rx=0 tx=0).stop 2023-11-28T18:27:55.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 msgr2=0x7fa3c4162e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c4162e20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400c500 tx=0x7fa3b400c9c0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 shutdown_connections 2023-11-28T18:27:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa3b00668d0 0x7fa3b0068d90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa3c4157b90 0x7fa3c41f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3c4151cd0 0x7fa3c41f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 --2- 172.21.15.7:0/4053267243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3c406e560 0x7fa3c4162e20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 >> 172.21.15.7:0/4053267243 conn(0x7fa3c40662e0 msgr2=0x7fa3c40c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:27:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 shutdown_connections 2023-11-28T18:27:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:55.860+0000 7fa3ca609640 1 -- 172.21.15.7:0/4053267243 wait complete. 2023-11-28T18:27:55.882 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=4) 2023-11-28T18:27:55.883 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-11-28T18:27:55.883 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:27:55.900 INFO:tasks.ceph.mds.f:Stopped 2023-11-28T18:27:55.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:27:56.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f27a1d3b640 1 Processor -- start 2023-11-28T18:27:56.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f27a1d3b640 1 -- start start 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f27a1d3b640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f279c15ca30 0x7f279c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f27a1d3b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f279c15ce10 0x7f279c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f27a1d3b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c15a620 con 0x7f279c0cafe0 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f27a1d3b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279c15a900 con 0x7f279c15ce10 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f27a1d3b640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279c15abe0 con 0x7f279c15ca30 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f279bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f279c15ce10 0x7f279c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f279bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f279c15ce10 0x7f279c0ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51746/0 (socket says 172.21.15.7:51746) 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.056+0000 7f279bfff640 1 -- 172.21.15.7:0/868263041 learned_addr learned my addr 172.21.15.7:0/868263041 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:27:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.060+0000 7f27a0d39640 1 --2- 172.21.15.7:0/868263041 >> v2:172.21.15.193:3300/0 conn(0x7f279c15ca30 0x7f279c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:56.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.060+0000 7f279bfff640 1 -- 172.21.15.7:0/868263041 >> v2:172.21.15.193:3300/0 conn(0x7f279c15ca30 msgr2=0x7f279c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:27:56.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.060+0000 7f279bfff640 1 --2- 172.21.15.7:0/868263041 >> v2:172.21.15.193:3300/0 conn(0x7f279c15ca30 0x7f279c159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.060+0000 7f279bfff640 1 -- 172.21.15.7:0/868263041 >> v1:172.21.15.193:6789/0 conn(0x7f279c0cafe0 legacy=0x7f279c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:27:56.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.060+0000 7f279bfff640 1 -- 172.21.15.7:0/868263041 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c0cbb10 con 0x7f279c15ce10 2023-11-28T18:27:56.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.060+0000 7f279bfff640 1 --2- 172.21.15.7:0/868263041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c0ca1d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f279c06c660 tx=0x7f278801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14752fe561629f7a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f279b7fe640 1 -- 172.21.15.7:0/868263041 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27880ac600 con 0x7f279c15ce10 2023-11-28T18:27:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f279b7fe640 1 -- 172.21.15.7:0/868263041 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27880a8050 con 0x7f279c15ce10 2023-11-28T18:27:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f279b7fe640 1 -- 172.21.15.7:0/868263041 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27880ac980 con 0x7f279c15ce10 2023-11-28T18:27:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 -- 172.21.15.7:0/868263041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 msgr2=0x7f279c0ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/868263041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c0ca1d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f279c06c660 tx=0x7f278801c490 comp rx=0 tx=0).stop 2023-11-28T18:27:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 -- 172.21.15.7:0/868263041 shutdown_connections 2023-11-28T18:27:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/868263041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c0ca1d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/868263041 >> v2:172.21.15.193:3300/0 conn(0x7f279c15ca30 0x7f279c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 -- 172.21.15.7:0/868263041 >> 172.21.15.7:0/868263041 conn(0x7f279c06c060 msgr2=0x7f279c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:27:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 -- 172.21.15.7:0/868263041 shutdown_connections 2023-11-28T18:27:56.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 -- 172.21.15.7:0/868263041 wait complete. 2023-11-28T18:27:56.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 Processor -- start 2023-11-28T18:27:56.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.064+0000 7f27a1d3b640 1 -- start start 2023-11-28T18:27:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f279c0cafe0 0x7f279c154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f279c15ca30 0x7f279c1596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c14b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c1542f0 con 0x7f279c15ce10 2023-11-28T18:27:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c155150 con 0x7f279c15ca30 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279c147200 con 0x7f279c0cafe0 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c14b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c14b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51750/0 (socket says 172.21.15.7:51750) 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 -- 172.21.15.7:0/1241306500 learned_addr learned my addr 172.21.15.7:0/1241306500 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f279bfff640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f279c15ca30 0x7f279c1596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a0d39640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f279c0cafe0 0x7f279c154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 -- 172.21.15.7:0/1241306500 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f279c0cafe0 msgr2=0x7f279c154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f279c0cafe0 0x7f279c154c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 -- 172.21.15.7:0/1241306500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f279c15ca30 msgr2=0x7f279c1596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f279c15ca30 0x7f279c1596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 -- 172.21.15.7:0/1241306500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880094d0 con 0x7f279c15ce10 2023-11-28T18:27:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a153a640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c14b4b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f279400c780 tx=0x7f279400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:56.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27997fa640 1 -- 172.21.15.7:0/1241306500 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794004b70 con 0x7f279c15ce10 2023-11-28T18:27:56.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27997fa640 1 -- 172.21.15.7:0/1241306500 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279400ce90 con 0x7f279c15ce10 2023-11-28T18:27:56.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279c153d70 con 0x7f279c15ce10 2023-11-28T18:27:56.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27997fa640 1 -- 172.21.15.7:0/1241306500 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279400a850 con 0x7f279c15ce10 2023-11-28T18:27:56.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c154010 con 0x7f279c15ce10 2023-11-28T18:27:56.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.068+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279c04e890 con 0x7f279c15ce10 2023-11-28T18:27:56.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.072+0000 7f27997fa640 1 -- 172.21.15.7:0/1241306500 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f279400a9b0 con 0x7f279c15ce10 2023-11-28T18:27:56.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.072+0000 7f27997fa640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2780066900 0x7f2780068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:56.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.072+0000 7f27997fa640 1 -- 172.21.15.7:0/1241306500 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f27940846a0 con 0x7f279c15ce10 2023-11-28T18:27:56.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.072+0000 7f27a0d39640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2780066900 0x7f2780068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:56.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.072+0000 7f27a0d39640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2780066900 0x7f2780068dc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2790007a60 tx=0x7f2790008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.076+0000 7f27997fa640 1 -- 172.21.15.7:0/1241306500 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2794051790 con 0x7f279c15ce10 2023-11-28T18:27:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.352+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f279c0cdca0 con 0x7f279c15ce10 2023-11-28T18:27:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27997fa640 1 -- 172.21.15.7:0/1241306500 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6557 (secure 0 0 0) 0x7f2794054db0 con 0x7f279c15ce10 2023-11-28T18:27:56.359 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:27:56.359 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:27:18.707264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:27:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-11-28T18:27:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2780066900 msgr2=0x7f2780068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2780066900 0x7f2780068dc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2790007a60 tx=0x7f2790008040 comp rx=0 tx=0).stop 2023-11-28T18:27:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 msgr2=0x7f279c14b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c14b4b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f279400c780 tx=0x7f279400cc40 comp rx=0 tx=0).stop 2023-11-28T18:27:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 shutdown_connections 2023-11-28T18:27:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2780066900 0x7f2780068dc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f279c15ce10 0x7f279c14b4b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f279c15ca30 0x7f279c1596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 --2- 172.21.15.7:0/1241306500 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f279c0cafe0 0x7f279c154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 >> 172.21.15.7:0/1241306500 conn(0x7f279c06c060 msgr2=0x7f279c0cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:27:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 shutdown_connections 2023-11-28T18:27:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:56.356+0000 7f27a1d3b640 1 -- 172.21.15.7:0/1241306500 wait complete. 2023-11-28T18:27:56.384 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:27:58.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:27:58.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858f51d640 1 Processor -- start 2023-11-28T18:27:58.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858f51d640 1 -- start start 2023-11-28T18:27:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858f51d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8588154300 0x7f8588159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858f51d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 0x7f85880ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858f51d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858815d490 con 0x7f858815dc50 2023-11-28T18:27:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858f51d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f858815a480 con 0x7f8588154300 2023-11-28T18:27:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858f51d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858815a760 con 0x7f858815e090 2023-11-28T18:27:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858d2b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8588154300 0x7f8588159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858d2b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8588154300 0x7f8588159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51766/0 (socket says 172.21.15.7:51766) 2023-11-28T18:27:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858d2b9640 1 -- 172.21.15.7:0/3275758640 learned_addr learned my addr 172.21.15.7:0/3275758640 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:27:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858d2b9640 1 -- 172.21.15.7:0/3275758640 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 msgr2=0x7f85880ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:27:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858d2b9640 1 --2- 172.21.15.7:0/3275758640 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 0x7f85880ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858d2b9640 1 -- 172.21.15.7:0/3275758640 >> v1:172.21.15.193:6789/0 conn(0x7f858815dc50 legacy=0x7f85880cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:27:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.532+0000 7f858d2b9640 1 -- 172.21.15.7:0/3275758640 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858815aa40 con 0x7f8588154300 2023-11-28T18:27:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858d2b9640 1 --2- 172.21.15.7:0/3275758640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588159e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8584009870 tx=0x7f858401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13669352990a22e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f857ffff640 1 -- 172.21.15.7:0/3275758640 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840ac600 con 0x7f8588154300 2023-11-28T18:27:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f857ffff640 1 -- 172.21.15.7:0/3275758640 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85840a8050 con 0x7f8588154300 2023-11-28T18:27:58.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f857ffff640 1 -- 172.21.15.7:0/3275758640 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840ac980 con 0x7f8588154300 2023-11-28T18:27:58.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858f51d640 1 -- 172.21.15.7:0/3275758640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 msgr2=0x7f8588159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:58.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858f51d640 1 --2- 172.21.15.7:0/3275758640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588159e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8584009870 tx=0x7f858401c4d0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858f51d640 1 -- 172.21.15.7:0/3275758640 shutdown_connections 2023-11-28T18:27:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858f51d640 1 --2- 172.21.15.7:0/3275758640 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 0x7f85880ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858f51d640 1 --2- 172.21.15.7:0/3275758640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588159e40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858f51d640 1 -- 172.21.15.7:0/3275758640 >> 172.21.15.7:0/3275758640 conn(0x7f85880662e0 msgr2=0x7f85880666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:27:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858f51d640 1 -- 172.21.15.7:0/3275758640 shutdown_connections 2023-11-28T18:27:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.536+0000 7f858f51d640 1 -- 172.21.15.7:0/3275758640 wait complete. 2023-11-28T18:27:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858f51d640 1 Processor -- start 2023-11-28T18:27:58.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858f51d640 1 -- start start 2023-11-28T18:27:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858f51d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858f51d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f858815dc50 0x7f8588140180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858f51d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 0x7f85881406c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858f51d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858815ca70 con 0x7f8588154300 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858f51d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858815c2d0 con 0x7f858815dc50 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858f51d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858813f530 con 0x7f858815e090 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858d2b9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858d2b9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51770/0 (socket says 172.21.15.7:51770) 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858d2b9640 1 -- 172.21.15.7:0/2579997498 learned_addr learned my addr 172.21.15.7:0/2579997498 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858cab8640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f858815dc50 0x7f8588140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.540+0000 7f858daba640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 0x7f85881406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858d2b9640 1 -- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 msgr2=0x7f85881406c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858d2b9640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 0x7f85881406c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858d2b9640 1 -- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f858815dc50 msgr2=0x7f8588140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858d2b9640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f858815dc50 0x7f8588140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858d2b9640 1 -- 172.21.15.7:0/2579997498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85840094d0 con 0x7f8588154300 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858cab8640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f858815dc50 0x7f8588140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858d2b9640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588151d90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f858401ab80 tx=0x7f858401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f857dffb640 1 -- 172.21.15.7:0/2579997498 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840b7830 con 0x7f8588154300 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f857dffb640 1 -- 172.21.15.7:0/2579997498 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85840b79b0 con 0x7f8588154300 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858815b3e0 con 0x7f8588154300 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f857dffb640 1 -- 172.21.15.7:0/2579997498 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840b7d60 con 0x7f8588154300 2023-11-28T18:27:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858815b9a0 con 0x7f8588154300 2023-11-28T18:27:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858804e890 con 0x7f8588154300 2023-11-28T18:27:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.544+0000 7f857dffb640 1 -- 172.21.15.7:0/2579997498 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f85840b3040 con 0x7f8588154300 2023-11-28T18:27:58.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.548+0000 7f857dffb640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f856c066900 0x7f856c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:27:58.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.548+0000 7f857dffb640 1 -- 172.21.15.7:0/2579997498 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f858411e8c0 con 0x7f8588154300 2023-11-28T18:27:58.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.548+0000 7f858cab8640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f856c066900 0x7f856c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:27:58.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.548+0000 7f858cab8640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f856c066900 0x7f856c068dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f85880c9d20 tx=0x7f8578009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:27:58.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.552+0000 7f857dffb640 1 -- 172.21.15.7:0/2579997498 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f85840eb9b0 con 0x7f8588154300 2023-11-28T18:27:58.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:27:58.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:27:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.768+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85880caf50 con 0x7f8588154300 2023-11-28T18:27:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f857dffb640 1 -- 172.21.15.7:0/2579997498 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6557 (secure 0 0 0) 0x7f85840eefd0 con 0x7f8588154300 2023-11-28T18:27:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:27:58.775 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:27:18.707264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:27:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-11-28T18:27:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f856c066900 msgr2=0x7f856c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f856c066900 0x7f856c068dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f85880c9d20 tx=0x7f8578009040 comp rx=0 tx=0).stop 2023-11-28T18:27:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 msgr2=0x7f8588151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:27:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588151d90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f858401ab80 tx=0x7f858401ad10 comp rx=0 tx=0).stop 2023-11-28T18:27:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 shutdown_connections 2023-11-28T18:27:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f856c066900 0x7f856c068dc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f858815e090 0x7f85881406c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f858815dc50 0x7f8588140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 --2- 172.21.15.7:0/2579997498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8588154300 0x7f8588151d90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:27:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 >> 172.21.15.7:0/2579997498 conn(0x7f85880662e0 msgr2=0x7f85880666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:27:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 shutdown_connections 2023-11-28T18:27:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:27:58.772+0000 7f858f51d640 1 -- 172.21.15.7:0/2579997498 wait complete. 2023-11-28T18:27:58.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:28:00.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:00.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.948+0000 7faa208bb640 1 Processor -- start 2023-11-28T18:28:00.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.948+0000 7faa208bb640 1 -- start start 2023-11-28T18:28:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.948+0000 7faa208bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c8010 0x7faa1c1507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.948+0000 7faa208bb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c0c5d10 con 0x7faa1c0c7270 2023-11-28T18:28:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.948+0000 7faa208bb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c1569b0 con 0x7faa1c0c7c30 2023-11-28T18:28:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.948+0000 7faa208bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c070670 con 0x7faa1c0c8010 2023-11-28T18:28:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faa1c0c7270 0x7faa1c0c61a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51068/0 (socket says 172.21.15.7:51068) 2023-11-28T18:28:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1bfff640 1 -- 172.21.15.7:0/2017765896 learned_addr learned my addr 172.21.15.7:0/2017765896 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:28:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1affd640 1 --2- 172.21.15.7:0/2017765896 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c8010 0x7faa1c1507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1a7fc640 1 -- 172.21.15.7:0/2017765896 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844048247 0 0) 0x7faa1c0c5d10 con 0x7faa1c0c7270 2023-11-28T18:28:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1affd640 1 -- 172.21.15.7:0/2017765896 >> v1:172.21.15.193:6789/0 conn(0x7faa1c0c7c30 legacy=0x7faa1c0c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1affd640 1 -- 172.21.15.7:0/2017765896 >> v1:172.21.15.7:6789/0 conn(0x7faa1c0c7270 legacy=0x7faa1c0c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1affd640 1 -- 172.21.15.7:0/2017765896 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c156c70 con 0x7faa1c0c8010 2023-11-28T18:28:00.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1affd640 1 --2- 172.21.15.7:0/2017765896 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c8010 0x7faa1c1507f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faa10009770 tx=0x7faa1001c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45d678f1efe41a30 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1a7fc640 1 -- 172.21.15.7:0/2017765896 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100ac600 con 0x7faa1c0c8010 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1a7fc640 1 -- 172.21.15.7:0/2017765896 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa100a8050 con 0x7faa1c0c8010 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.952+0000 7faa1a7fc640 1 -- 172.21.15.7:0/2017765896 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100ac980 con 0x7faa1c0c8010 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- 172.21.15.7:0/2017765896 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c8010 msgr2=0x7faa1c1507f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 --2- 172.21.15.7:0/2017765896 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c8010 0x7faa1c1507f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faa10009770 tx=0x7faa1001c380 comp rx=0 tx=0).stop 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- 172.21.15.7:0/2017765896 shutdown_connections 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 --2- 172.21.15.7:0/2017765896 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c8010 0x7faa1c1507f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- 172.21.15.7:0/2017765896 >> 172.21.15.7:0/2017765896 conn(0x7faa1c06c370 msgr2=0x7faa1c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- 172.21.15.7:0/2017765896 shutdown_connections 2023-11-28T18:28:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- 172.21.15.7:0/2017765896 wait complete. 2023-11-28T18:28:00.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 Processor -- start 2023-11-28T18:28:00.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- start start 2023-11-28T18:28:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c7270 0x7faa1c0cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1c0c7c30 0x7faa1c13fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa1c0c8010 0x7faa1c155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:00.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c13a460 con 0x7faa1c0c7c30 2023-11-28T18:28:00.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c13f0b0 con 0x7faa1c0c8010 2023-11-28T18:28:00.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa208bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c13a1c0 con 0x7faa1c0c7270 2023-11-28T18:28:00.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.956+0000 7faa1affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1c0c7c30 0x7faa1c13fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:00.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1c0c7c30 0x7faa1c13fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51782/0 (socket says 172.21.15.7:51782) 2023-11-28T18:28:00.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1affd640 1 -- 172.21.15.7:0/4208653333 learned_addr learned my addr 172.21.15.7:0/4208653333 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:00.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1b7fe640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c7270 0x7faa1c0cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:00.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1bfff640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa1c0c8010 0x7faa1c155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:00.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1affd640 1 -- 172.21.15.7:0/4208653333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c7270 msgr2=0x7faa1c0cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:00.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1affd640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c7270 0x7faa1c0cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:00.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1affd640 1 -- 172.21.15.7:0/4208653333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa1c0c8010 msgr2=0x7faa1c155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:00.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1affd640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa1c0c8010 0x7faa1c155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:00.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1affd640 1 -- 172.21.15.7:0/4208653333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10009420 con 0x7faa1c0c7c30 2023-11-28T18:28:00.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa1affd640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1c0c7c30 0x7faa1c13fb00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faa1001aa30 tx=0x7faa10019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa03fff640 1 -- 172.21.15.7:0/4208653333 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100be070 con 0x7faa1c0c7c30 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa03fff640 1 -- 172.21.15.7:0/4208653333 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa100b6740 con 0x7faa1c0c7c30 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa03fff640 1 -- 172.21.15.7:0/4208653333 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100b6af0 con 0x7faa1c0c7c30 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c13de40 con 0x7faa1c0c7c30 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c13f5e0 con 0x7faa1c0c7c30 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c157210 con 0x7faa1c0c7c30 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.960+0000 7faa03fff640 1 -- 172.21.15.7:0/4208653333 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7faa100b3040 con 0x7faa1c0c7c30 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.964+0000 7faa03fff640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa080668d0 0x7faa08068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.964+0000 7faa03fff640 1 -- 172.21.15.7:0/4208653333 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7faa1011e4e0 con 0x7faa1c0c7c30 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.964+0000 7faa1b7fe640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa080668d0 0x7faa08068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.964+0000 7faa1b7fe640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa080668d0 0x7faa08068d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009fd0 tx=0x7faa0c009ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:00.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:00.968+0000 7faa03fff640 1 -- 172.21.15.7:0/4208653333 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faa1c157210 con 0x7faa1c0c7c30 2023-11-28T18:28:01.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.172+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa1c157210 con 0x7faa1c0c7c30 2023-11-28T18:28:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa03fff640 1 -- 172.21.15.7:0/4208653333 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6557 (secure 0 0 0) 0x7faa1c157210 con 0x7faa1c0c7c30 2023-11-28T18:28:01.179 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:01.179 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:27:18.707264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-11-28T18:28:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa080668d0 msgr2=0x7faa08068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa080668d0 0x7faa08068d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009fd0 tx=0x7faa0c009ac0 comp rx=0 tx=0).stop 2023-11-28T18:28:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1c0c7c30 msgr2=0x7faa1c13fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1c0c7c30 0x7faa1c13fb00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faa1001aa30 tx=0x7faa10019120 comp rx=0 tx=0).stop 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 shutdown_connections 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa080668d0 0x7faa08068d90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa1c0c8010 0x7faa1c155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1c0c7c30 0x7faa1c13fb00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 --2- 172.21.15.7:0/4208653333 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa1c0c7270 0x7faa1c0cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 >> 172.21.15.7:0/4208653333 conn(0x7faa1c06c370 msgr2=0x7faa1c071940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 shutdown_connections 2023-11-28T18:28:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:01.176+0000 7faa208bb640 1 -- 172.21.15.7:0/4208653333 wait complete. 2023-11-28T18:28:01.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:28:03.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:03.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.368+0000 7fbf67b80640 1 Processor -- start 2023-11-28T18:28:03.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.368+0000 7fbf67b80640 1 -- start start 2023-11-28T18:28:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf67b80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cbb70 0x7fbf600cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf67b80640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbf600cc810 0x7fbf60072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf67b80640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf600ca920 con 0x7fbf601542e0 2023-11-28T18:28:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf67b80640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf60071d50 con 0x7fbf600cc810 2023-11-28T18:28:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf67b80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60072030 con 0x7fbf600cbb70 2023-11-28T18:28:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6611d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbf600cc810 0x7fbf60072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6591c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fbf601542e0 0x7fbf6015e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:44110/0 (socket says 172.21.15.7:44110) 2023-11-28T18:28:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6591c640 1 -- 172.21.15.7:0/1721122054 learned_addr learned my addr 172.21.15.7:0/1721122054 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:28:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6511b640 1 --2- 172.21.15.7:0/1721122054 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cbb70 0x7fbf600cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6611d640 1 -- 172.21.15.7:0/1721122054 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cbb70 msgr2=0x7fbf600cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:03.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6611d640 1 --2- 172.21.15.7:0/1721122054 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cbb70 0x7fbf600cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6611d640 1 -- 172.21.15.7:0/1721122054 >> v1:172.21.15.193:6789/0 conn(0x7fbf601542e0 legacy=0x7fbf6015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:03.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6611d640 1 -- 172.21.15.7:0/1721122054 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf600c4cd0 con 0x7fbf600cc810 2023-11-28T18:28:03.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.372+0000 7fbf6491a640 1 -- 172.21.15.7:0/1721122054 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907529577 0 0) 0x7fbf600ca920 con 0x7fbf601542e0 2023-11-28T18:28:03.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf6611d640 1 --2- 172.21.15.7:0/1721122054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf600cc810 0x7fbf60072450 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009d90 tx=0x7fbf5801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b245996d25b3cad server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf6491a640 1 -- 172.21.15.7:0/1721122054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580a5920 con 0x7fbf600cc810 2023-11-28T18:28:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf6491a640 1 -- 172.21.15.7:0/1721122054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf580a5a80 con 0x7fbf600cc810 2023-11-28T18:28:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf6491a640 1 -- 172.21.15.7:0/1721122054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580a5e10 con 0x7fbf600cc810 2023-11-28T18:28:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf67b80640 1 -- 172.21.15.7:0/1721122054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf600cc810 msgr2=0x7fbf60072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf67b80640 1 --2- 172.21.15.7:0/1721122054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf600cc810 0x7fbf60072450 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009d90 tx=0x7fbf5801c950 comp rx=0 tx=0).stop 2023-11-28T18:28:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf67b80640 1 -- 172.21.15.7:0/1721122054 shutdown_connections 2023-11-28T18:28:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf67b80640 1 --2- 172.21.15.7:0/1721122054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf600cc810 0x7fbf60072450 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf67b80640 1 --2- 172.21.15.7:0/1721122054 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cbb70 0x7fbf600cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf67b80640 1 -- 172.21.15.7:0/1721122054 >> 172.21.15.7:0/1721122054 conn(0x7fbf600717e0 msgr2=0x7fbf600662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf67b80640 1 -- 172.21.15.7:0/1721122054 shutdown_connections 2023-11-28T18:28:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.376+0000 7fbf67b80640 1 -- 172.21.15.7:0/1721122054 wait complete. 2023-11-28T18:28:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf67b80640 1 Processor -- start 2023-11-28T18:28:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf67b80640 1 -- start start 2023-11-28T18:28:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf67b80640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600cbb70 0x7fbf601500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf67b80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cc810 0x7fbf601f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf67b80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf601542e0 0x7fbf60158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf67b80640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf601f6630 con 0x7fbf601542e0 2023-11-28T18:28:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf67b80640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60157030 con 0x7fbf600cbb70 2023-11-28T18:28:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf67b80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60157310 con 0x7fbf600cc810 2023-11-28T18:28:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf601542e0 0x7fbf60158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf601542e0 0x7fbf60158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51792/0 (socket says 172.21.15.7:51792) 2023-11-28T18:28:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6591c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600cbb70 0x7fbf601500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 -- 172.21.15.7:0/3947050973 learned_addr learned my addr 172.21.15.7:0/3947050973 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 -- 172.21.15.7:0/3947050973 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cc810 msgr2=0x7fbf601f6a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T18:28:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cc810 0x7fbf601f6a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 -- 172.21.15.7:0/3947050973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600cbb70 msgr2=0x7fbf601500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600cbb70 0x7fbf601500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 -- 172.21.15.7:0/3947050973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf58009a70 con 0x7fbf601542e0 2023-11-28T18:28:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.380+0000 7fbf6611d640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf601542e0 0x7fbf60158460 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbf580b0040 tx=0x7fbf580196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf567fc640 1 -- 172.21.15.7:0/3947050973 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580ad850 con 0x7fbf601542e0 2023-11-28T18:28:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf567fc640 1 -- 172.21.15.7:0/3947050973 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf580ad9b0 con 0x7fbf601542e0 2023-11-28T18:28:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf567fc640 1 -- 172.21.15.7:0/3947050973 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580add60 con 0x7fbf601542e0 2023-11-28T18:28:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf601575f0 con 0x7fbf601542e0 2023-11-28T18:28:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf601f6250 con 0x7fbf601542e0 2023-11-28T18:28:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6006c670 con 0x7fbf601542e0 2023-11-28T18:28:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf567fc640 1 -- 172.21.15.7:0/3947050973 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbf580b4050 con 0x7fbf601542e0 2023-11-28T18:28:03.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf567fc640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf440668d0 0x7fbf44068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:03.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf567fc640 1 -- 172.21.15.7:0/3947050973 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fbf5811f300 con 0x7fbf601542e0 2023-11-28T18:28:03.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.384+0000 7fbf6591c640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf440668d0 0x7fbf44068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:03.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.388+0000 7fbf6591c640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf440668d0 0x7fbf44068d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c000f70 tx=0x7fbf5c007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:03.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.392+0000 7fbf567fc640 1 -- 172.21.15.7:0/3947050973 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbf580ec510 con 0x7fbf601542e0 2023-11-28T18:28:03.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.608+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf601f6770 con 0x7fbf601542e0 2023-11-28T18:28:03.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.608+0000 7fbf567fc640 1 -- 172.21.15.7:0/3947050973 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6557 (secure 0 0 0) 0x7fbf601f6770 con 0x7fbf601542e0 2023-11-28T18:28:03.614 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:03.614 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:27:18.707264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:03.615 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-11-28T18:28:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf440668d0 msgr2=0x7fbf44068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf440668d0 0x7fbf44068d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c000f70 tx=0x7fbf5c007df0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf601542e0 msgr2=0x7fbf60158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf601542e0 0x7fbf60158460 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbf580b0040 tx=0x7fbf580196f0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 shutdown_connections 2023-11-28T18:28:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbf440668d0 0x7fbf44068d90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf601542e0 0x7fbf60158460 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbf600cc810 0x7fbf601f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 --2- 172.21.15.7:0/3947050973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf600cbb70 0x7fbf601500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 >> 172.21.15.7:0/3947050973 conn(0x7fbf600717e0 msgr2=0x7fbf60157c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 shutdown_connections 2023-11-28T18:28:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:03.612+0000 7fbf67b80640 1 -- 172.21.15.7:0/3947050973 wait complete. 2023-11-28T18:28:03.636 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:28:05.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.784+0000 7f6fbdb03640 1 Processor -- start 2023-11-28T18:28:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.784+0000 7f6fbdb03640 1 -- start start 2023-11-28T18:28:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fbdb03640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ca30 0x7f6fb8159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fbdb03640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6fb815ce10 0x7f6fb80ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fbdb03640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb815a620 con 0x7f6fb80cafe0 2023-11-28T18:28:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fbdb03640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb815a900 con 0x7f6fb815ce10 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fbdb03640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb815abe0 con 0x7f6fb815ca30 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fb7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6fb815ce10 0x7f6fb80ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fb7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6fb815ce10 0x7f6fb80ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40600/0 (socket says 172.21.15.7:40600) 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fb7fff640 1 -- 172.21.15.7:0/1940952649 learned_addr learned my addr 172.21.15.7:0/1940952649 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fbcb01640 1 --2- 172.21.15.7:0/1940952649 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ca30 0x7f6fb8159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fb7fff640 1 -- 172.21.15.7:0/1940952649 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ca30 msgr2=0x7f6fb8159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fb7fff640 1 --2- 172.21.15.7:0/1940952649 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ca30 0x7f6fb8159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fb7fff640 1 -- 172.21.15.7:0/1940952649 >> v1:172.21.15.193:6789/0 conn(0x7f6fb80cafe0 legacy=0x7f6fb80cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.788+0000 7f6fb7fff640 1 -- 172.21.15.7:0/1940952649 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80cbb10 con 0x7f6fb815ce10 2023-11-28T18:28:05.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fb7fff640 1 --2- 172.21.15.7:0/1940952649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb815ce10 0x7f6fb80ca1d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009870 tx=0x7f6fa401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b4fbc1eb2764e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:05.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fb77fe640 1 -- 172.21.15.7:0/1940952649 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa40a8a30 con 0x7f6fb815ce10 2023-11-28T18:28:05.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fb77fe640 1 -- 172.21.15.7:0/1940952649 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa40a8b90 con 0x7f6fb815ce10 2023-11-28T18:28:05.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fb77fe640 1 -- 172.21.15.7:0/1940952649 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa40b0440 con 0x7f6fb815ce10 2023-11-28T18:28:05.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 -- 172.21.15.7:0/1940952649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb815ce10 msgr2=0x7f6fb80ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/1940952649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb815ce10 0x7f6fb80ca1d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009870 tx=0x7f6fa401ed30 comp rx=0 tx=0).stop 2023-11-28T18:28:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 -- 172.21.15.7:0/1940952649 shutdown_connections 2023-11-28T18:28:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/1940952649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb815ce10 0x7f6fb80ca1d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/1940952649 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ca30 0x7f6fb8159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 -- 172.21.15.7:0/1940952649 >> 172.21.15.7:0/1940952649 conn(0x7f6fb806c060 msgr2=0x7f6fb80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 -- 172.21.15.7:0/1940952649 shutdown_connections 2023-11-28T18:28:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 -- 172.21.15.7:0/1940952649 wait complete. 2023-11-28T18:28:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 Processor -- start 2023-11-28T18:28:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.792+0000 7f6fbdb03640 1 -- start start 2023-11-28T18:28:05.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb80cafe0 0x7f6fb8154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:05.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fb815ca30 0x7f6fb81553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:05.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ce10 0x7f6fb814e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:05.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb81e48e0 con 0x7f6fb80cafe0 2023-11-28T18:28:05.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80c87a0 con 0x7f6fb815ca30 2023-11-28T18:28:05.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb81e4ad0 con 0x7f6fb815ce10 2023-11-28T18:28:05.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb80cafe0 0x7f6fb8154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb80cafe0 0x7f6fb8154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40608/0 (socket says 172.21.15.7:40608) 2023-11-28T18:28:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 -- 172.21.15.7:0/2534948765 learned_addr learned my addr 172.21.15.7:0/2534948765 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbd302640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ce10 0x7f6fb814e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fb7fff640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fb815ca30 0x7f6fb81553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 -- 172.21.15.7:0/2534948765 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ce10 msgr2=0x7f6fb814e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ce10 0x7f6fb814e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 -- 172.21.15.7:0/2534948765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fb815ca30 msgr2=0x7f6fb81553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fb815ca30 0x7f6fb81553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 -- 172.21.15.7:0/2534948765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40094d0 con 0x7f6fb80cafe0 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbcb01640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb80cafe0 0x7f6fb8154ea0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00e930 tx=0x7f6fac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fb57fa640 1 -- 172.21.15.7:0/2534948765 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac009690 con 0x7f6fb80cafe0 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fb57fa640 1 -- 172.21.15.7:0/2534948765 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac00ca00 con 0x7f6fb80cafe0 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fb57fa640 1 -- 172.21.15.7:0/2534948765 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac00cd70 con 0x7f6fb80cafe0 2023-11-28T18:28:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb81e4db0 con 0x7f6fb80cafe0 2023-11-28T18:28:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb81e5370 con 0x7f6fb80cafe0 2023-11-28T18:28:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.796+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb804e890 con 0x7f6fb80cafe0 2023-11-28T18:28:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.800+0000 7f6fb57fa640 1 -- 172.21.15.7:0/2534948765 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6fac013470 con 0x7f6fb80cafe0 2023-11-28T18:28:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.800+0000 7f6fb57fa640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f9c066900 0x7f6f9c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.800+0000 7f6fb57fa640 1 -- 172.21.15.7:0/2534948765 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f6fac0846b0 con 0x7f6fb80cafe0 2023-11-28T18:28:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.800+0000 7f6fb7fff640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f9c066900 0x7f6f9c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:05.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.800+0000 7f6fb7fff640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f9c066900 0x7f6f9c068dc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40b2000 tx=0x7f6fa4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:05.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:05.804+0000 7f6fb57fa640 1 -- 172.21.15.7:0/2534948765 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6fac051980 con 0x7f6fb80cafe0 2023-11-28T18:28:06.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.020+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fb80cad10 con 0x7f6fb80cafe0 2023-11-28T18:28:06.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.020+0000 7f6fb57fa640 1 -- 172.21.15.7:0/2534948765 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6557 (secure 0 0 0) 0x7f6fac054fa0 con 0x7f6fb80cafe0 2023-11-28T18:28:06.026 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:06.026 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:27:18.707264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:06.026 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-11-28T18:28:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f9c066900 msgr2=0x7f6f9c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f9c066900 0x7f6f9c068dc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40b2000 tx=0x7f6fa4009040 comp rx=0 tx=0).stop 2023-11-28T18:28:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb80cafe0 msgr2=0x7f6fb8154ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb80cafe0 0x7f6fb8154ea0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00e930 tx=0x7f6fac00edf0 comp rx=0 tx=0).stop 2023-11-28T18:28:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 shutdown_connections 2023-11-28T18:28:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f9c066900 0x7f6f9c068dc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6fb815ce10 0x7f6fb814e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fb815ca30 0x7f6fb81553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 --2- 172.21.15.7:0/2534948765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb80cafe0 0x7f6fb8154ea0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 >> 172.21.15.7:0/2534948765 conn(0x7f6fb806c060 msgr2=0x7f6fb815b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 shutdown_connections 2023-11-28T18:28:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:06.024+0000 7f6fbdb03640 1 -- 172.21.15.7:0/2534948765 wait complete. 2023-11-28T18:28:06.048 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:28:08.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f54a5ff2640 1 Processor -- start 2023-11-28T18:28:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f54a5ff2640 1 -- start start 2023-11-28T18:28:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f54a5ff2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54a015ce10 0x7f54a00ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f54a5ff2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a015a3a0 con 0x7f54a015ca30 2023-11-28T18:28:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f54a5ff2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a015a620 con 0x7f54a00cafe0 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f54a5ff2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a015a900 con 0x7f54a015ce10 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f549ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54a015ce10 0x7f54a00ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f549ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54a015ce10 0x7f54a00ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40610/0 (socket says 172.21.15.7:40610) 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f54a4ff0640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f54a015ca30 0x7f54a0159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46100/0 (socket says 172.21.15.7:46100) 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f549ffff640 1 -- 172.21.15.7:0/2783811181 learned_addr learned my addr 172.21.15.7:0/2783811181 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f549ffff640 1 -- 172.21.15.7:0/2783811181 >> v1:172.21.15.193:6789/0 conn(0x7f54a00cafe0 legacy=0x7f54a00cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f549ffff640 1 -- 172.21.15.7:0/2783811181 >> v1:172.21.15.7:6789/0 conn(0x7f54a015ca30 legacy=0x7f54a0159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f549ffff640 1 -- 172.21.15.7:0/2783811181 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a015abe0 con 0x7f54a015ce10 2023-11-28T18:28:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.212+0000 7f549f7fe640 1 -- 172.21.15.7:0/2783811181 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557263346 0 0) 0x7f54a015a3a0 con 0x7f54a015ca30 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f549ffff640 1 --2- 172.21.15.7:0/2783811181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a015ce10 0x7f54a00ca1d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f548c009870 tx=0x7f548c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b130ef5f42c8f4f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f549f7fe640 1 -- 172.21.15.7:0/2783811181 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c0ab620 con 0x7f54a015ce10 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f549f7fe640 1 -- 172.21.15.7:0/2783811181 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c0ab780 con 0x7f54a015ce10 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f549f7fe640 1 -- 172.21.15.7:0/2783811181 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c0abb10 con 0x7f54a015ce10 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2783811181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a015ce10 msgr2=0x7f54a00ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f54a5ff2640 1 --2- 172.21.15.7:0/2783811181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a015ce10 0x7f54a00ca1d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f548c009870 tx=0x7f548c01c510 comp rx=0 tx=0).stop 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2783811181 shutdown_connections 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f54a5ff2640 1 --2- 172.21.15.7:0/2783811181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a015ce10 0x7f54a00ca1d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2783811181 >> 172.21.15.7:0/2783811181 conn(0x7f54a006c060 msgr2=0x7f54a00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2783811181 shutdown_connections 2023-11-28T18:28:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.216+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2783811181 wait complete. 2023-11-28T18:28:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 Processor -- start 2023-11-28T18:28:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 -- start start 2023-11-28T18:28:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a00cafe0 0x7f54a00c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a015ca30 0x7f54a0157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f54a015ce10 0x7f54a01f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a01521f0 con 0x7f54a00cafe0 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0157650 con 0x7f54a015ca30 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0152fb0 con 0x7f54a015ce10 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a00cafe0 0x7f54a00c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a00cafe0 0x7f54a00c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40626/0 (socket says 172.21.15.7:40626) 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 -- 172.21.15.7:0/2180900917 learned_addr learned my addr 172.21.15.7:0/2180900917 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f549ffff640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a015ca30 0x7f54a0157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a57f1640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f54a015ce10 0x7f54a01f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 -- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f54a015ce10 msgr2=0x7f54a01f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f54a015ce10 0x7f54a01f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 -- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a015ca30 msgr2=0x7f54a0157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a015ca30 0x7f54a0157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 -- 172.21.15.7:0/2180900917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c0094d0 con 0x7f54a00cafe0 2023-11-28T18:28:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f549ffff640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a015ca30 0x7f54a0157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:28:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a4ff0640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a00cafe0 0x7f54a00c5c70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f549400c500 tx=0x7f549400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f549d7fa640 1 -- 172.21.15.7:0/2180900917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940074e0 con 0x7f54a00cafe0 2023-11-28T18:28:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f549d7fa640 1 -- 172.21.15.7:0/2180900917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5494007640 con 0x7f54a00cafe0 2023-11-28T18:28:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a01523e0 con 0x7f54a00cafe0 2023-11-28T18:28:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f549d7fa640 1 -- 172.21.15.7:0/2180900917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5494004a30 con 0x7f54a00cafe0 2023-11-28T18:28:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.220+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a0152960 con 0x7f54a00cafe0 2023-11-28T18:28:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.224+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a004e890 con 0x7f54a00cafe0 2023-11-28T18:28:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.224+0000 7f549d7fa640 1 -- 172.21.15.7:0/2180900917 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5494004b90 con 0x7f54a00cafe0 2023-11-28T18:28:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.224+0000 7f549d7fa640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54840668d0 0x7f5484068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.224+0000 7f549d7fa640 1 -- 172.21.15.7:0/2180900917 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f54940837c0 con 0x7f54a00cafe0 2023-11-28T18:28:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.224+0000 7f549ffff640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54840668d0 0x7f5484068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.228+0000 7f549ffff640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54840668d0 0x7f5484068d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f548c0af000 tx=0x7f548c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.232+0000 7f549d7fa640 1 -- 172.21.15.7:0/2180900917 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5494050940 con 0x7f54a00cafe0 2023-11-28T18:28:08.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.492+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54a00c9f40 con 0x7f54a00cafe0 2023-11-28T18:28:08.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f549d7fa640 1 -- 172.21.15.7:0/2180900917 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6557 (secure 0 0 0) 0x7f5494018090 con 0x7f54a00cafe0 2023-11-28T18:28:08.499 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:08.499 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:27:18.707264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:08.499 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-11-28T18:28:08.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54840668d0 msgr2=0x7f5484068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:08.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54840668d0 0x7f5484068d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f548c0af000 tx=0x7f548c0a7000 comp rx=0 tx=0).stop 2023-11-28T18:28:08.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a00cafe0 msgr2=0x7f54a00c5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:08.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a00cafe0 0x7f54a00c5c70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f549400c500 tx=0x7f549400c9c0 comp rx=0 tx=0).stop 2023-11-28T18:28:08.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 shutdown_connections 2023-11-28T18:28:08.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54840668d0 0x7f5484068d90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:08.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f54a015ce10 0x7f54a01f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:08.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a015ca30 0x7f54a0157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:08.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 --2- 172.21.15.7:0/2180900917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54a00cafe0 0x7f54a00c5c70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:08.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 >> 172.21.15.7:0/2180900917 conn(0x7f54a006c060 msgr2=0x7f54a015c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:08.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 shutdown_connections 2023-11-28T18:28:08.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:08.496+0000 7f54a5ff2640 1 -- 172.21.15.7:0/2180900917 wait complete. 2023-11-28T18:28:08.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:28:10.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:10.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b9cc8640 1 Processor -- start 2023-11-28T18:28:10.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b9cc8640 1 -- start start 2023-11-28T18:28:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b9cc8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe3b40cafe0 0x7fe3b40cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b9cc8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b415a3a0 con 0x7fe3b415ca30 2023-11-28T18:28:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b9cc8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b415a620 con 0x7fe3b415ce10 2023-11-28T18:28:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b9cc8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b415a900 con 0x7fe3b40cafe0 2023-11-28T18:28:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b94c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe3b40cafe0 0x7fe3b40cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b94c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe3b40cafe0 0x7fe3b40cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40640/0 (socket says 172.21.15.7:40640) 2023-11-28T18:28:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b8cc6640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe3b415ca30 0x7fe3b4159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46104/0 (socket says 172.21.15.7:46104) 2023-11-28T18:28:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b94c7640 1 -- 172.21.15.7:0/2180909677 learned_addr learned my addr 172.21.15.7:0/2180909677 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:10.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b94c7640 1 -- 172.21.15.7:0/2180909677 >> v1:172.21.15.193:6789/0 conn(0x7fe3b415ce10 legacy=0x7fe3b40ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:10.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b94c7640 1 -- 172.21.15.7:0/2180909677 >> v1:172.21.15.7:6789/0 conn(0x7fe3b415ca30 legacy=0x7fe3b4159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:10.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.684+0000 7fe3b94c7640 1 -- 172.21.15.7:0/2180909677 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b415abe0 con 0x7fe3b40cafe0 2023-11-28T18:28:10.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b94c7640 1 --2- 172.21.15.7:0/2180909677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b40cc5f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac009e50 tx=0x7fe3ac01c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5014f533e17840 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:10.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b37fe640 1 -- 172.21.15.7:0/2180909677 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0a57c0 con 0x7fe3b40cafe0 2023-11-28T18:28:10.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b37fe640 1 -- 172.21.15.7:0/2180909677 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ac0a5920 con 0x7fe3b40cafe0 2023-11-28T18:28:10.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b37fe640 1 -- 172.21.15.7:0/2180909677 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0a5cb0 con 0x7fe3b40cafe0 2023-11-28T18:28:10.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/2180909677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 msgr2=0x7fe3b40cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:10.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b9cc8640 1 --2- 172.21.15.7:0/2180909677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b40cc5f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac009e50 tx=0x7fe3ac01c9e0 comp rx=0 tx=0).stop 2023-11-28T18:28:10.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/2180909677 shutdown_connections 2023-11-28T18:28:10.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b9cc8640 1 --2- 172.21.15.7:0/2180909677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b40cc5f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:10.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/2180909677 >> 172.21.15.7:0/2180909677 conn(0x7fe3b406c060 msgr2=0x7fe3b40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:10.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/2180909677 shutdown_connections 2023-11-28T18:28:10.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.688+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/2180909677 wait complete. 2023-11-28T18:28:10.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 Processor -- start 2023-11-28T18:28:10.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 -- start start 2023-11-28T18:28:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b4156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3b415ca30 0x7fe3b414c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b415ce10 0x7fe3b414c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b4155fd0 con 0x7fe3b40cafe0 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41576e0 con 0x7fe3b415ce10 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b4155520 con 0x7fe3b415ca30 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b4156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b4156540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40656/0 (socket says 172.21.15.7:40656) 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 -- 172.21.15.7:0/1934957684 learned_addr learned my addr 172.21.15.7:0/1934957684 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b94c7640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b415ce10 0x7fe3b414c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b3fff640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3b415ca30 0x7fe3b414c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 -- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3b415ca30 msgr2=0x7fe3b414c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3b415ca30 0x7fe3b414c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 -- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b415ce10 msgr2=0x7fe3b414c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:10.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b415ce10 0x7fe3b414c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 -- 172.21.15.7:0/1934957684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac009b30 con 0x7fe3b40cafe0 2023-11-28T18:28:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b94c7640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b415ce10 0x7fe3b414c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:28:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b8cc6640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b4156540 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800c490 tx=0x7fe3a800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b17fa640 1 -- 172.21.15.7:0/1934957684 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8007480 con 0x7fe3b40cafe0 2023-11-28T18:28:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b17fa640 1 -- 172.21.15.7:0/1934957684 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a80075e0 con 0x7fe3b40cafe0 2023-11-28T18:28:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b17fa640 1 -- 172.21.15.7:0/1934957684 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8004a10 con 0x7fe3b40cafe0 2023-11-28T18:28:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.692+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b4155730 con 0x7fe3b40cafe0 2023-11-28T18:28:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.696+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b4155cf0 con 0x7fe3b40cafe0 2023-11-28T18:28:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.696+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b404e890 con 0x7fe3b40cafe0 2023-11-28T18:28:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.696+0000 7fe3b17fa640 1 -- 172.21.15.7:0/1934957684 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe3a8004b70 con 0x7fe3b40cafe0 2023-11-28T18:28:10.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.696+0000 7fe3b17fa640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3980668d0 0x7fe398068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:10.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.696+0000 7fe3b17fa640 1 -- 172.21.15.7:0/1934957684 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fe3a8083760 con 0x7fe3b40cafe0 2023-11-28T18:28:10.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.696+0000 7fe3b3fff640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3980668d0 0x7fe398068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:10.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.700+0000 7fe3b3fff640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3980668d0 0x7fe398068d90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40d0460 tx=0x7fe3a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:10.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.700+0000 7fe3b17fa640 1 -- 172.21.15.7:0/1934957684 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe3a8050840 con 0x7fe3b40cafe0 2023-11-28T18:28:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.912+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3b40c9390 con 0x7fe3b40cafe0 2023-11-28T18:28:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.912+0000 7fe3b17fa640 1 -- 172.21.15.7:0/1934957684 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6557 (secure 0 0 0) 0x7fe3a8018090 con 0x7fe3b40cafe0 2023-11-28T18:28:10.918 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:10.918 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:27:18.707264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"f","rank":4,"incarnation":21,"state":"up:active","state_seq":48,"addr":"172.21.15.193:6835/4143265275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4143265275},{"type":"v1","addr":"172.21.15.193:6835","nonce":4143265275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:10.918 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-11-28T18:28:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3980668d0 msgr2=0x7fe398068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3980668d0 0x7fe398068d90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40d0460 tx=0x7fe3a0009040 comp rx=0 tx=0).stop 2023-11-28T18:28:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 msgr2=0x7fe3b4156540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b4156540 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800c490 tx=0x7fe3a800c950 comp rx=0 tx=0).stop 2023-11-28T18:28:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 shutdown_connections 2023-11-28T18:28:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3980668d0 0x7fe398068d90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b415ce10 0x7fe3b414c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3b415ca30 0x7fe3b414c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 --2- 172.21.15.7:0/1934957684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b40cafe0 0x7fe3b4156540 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 >> 172.21.15.7:0/1934957684 conn(0x7fe3b406c060 msgr2=0x7fe3b4157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 shutdown_connections 2023-11-28T18:28:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:10.916+0000 7fe3b9cc8640 1 -- 172.21.15.7:0/1934957684 wait complete. 2023-11-28T18:28:10.943 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:28:12.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:13.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2945348640 1 Processor -- start 2023-11-28T18:28:13.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2945348640 1 -- start start 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2945348640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2940153e10 0x7f294006f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2945348640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f294006ed80 0x7f29400c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2945348640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29400c9360 0x7f294015a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2945348640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2940159c30 con 0x7f2940153e10 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2945348640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294015d3c0 con 0x7f294006ed80 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2945348640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294015d6a0 con 0x7f29400c9360 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2940153e10 0x7f294006f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2940153e10 0x7f294006f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40664/0 (socket says 172.21.15.7:40664) 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293ffff640 1 -- 172.21.15.7:0/3697130237 learned_addr learned my addr 172.21.15.7:0/3697130237 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293f7fe640 1 --2- 172.21.15.7:0/3697130237 >> v2:172.21.15.193:3300/0 conn(0x7f294006ed80 0x7f29400c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2944b47640 1 --2- 172.21.15.7:0/3697130237 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29400c9360 0x7f294015a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293ffff640 1 -- 172.21.15.7:0/3697130237 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29400c9360 msgr2=0x7f294015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:13.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293ffff640 1 --2- 172.21.15.7:0/3697130237 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29400c9360 0x7f294015a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293ffff640 1 -- 172.21.15.7:0/3697130237 >> v2:172.21.15.193:3300/0 conn(0x7f294006ed80 msgr2=0x7f29400c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:13.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293ffff640 1 --2- 172.21.15.7:0/3697130237 >> v2:172.21.15.193:3300/0 conn(0x7f294006ed80 0x7f29400c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f293ffff640 1 -- 172.21.15.7:0/3697130237 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294015d980 con 0x7f2940153e10 2023-11-28T18:28:13.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.096+0000 7f2944b47640 1 --2- 172.21.15.7:0/3697130237 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29400c9360 0x7f294015a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:28:13.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f293ffff640 1 --2- 172.21.15.7:0/3697130237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940153e10 0x7f294006f5d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f29400c7510 tx=0x7f293801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2921f6ed452d54ff server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:13.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f293effd640 1 -- 172.21.15.7:0/3697130237 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380af400 con 0x7f2940153e10 2023-11-28T18:28:13.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f293effd640 1 -- 172.21.15.7:0/3697130237 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29380ab050 con 0x7f2940153e10 2023-11-28T18:28:13.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f293effd640 1 -- 172.21.15.7:0/3697130237 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380af780 con 0x7f2940153e10 2023-11-28T18:28:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 -- 172.21.15.7:0/3697130237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940153e10 msgr2=0x7f294006f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 --2- 172.21.15.7:0/3697130237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940153e10 0x7f294006f5d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f29400c7510 tx=0x7f293801f1c0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 -- 172.21.15.7:0/3697130237 shutdown_connections 2023-11-28T18:28:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 --2- 172.21.15.7:0/3697130237 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29400c9360 0x7f294015a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 --2- 172.21.15.7:0/3697130237 >> v2:172.21.15.193:3300/0 conn(0x7f294006ed80 0x7f29400c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 --2- 172.21.15.7:0/3697130237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940153e10 0x7f294006f5d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 -- 172.21.15.7:0/3697130237 >> 172.21.15.7:0/3697130237 conn(0x7f294006a5f0 msgr2=0x7f294006aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 -- 172.21.15.7:0/3697130237 shutdown_connections 2023-11-28T18:28:13.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.100+0000 7f2945348640 1 -- 172.21.15.7:0/3697130237 wait complete. 2023-11-28T18:28:13.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 Processor -- start 2023-11-28T18:28:13.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 -- start start 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f294006ed80 0x7f29401f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c9360 0x7f294014cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2940153e10 0x7f2940066540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294014b350 con 0x7f29400c9360 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294014ab40 con 0x7f2940153e10 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2940152bd0 con 0x7f294006ed80 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c9360 0x7f294014cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c9360 0x7f294014cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40672/0 (socket says 172.21.15.7:40672) 2023-11-28T18:28:13.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 -- 172.21.15.7:0/3942134447 learned_addr learned my addr 172.21.15.7:0/3942134447 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2944b47640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2940153e10 0x7f2940066540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293ffff640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f294006ed80 0x7f29401f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 -- 172.21.15.7:0/3942134447 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f294006ed80 msgr2=0x7f29401f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f294006ed80 0x7f29401f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 -- 172.21.15.7:0/3942134447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2940153e10 msgr2=0x7f2940066540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2940153e10 0x7f2940066540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 -- 172.21.15.7:0/3942134447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2938009b30 con 0x7f29400c9360 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293f7fe640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c9360 0x7f294014cf30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f293400e870 tx=0x7f293400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293cff9640 1 -- 172.21.15.7:0/3942134447 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934009510 con 0x7f29400c9360 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293cff9640 1 -- 172.21.15.7:0/3942134447 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293400eec0 con 0x7f29400c9360 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f293cff9640 1 -- 172.21.15.7:0/3942134447 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293400cae0 con 0x7f29400c9360 2023-11-28T18:28:13.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294014ad90 con 0x7f29400c9360 2023-11-28T18:28:13.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.104+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294014ca10 con 0x7f29400c9360 2023-11-28T18:28:13.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.108+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294014c850 con 0x7f29400c9360 2023-11-28T18:28:13.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.108+0000 7f293cff9640 1 -- 172.21.15.7:0/3942134447 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f293400cc40 con 0x7f29400c9360 2023-11-28T18:28:13.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.108+0000 7f293cff9640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f292c0668d0 0x7f292c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.108+0000 7f293cff9640 1 -- 172.21.15.7:0/3942134447 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f2934084a10 con 0x7f29400c9360 2023-11-28T18:28:13.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.108+0000 7f293ffff640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f292c0668d0 0x7f292c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.108+0000 7f293ffff640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f292c0668d0 0x7f292c068d90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f29380b1000 tx=0x7f29380aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:13.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.112+0000 7f293cff9640 1 -- 172.21.15.7:0/3942134447 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2934051ab0 con 0x7f29400c9360 2023-11-28T18:28:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.392+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f294004e890 con 0x7f29400c9360 2023-11-28T18:28:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.392+0000 7f293cff9640 1 -- 172.21.15.7:0/3942134447 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5763 (secure 0 0 0) 0x7f2934021020 con 0x7f29400c9360 2023-11-28T18:28:13.396 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:13.397 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:12.173866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-11-28T18:28:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f292c0668d0 msgr2=0x7f292c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f292c0668d0 0x7f292c068d90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f29380b1000 tx=0x7f29380aa000 comp rx=0 tx=0).stop 2023-11-28T18:28:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c9360 msgr2=0x7f294014cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c9360 0x7f294014cf30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f293400e870 tx=0x7f293400ec70 comp rx=0 tx=0).stop 2023-11-28T18:28:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 shutdown_connections 2023-11-28T18:28:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f292c0668d0 0x7f292c068d90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2940153e10 0x7f2940066540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c9360 0x7f294014cf30 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 --2- 172.21.15.7:0/3942134447 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f294006ed80 0x7f29401f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 >> 172.21.15.7:0/3942134447 conn(0x7f294006a5f0 msgr2=0x7f29400c8470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 shutdown_connections 2023-11-28T18:28:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.396+0000 7f2945348640 1 -- 172.21.15.7:0/3942134447 wait complete. 2023-11-28T18:28:13.422 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-11-28T18:28:13.422 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:28:13.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:13.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.564+0000 7f0965df8640 1 Processor -- start 2023-11-28T18:28:13.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.564+0000 7f0965df8640 1 -- start start 2023-11-28T18:28:13.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.564+0000 7f0965df8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f096015ca30 0x7f0960159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.564+0000 7f0965df8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f09600cafe0 0x7f09600cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.564+0000 7f0965df8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096015a620 con 0x7f096015ce10 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.564+0000 7f0965df8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f096015a900 con 0x7f09600cafe0 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.564+0000 7f0965df8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f096015abe0 con 0x7f096015ca30 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f09655f7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f09600cafe0 0x7f09600cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f095ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f096015ce10 0x7f09600ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45808/0 (socket says 172.21.15.7:45808) 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f095ffff640 1 -- 172.21.15.7:0/1778674311 learned_addr learned my addr 172.21.15.7:0/1778674311 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f0964df6640 1 --2- 172.21.15.7:0/1778674311 >> v2:172.21.15.193:3300/0 conn(0x7f096015ca30 0x7f0960159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f09655f7640 1 -- 172.21.15.7:0/1778674311 >> v2:172.21.15.193:3300/0 conn(0x7f096015ca30 msgr2=0x7f0960159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f09655f7640 1 --2- 172.21.15.7:0/1778674311 >> v2:172.21.15.193:3300/0 conn(0x7f096015ca30 0x7f0960159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f09655f7640 1 -- 172.21.15.7:0/1778674311 >> v1:172.21.15.7:6789/0 conn(0x7f096015ce10 legacy=0x7f09600ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f09655f7640 1 -- 172.21.15.7:0/1778674311 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09600cbb10 con 0x7f09600cafe0 2023-11-28T18:28:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.568+0000 7f09655f7640 1 --2- 172.21.15.7:0/1778674311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09600cc5f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f096006c660 tx=0x7f095801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a0e1ca2c0c5898 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:13.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f095f7fe640 1 -- 172.21.15.7:0/1778674311 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580a58b0 con 0x7f09600cafe0 2023-11-28T18:28:13.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f095f7fe640 1 -- 172.21.15.7:0/1778674311 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09580a5a10 con 0x7f09600cafe0 2023-11-28T18:28:13.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f095f7fe640 1 -- 172.21.15.7:0/1778674311 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580a5da0 con 0x7f09600cafe0 2023-11-28T18:28:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 -- 172.21.15.7:0/1778674311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 msgr2=0x7f09600cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 --2- 172.21.15.7:0/1778674311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09600cc5f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f096006c660 tx=0x7f095801c8e0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 -- 172.21.15.7:0/1778674311 shutdown_connections 2023-11-28T18:28:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 --2- 172.21.15.7:0/1778674311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09600cc5f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 --2- 172.21.15.7:0/1778674311 >> v2:172.21.15.193:3300/0 conn(0x7f096015ca30 0x7f0960159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 -- 172.21.15.7:0/1778674311 >> 172.21.15.7:0/1778674311 conn(0x7f096006c060 msgr2=0x7f09600662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 -- 172.21.15.7:0/1778674311 shutdown_connections 2023-11-28T18:28:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 -- 172.21.15.7:0/1778674311 wait complete. 2023-11-28T18:28:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 Processor -- start 2023-11-28T18:28:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 -- start start 2023-11-28T18:28:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09601f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f096015ca30 0x7f09601564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.572+0000 7f0965df8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096015ce10 0x7f0960156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0965df8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09600c2f30 con 0x7f09600cafe0 2023-11-28T18:28:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0965df8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0960156060 con 0x7f096015ce10 2023-11-28T18:28:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0965df8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09601f6b00 con 0x7f096015ca30 2023-11-28T18:28:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09601f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09601f6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47466/0 (socket says 172.21.15.7:47466) 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 -- 172.21.15.7:0/616608079 learned_addr learned my addr 172.21.15.7:0/616608079 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f095ffff640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f096015ca30 0x7f09601564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f09655f7640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096015ce10 0x7f0960156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 -- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f096015ca30 msgr2=0x7f09601564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f096015ca30 0x7f09601564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 -- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096015ce10 msgr2=0x7f0960156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096015ce10 0x7f0960156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 -- 172.21.15.7:0/616608079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958009a70 con 0x7f09600cafe0 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f09655f7640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096015ce10 0x7f0960156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0964df6640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09601f6230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f095400d5f0 tx=0x7f095400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f095d7fa640 1 -- 172.21.15.7:0/616608079 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540125f0 con 0x7f09600cafe0 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f095d7fa640 1 -- 172.21.15.7:0/616608079 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0954012750 con 0x7f09600cafe0 2023-11-28T18:28:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f095d7fa640 1 -- 172.21.15.7:0/616608079 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954012ae0 con 0x7f09600cafe0 2023-11-28T18:28:13.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09601f6dc0 con 0x7f09600cafe0 2023-11-28T18:28:13.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09600c3440 con 0x7f09600cafe0 2023-11-28T18:28:13.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096004e890 con 0x7f09600cafe0 2023-11-28T18:28:13.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.576+0000 7f095d7fa640 1 -- 172.21.15.7:0/616608079 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0954012c40 con 0x7f09600cafe0 2023-11-28T18:28:13.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.580+0000 7f095d7fa640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0944066900 0x7f0944068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:13.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.580+0000 7f095d7fa640 1 -- 172.21.15.7:0/616608079 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f0954084af0 con 0x7f09600cafe0 2023-11-28T18:28:13.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.580+0000 7f095ffff640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0944066900 0x7f0944068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:13.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.580+0000 7f095ffff640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0944066900 0x7f0944068dc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0960066550 tx=0x7f094c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:13.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.584+0000 7f095d7fa640 1 -- 172.21.15.7:0/616608079 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0954051ae0 con 0x7f09600cafe0 2023-11-28T18:28:13.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.796+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09600cad10 con 0x7f09600cafe0 2023-11-28T18:28:13.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.796+0000 7f095d7fa640 1 -- 172.21.15.7:0/616608079 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5763 (secure 0 0 0) 0x7f0954007380 con 0x7f09600cafe0 2023-11-28T18:28:13.802 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:13.802 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:12.173866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:13.802 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-11-28T18:28:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0944066900 msgr2=0x7f0944068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0944066900 0x7f0944068dc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0960066550 tx=0x7f094c009040 comp rx=0 tx=0).stop 2023-11-28T18:28:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 msgr2=0x7f09601f6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09601f6230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f095400d5f0 tx=0x7f095400dab0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 shutdown_connections 2023-11-28T18:28:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0944066900 0x7f0944068dc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096015ce10 0x7f0960156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f096015ca30 0x7f09601564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 --2- 172.21.15.7:0/616608079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09600cafe0 0x7f09601f6230 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 >> 172.21.15.7:0/616608079 conn(0x7f096006c060 msgr2=0x7f096015bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 shutdown_connections 2023-11-28T18:28:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:13.800+0000 7f0965df8640 1 -- 172.21.15.7:0/616608079 wait complete. 2023-11-28T18:28:13.824 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4284": { "addr": "172.21.15.7:6835/1419029356", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1419029356, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1419029356, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4284, "incarnation": 13, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4285": { "addr": "172.21.15.193:6837/3911527535", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 3911527535, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 3911527535, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 11, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.7:6838/2690556675", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 2690556675, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 2690556675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:28:12.173866+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 4288, "mds_2": 4285, "mds_3": 4284, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:28:15.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:15.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd8393fd640 1 Processor -- start 2023-11-28T18:28:15.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd8393fd640 1 -- start start 2023-11-28T18:28:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd8393fd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fd8340cafe0 0x7fd8340cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd8393fd640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83415a3a0 con 0x7fd83415ce10 2023-11-28T18:28:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd8393fd640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83415a620 con 0x7fd83415ca30 2023-11-28T18:28:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd8393fd640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83415a900 con 0x7fd8340cafe0 2023-11-28T18:28:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd8337fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd83415ce10 0x7fd8340ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45810/0 (socket says 172.21.15.7:45810) 2023-11-28T18:28:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd8337fe640 1 -- 172.21.15.7:0/3048712473 learned_addr learned my addr 172.21.15.7:0/3048712473 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:28:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd838bfc640 1 --2- 172.21.15.7:0/3048712473 >> v2:172.21.15.193:3300/0 conn(0x7fd8340cafe0 0x7fd8340cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd832ffd640 1 -- 172.21.15.7:0/3048712473 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230347515 0 0) 0x7fd83415a3a0 con 0x7fd83415ce10 2023-11-28T18:28:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd832ffd640 1 -- 172.21.15.7:0/3048712473 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd81c003530 con 0x7fd83415ce10 2023-11-28T18:28:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd832ffd640 1 -- 172.21.15.7:0/3048712473 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814009038 0 0) 0x7fd83415a620 con 0x7fd83415ca30 2023-11-28T18:28:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd832ffd640 1 -- 172.21.15.7:0/3048712473 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd83415a3a0 con 0x7fd83415ca30 2023-11-28T18:28:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd838bfc640 1 -- 172.21.15.7:0/3048712473 >> v1:172.21.15.193:6789/0 conn(0x7fd83415ca30 legacy=0x7fd834159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd838bfc640 1 -- 172.21.15.7:0/3048712473 >> v1:172.21.15.7:6789/0 conn(0x7fd83415ce10 legacy=0x7fd8340ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.988+0000 7fd838bfc640 1 -- 172.21.15.7:0/3048712473 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83415abe0 con 0x7fd8340cafe0 2023-11-28T18:28:15.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd838bfc640 1 --2- 172.21.15.7:0/3048712473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8340cafe0 0x7fd8340cc5f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009d90 tx=0x7fd82c01c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beb0cae996f3366f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:15.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd832ffd640 1 -- 172.21.15.7:0/3048712473 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0a58f0 con 0x7fd8340cafe0 2023-11-28T18:28:15.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd832ffd640 1 -- 172.21.15.7:0/3048712473 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c0a5a50 con 0x7fd8340cafe0 2023-11-28T18:28:15.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd832ffd640 1 -- 172.21.15.7:0/3048712473 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0a5de0 con 0x7fd8340cafe0 2023-11-28T18:28:15.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd8393fd640 1 -- 172.21.15.7:0/3048712473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8340cafe0 msgr2=0x7fd8340cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:15.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd8393fd640 1 --2- 172.21.15.7:0/3048712473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8340cafe0 0x7fd8340cc5f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009d90 tx=0x7fd82c01c920 comp rx=0 tx=0).stop 2023-11-28T18:28:15.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd8393fd640 1 -- 172.21.15.7:0/3048712473 shutdown_connections 2023-11-28T18:28:15.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd8393fd640 1 --2- 172.21.15.7:0/3048712473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8340cafe0 0x7fd8340cc5f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:15.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd8393fd640 1 -- 172.21.15.7:0/3048712473 >> 172.21.15.7:0/3048712473 conn(0x7fd83406c060 msgr2=0x7fd8340662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:15.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd8393fd640 1 -- 172.21.15.7:0/3048712473 shutdown_connections 2023-11-28T18:28:15.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.992+0000 7fd8393fd640 1 -- 172.21.15.7:0/3048712473 wait complete. 2023-11-28T18:28:15.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8393fd640 1 Processor -- start 2023-11-28T18:28:16.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8393fd640 1 -- start start 2023-11-28T18:28:16.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8393fd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8340cafe0 0x7fd8341f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:16.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8393fd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd83415ca30 0x7fd8340c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:16.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8393fd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd83415ce10 0x7fd8340c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:16.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8393fd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8341f6db0 con 0x7fd8340cafe0 2023-11-28T18:28:16.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8393fd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340c3000 con 0x7fd83415ce10 2023-11-28T18:28:16.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8393fd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340c8990 con 0x7fd83415ca30 2023-11-28T18:28:16.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd833fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8340cafe0 0x7fd8341f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:16.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd833fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8340cafe0 0x7fd8341f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47468/0 (socket says 172.21.15.7:47468) 2023-11-28T18:28:16.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd833fff640 1 -- 172.21.15.7:0/4000132885 learned_addr learned my addr 172.21.15.7:0/4000132885 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:16.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd838bfc640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd83415ce10 0x7fd8340c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:16.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd8337fe640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd83415ca30 0x7fd8340c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:16.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd833fff640 1 -- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd83415ca30 msgr2=0x7fd8340c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:16.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd833fff640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd83415ca30 0x7fd8340c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd833fff640 1 -- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd83415ce10 msgr2=0x7fd8340c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd833fff640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd83415ce10 0x7fd8340c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd833fff640 1 -- 172.21.15.7:0/4000132885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c009a70 con 0x7fd8340cafe0 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:15.996+0000 7fd838bfc640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd83415ce10 0x7fd8340c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd833fff640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8340cafe0 0x7fd8341f6870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c490 tx=0x7fd82400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd830ff9640 1 -- 172.21.15.7:0/4000132885 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240052d0 con 0x7fd8340cafe0 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd830ff9640 1 -- 172.21.15.7:0/4000132885 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd824005430 con 0x7fd8340cafe0 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8340c32a0 con 0x7fd8340cafe0 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd830ff9640 1 -- 172.21.15.7:0/4000132885 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824012870 con 0x7fd8340cafe0 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8340c3860 con 0x7fd8340cafe0 2023-11-28T18:28:16.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8340c3a80 con 0x7fd8340cafe0 2023-11-28T18:28:16.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd830ff9640 1 -- 172.21.15.7:0/4000132885 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd8240129d0 con 0x7fd8340cafe0 2023-11-28T18:28:16.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd830ff9640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8200668d0 0x7fd820068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:16.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd830ff9640 1 -- 172.21.15.7:0/4000132885 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fd824084280 con 0x7fd8340cafe0 2023-11-28T18:28:16.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.000+0000 7fd8337fe640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8200668d0 0x7fd820068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:16.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.004+0000 7fd8337fe640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8200668d0 0x7fd820068d90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd8340c9320 tx=0x7fd828009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:16.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.004+0000 7fd830ff9640 1 -- 172.21.15.7:0/4000132885 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd824051270 con 0x7fd8340cafe0 2023-11-28T18:28:16.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.208+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd83404e890 con 0x7fd8340cafe0 2023-11-28T18:28:16.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.208+0000 7fd830ff9640 1 -- 172.21.15.7:0/4000132885 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5763 (secure 0 0 0) 0x7fd824054890 con 0x7fd8340cafe0 2023-11-28T18:28:16.215 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:16.215 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:12.173866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-11-28T18:28:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8200668d0 msgr2=0x7fd820068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8200668d0 0x7fd820068d90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd8340c9320 tx=0x7fd828009040 comp rx=0 tx=0).stop 2023-11-28T18:28:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8340cafe0 msgr2=0x7fd8341f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8340cafe0 0x7fd8341f6870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c490 tx=0x7fd82400c950 comp rx=0 tx=0).stop 2023-11-28T18:28:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 shutdown_connections 2023-11-28T18:28:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8200668d0 0x7fd820068d90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd83415ce10 0x7fd8340c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd83415ca30 0x7fd8340c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 --2- 172.21.15.7:0/4000132885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8340cafe0 0x7fd8341f6870 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 >> 172.21.15.7:0/4000132885 conn(0x7fd83406c060 msgr2=0x7fd8341582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 shutdown_connections 2023-11-28T18:28:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:16.212+0000 7fd8393fd640 1 -- 172.21.15.7:0/4000132885 wait complete. 2023-11-28T18:28:16.234 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:28:18.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:18.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a95d1640 1 Processor -- start 2023-11-28T18:28:18.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a95d1640 1 -- start start 2023-11-28T18:28:18.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a95d1640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f88a4153f00 0x7f88a40ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:18.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a95d1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88a40cb020 0x7f88a40b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:18.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a95d1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a415d420 con 0x7f88a40cb5a0 2023-11-28T18:28:18.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a95d1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a40ca900 con 0x7f88a40cb020 2023-11-28T18:28:18.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a95d1640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a415d700 con 0x7f88a4153f00 2023-11-28T18:28:18.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a27fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88a40cb020 0x7f88a40b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:18.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a27fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88a40cb020 0x7f88a40b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47480/0 (socket says 172.21.15.7:47480) 2023-11-28T18:28:18.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a27fc640 1 -- 172.21.15.7:0/3466846004 learned_addr learned my addr 172.21.15.7:0/3466846004 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:18.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a2ffd640 1 --2- 172.21.15.7:0/3466846004 >> v2:172.21.15.193:3300/0 conn(0x7f88a4153f00 0x7f88a40ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:18.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a27fc640 1 -- 172.21.15.7:0/3466846004 >> v2:172.21.15.193:3300/0 conn(0x7f88a4153f00 msgr2=0x7f88a40ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:18.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a27fc640 1 --2- 172.21.15.7:0/3466846004 >> v2:172.21.15.193:3300/0 conn(0x7f88a4153f00 0x7f88a40ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a27fc640 1 -- 172.21.15.7:0/3466846004 >> v1:172.21.15.7:6789/0 conn(0x7f88a40cb5a0 legacy=0x7f88a40cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:18.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.400+0000 7f88a27fc640 1 -- 172.21.15.7:0/3466846004 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a415d9e0 con 0x7f88a40cb020 2023-11-28T18:28:18.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a27fc640 1 --2- 172.21.15.7:0/3466846004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb020 0x7f88a40b1fc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f88a4066f20 tx=0x7f889801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7f35139ef100e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:18.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a1ffb640 1 -- 172.21.15.7:0/3466846004 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88980aa430 con 0x7f88a40cb020 2023-11-28T18:28:18.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a1ffb640 1 -- 172.21.15.7:0/3466846004 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88980aa590 con 0x7f88a40cb020 2023-11-28T18:28:18.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a1ffb640 1 -- 172.21.15.7:0/3466846004 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88980aa920 con 0x7f88a40cb020 2023-11-28T18:28:18.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a95d1640 1 -- 172.21.15.7:0/3466846004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb020 msgr2=0x7f88a40b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:18.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3466846004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb020 0x7f88a40b1fc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f88a4066f20 tx=0x7f889801c4d0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a95d1640 1 -- 172.21.15.7:0/3466846004 shutdown_connections 2023-11-28T18:28:18.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3466846004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb020 0x7f88a40b1fc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3466846004 >> v2:172.21.15.193:3300/0 conn(0x7f88a4153f00 0x7f88a40ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a95d1640 1 -- 172.21.15.7:0/3466846004 >> 172.21.15.7:0/3466846004 conn(0x7f88a406c060 msgr2=0x7f88a40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:18.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a95d1640 1 -- 172.21.15.7:0/3466846004 shutdown_connections 2023-11-28T18:28:18.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.404+0000 7f88a95d1640 1 -- 172.21.15.7:0/3466846004 wait complete. 2023-11-28T18:28:18.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a95d1640 1 Processor -- start 2023-11-28T18:28:18.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a95d1640 1 -- start start 2023-11-28T18:28:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a95d1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88a40cb020 0x7f88a4157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a95d1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb5a0 0x7f88a4155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a95d1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88a4153f00 0x7f88a4155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a95d1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41549c0 con 0x7f88a40cb5a0 2023-11-28T18:28:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a95d1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41546e0 con 0x7f88a40cb020 2023-11-28T18:28:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a95d1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a4158ab0 con 0x7f88a4153f00 2023-11-28T18:28:18.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb5a0 0x7f88a4155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:18.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb5a0 0x7f88a4155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47488/0 (socket says 172.21.15.7:47488) 2023-11-28T18:28:18.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 -- 172.21.15.7:0/3834803624 learned_addr learned my addr 172.21.15.7:0/3834803624 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:18.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a2ffd640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88a40cb020 0x7f88a4157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:18.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a37fe640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88a4153f00 0x7f88a4155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:18.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 -- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88a4153f00 msgr2=0x7f88a4155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:18.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88a4153f00 0x7f88a4155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 -- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88a40cb020 msgr2=0x7f88a4157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:18.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88a40cb020 0x7f88a4157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 -- 172.21.15.7:0/3834803624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88980094d0 con 0x7f88a40cb5a0 2023-11-28T18:28:18.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a2ffd640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88a40cb020 0x7f88a4157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:28:18.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a37fe640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88a4153f00 0x7f88a4155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:28:18.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f88a27fc640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb5a0 0x7f88a4155300 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f88980190e0 tx=0x7f889801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:18.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f888b7fe640 1 -- 172.21.15.7:0/3834803624 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88980aa430 con 0x7f88a40cb5a0 2023-11-28T18:28:18.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f888b7fe640 1 -- 172.21.15.7:0/3834803624 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88980b3800 con 0x7f88a40cb5a0 2023-11-28T18:28:18.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.408+0000 7f888b7fe640 1 -- 172.21.15.7:0/3834803624 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88980b3bb0 con 0x7f88a40cb5a0 2023-11-28T18:28:18.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.412+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a4158d90 con 0x7f88a40cb5a0 2023-11-28T18:28:18.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.412+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a4159350 con 0x7f88a40cb5a0 2023-11-28T18:28:18.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.412+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a4156030 con 0x7f88a40cb5a0 2023-11-28T18:28:18.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.412+0000 7f888b7fe640 1 -- 172.21.15.7:0/3834803624 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f88980b3d10 con 0x7f88a40cb5a0 2023-11-28T18:28:18.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.412+0000 7f888b7fe640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88900668d0 0x7f8890068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:18.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.412+0000 7f888b7fe640 1 -- 172.21.15.7:0/3834803624 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f889811ea20 con 0x7f88a40cb5a0 2023-11-28T18:28:18.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.412+0000 7f88a2ffd640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88900668d0 0x7f8890068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:18.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.416+0000 7f88a2ffd640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88900668d0 0x7f8890068d90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8894007a70 tx=0x7f8894008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:18.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.420+0000 7f888b7fe640 1 -- 172.21.15.7:0/3834803624 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f88980ebc80 con 0x7f88a40cb5a0 2023-11-28T18:28:18.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.656+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88a404e890 con 0x7f88a40cb5a0 2023-11-28T18:28:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.656+0000 7f888b7fe640 1 -- 172.21.15.7:0/3834803624 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f88980c3020 con 0x7f88a40cb5a0 2023-11-28T18:28:18.660 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:18.660 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:17.189476+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:resolve","state_seq":34,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:18.661 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 29 2023-11-28T18:28:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88900668d0 msgr2=0x7f8890068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88900668d0 0x7f8890068d90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8894007a70 tx=0x7f8894008040 comp rx=0 tx=0).stop 2023-11-28T18:28:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb5a0 msgr2=0x7f88a4155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb5a0 0x7f88a4155300 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f88980190e0 tx=0x7f889801ad10 comp rx=0 tx=0).stop 2023-11-28T18:28:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 shutdown_connections 2023-11-28T18:28:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88900668d0 0x7f8890068d90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88a4153f00 0x7f88a4155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88a40cb5a0 0x7f88a4155300 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 --2- 172.21.15.7:0/3834803624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88a40cb020 0x7f88a4157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 >> 172.21.15.7:0/3834803624 conn(0x7f88a406c060 msgr2=0x7f88a40ce710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 shutdown_connections 2023-11-28T18:28:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:18.660+0000 7f88a95d1640 1 -- 172.21.15.7:0/3834803624 wait complete. 2023-11-28T18:28:18.684 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:28:20.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:20.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c10c79640 1 Processor -- start 2023-11-28T18:28:20.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c10c79640 1 -- start start 2023-11-28T18:28:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c10c79640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c159e40 0x7f8c0c0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c10c79640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c15e360 con 0x7f8c0c0cb6b0 2023-11-28T18:28:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c10c79640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c15d490 con 0x7f8c0c154300 2023-11-28T18:28:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c10c79640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c15a280 con 0x7f8c0c159e40 2023-11-28T18:28:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c0ad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8c0c0cb6b0 0x7f8c0c0ca410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45830/0 (socket says 172.21.15.7:45830) 2023-11-28T18:28:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c0ad9d640 1 -- 172.21.15.7:0/4171610948 learned_addr learned my addr 172.21.15.7:0/4171610948 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:28:20.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c09d9b640 1 --2- 172.21.15.7:0/4171610948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c159e40 0x7f8c0c0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:20.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c0959a640 1 -- 172.21.15.7:0/4171610948 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435256391 0 0) 0x7f8c0c15e360 con 0x7f8c0c0cb6b0 2023-11-28T18:28:20.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c0959a640 1 -- 172.21.15.7:0/4171610948 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf4003530 con 0x7f8c0c0cb6b0 2023-11-28T18:28:20.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c0959a640 1 -- 172.21.15.7:0/4171610948 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156965073 0 0) 0x7f8c0c15d490 con 0x7f8c0c154300 2023-11-28T18:28:20.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c0959a640 1 -- 172.21.15.7:0/4171610948 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c15e360 con 0x7f8c0c154300 2023-11-28T18:28:20.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c09d9b640 1 -- 172.21.15.7:0/4171610948 >> v1:172.21.15.193:6789/0 conn(0x7f8c0c154300 legacy=0x7f8c0c15dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:20.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c09d9b640 1 -- 172.21.15.7:0/4171610948 >> v1:172.21.15.7:6789/0 conn(0x7f8c0c0cb6b0 legacy=0x7f8c0c0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:20.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.848+0000 7f8c09d9b640 1 -- 172.21.15.7:0/4171610948 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c15a480 con 0x7f8c0c159e40 2023-11-28T18:28:20.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.852+0000 7f8c09d9b640 1 --2- 172.21.15.7:0/4171610948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c159e40 0x7f8c0c0cb170 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8c000097f0 tx=0x7f8c0001c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f67af5822b7756 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:20.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.852+0000 7f8c0959a640 1 -- 172.21.15.7:0/4171610948 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000ac600 con 0x7f8c0c159e40 2023-11-28T18:28:20.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.852+0000 7f8c0959a640 1 -- 172.21.15.7:0/4171610948 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c000a8050 con 0x7f8c0c159e40 2023-11-28T18:28:20.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.852+0000 7f8c0959a640 1 -- 172.21.15.7:0/4171610948 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000ac980 con 0x7f8c0c159e40 2023-11-28T18:28:20.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 -- 172.21.15.7:0/4171610948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c159e40 msgr2=0x7f8c0c0cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:20.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 --2- 172.21.15.7:0/4171610948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c159e40 0x7f8c0c0cb170 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8c000097f0 tx=0x7f8c0001c490 comp rx=0 tx=0).stop 2023-11-28T18:28:20.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 -- 172.21.15.7:0/4171610948 shutdown_connections 2023-11-28T18:28:20.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 --2- 172.21.15.7:0/4171610948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c159e40 0x7f8c0c0cb170 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:20.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 -- 172.21.15.7:0/4171610948 >> 172.21.15.7:0/4171610948 conn(0x7f8c0c0662e0 msgr2=0x7f8c0c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:20.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 -- 172.21.15.7:0/4171610948 shutdown_connections 2023-11-28T18:28:20.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 -- 172.21.15.7:0/4171610948 wait complete. 2023-11-28T18:28:20.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 Processor -- start 2023-11-28T18:28:20.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.856+0000 7f8c10c79640 1 -- start start 2023-11-28T18:28:20.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c0cb6b0 0x7f8c0c14edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:20.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c0c154300 0x7f8c0c14f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c0c159e40 0x7f8c0c13bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1f7110 con 0x7f8c0c154300 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c15c3e0 con 0x7f8c0c159e40 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c15c6c0 con 0x7f8c0c0cb6b0 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c0c154300 0x7f8c0c14f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c0c154300 0x7f8c0c14f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47498/0 (socket says 172.21.15.7:47498) 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 -- 172.21.15.7:0/2510520130 learned_addr learned my addr 172.21.15.7:0/2510520130 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c0a59c640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c0cb6b0 0x7f8c0c14edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:20.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c0ad9d640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c0c159e40 0x7f8c0c13bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:20.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 -- 172.21.15.7:0/2510520130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c0cb6b0 msgr2=0x7f8c0c14edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c0cb6b0 0x7f8c0c14edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 -- 172.21.15.7:0/2510520130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c0c159e40 msgr2=0x7f8c0c13bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c0c159e40 0x7f8c0c13bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 -- 172.21.15.7:0/2510520130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c00009420 con 0x7f8c0c154300 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c09d9b640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c0c154300 0x7f8c0c14f2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c0001a0c0 tx=0x7f8c0001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8bfaffd640 1 -- 172.21.15.7:0/2510520130 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000b7450 con 0x7f8c0c154300 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8bfaffd640 1 -- 172.21.15.7:0/2510520130 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c000b75d0 con 0x7f8c0c154300 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8bfaffd640 1 -- 172.21.15.7:0/2510520130 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000b7980 con 0x7f8c0c154300 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c15c9a0 con 0x7f8c0c154300 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c14e890 con 0x7f8c0c154300 2023-11-28T18:28:20.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.860+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c153e70 con 0x7f8c0c154300 2023-11-28T18:28:20.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.864+0000 7f8bfaffd640 1 -- 172.21.15.7:0/2510520130 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8c000b3070 con 0x7f8c0c154300 2023-11-28T18:28:20.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.864+0000 7f8bfaffd640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bec0668d0 0x7f8bec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:20.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.864+0000 7f8bfaffd640 1 -- 172.21.15.7:0/2510520130 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f8c0011e920 con 0x7f8c0c154300 2023-11-28T18:28:20.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.864+0000 7f8c0a59c640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bec0668d0 0x7f8bec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:20.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.864+0000 7f8c0a59c640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bec0668d0 0x7f8bec068d90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0011f0 tx=0x7f8bfc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:20.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:20.868+0000 7f8bfaffd640 1 -- 172.21.15.7:0/2510520130 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8c000eba50 con 0x7f8c0c154300 2023-11-28T18:28:21.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.116+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c0c04e890 con 0x7f8c0c154300 2023-11-28T18:28:21.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.116+0000 7f8bfaffd640 1 -- 172.21.15.7:0/2510520130 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5761 (secure 0 0 0) 0x7f8c000b1090 con 0x7f8c0c154300 2023-11-28T18:28:21.121 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:21.121 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:18.708034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:resolve","state_seq":34,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:21.121 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 30 2023-11-28T18:28:21.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bec0668d0 msgr2=0x7f8bec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:21.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bec0668d0 0x7f8bec068d90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0011f0 tx=0x7f8bfc009040 comp rx=0 tx=0).stop 2023-11-28T18:28:21.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c0c154300 msgr2=0x7f8c0c14f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c0c154300 0x7f8c0c14f2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c0001a0c0 tx=0x7f8c0001acd0 comp rx=0 tx=0).stop 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 shutdown_connections 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bec0668d0 0x7f8bec068d90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c0c159e40 0x7f8c0c13bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c0c154300 0x7f8c0c14f2f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 --2- 172.21.15.7:0/2510520130 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c0c0cb6b0 0x7f8c0c14edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 >> 172.21.15.7:0/2510520130 conn(0x7f8c0c0662e0 msgr2=0x7f8c0c14a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 shutdown_connections 2023-11-28T18:28:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:21.120+0000 7f8c10c79640 1 -- 172.21.15.7:0/2510520130 wait complete. 2023-11-28T18:28:21.142 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:28:23.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:23.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2ba2c8640 1 Processor -- start 2023-11-28T18:28:23.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2ba2c8640 1 -- start start 2023-11-28T18:28:23.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2ba2c8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2b4153ee0 0x7ff2b40cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:23.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2ba2c8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4159c90 0x7ff2b415d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:23.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2ba2c8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff2b415def0 0x7ff2b40ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:23.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2ba2c8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b415d240 con 0x7ff2b4153ee0 2023-11-28T18:28:23.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2ba2c8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b415d520 con 0x7ff2b415def0 2023-11-28T18:28:23.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2ba2c8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b40c4bc0 con 0x7ff2b4159c90 2023-11-28T18:28:23.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b92c6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2b4153ee0 0x7ff2b40cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:23.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b92c6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2b4153ee0 0x7ff2b40cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47504/0 (socket says 172.21.15.7:47504) 2023-11-28T18:28:23.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b92c6640 1 -- 172.21.15.7:0/1145551966 learned_addr learned my addr 172.21.15.7:0/1145551966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:23.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b8ac5640 1 --2- 172.21.15.7:0/1145551966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4159c90 0x7ff2b415d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:23.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b9ac7640 1 --2- 172.21.15.7:0/1145551966 >> v2:172.21.15.193:3300/0 conn(0x7ff2b415def0 0x7ff2b40ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:23.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b92c6640 1 -- 172.21.15.7:0/1145551966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4159c90 msgr2=0x7ff2b415d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:23.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b92c6640 1 --2- 172.21.15.7:0/1145551966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4159c90 0x7ff2b415d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b92c6640 1 -- 172.21.15.7:0/1145551966 >> v2:172.21.15.193:3300/0 conn(0x7ff2b415def0 msgr2=0x7ff2b40ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:23.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b92c6640 1 --2- 172.21.15.7:0/1145551966 >> v2:172.21.15.193:3300/0 conn(0x7ff2b415def0 0x7ff2b40ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.308+0000 7ff2b92c6640 1 -- 172.21.15.7:0/1145551966 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b415a240 con 0x7ff2b4153ee0 2023-11-28T18:28:23.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2b92c6640 1 --2- 172.21.15.7:0/1145551966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4153ee0 0x7ff2b40cb600 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff2b406baa0 tx=0x7ff2b001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04eb2f11c5589e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:23.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2abfff640 1 -- 172.21.15.7:0/1145551966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b00a89b0 con 0x7ff2b4153ee0 2023-11-28T18:28:23.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2abfff640 1 -- 172.21.15.7:0/1145551966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b00a8b10 con 0x7ff2b4153ee0 2023-11-28T18:28:23.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2abfff640 1 -- 172.21.15.7:0/1145551966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b00a8ea0 con 0x7ff2b4153ee0 2023-11-28T18:28:23.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/1145551966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4153ee0 msgr2=0x7ff2b40cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:23.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/1145551966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4153ee0 0x7ff2b40cb600 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff2b406baa0 tx=0x7ff2b001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/1145551966 shutdown_connections 2023-11-28T18:28:23.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/1145551966 >> v2:172.21.15.193:3300/0 conn(0x7ff2b415def0 0x7ff2b40ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/1145551966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b4159c90 0x7ff2b415d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/1145551966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4153ee0 0x7ff2b40cb600 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.312+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/1145551966 >> 172.21.15.7:0/1145551966 conn(0x7ff2b4071e20 msgr2=0x7ff2b4072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:23.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/1145551966 shutdown_connections 2023-11-28T18:28:23.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/1145551966 wait complete. 2023-11-28T18:28:23.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 Processor -- start 2023-11-28T18:28:23.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 -- start start 2023-11-28T18:28:23.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b4153ee0 0x7ff2b414a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:23.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4159c90 0x7ff2b40cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:23.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b415def0 0x7ff2b4152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:23.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b40c6930 con 0x7ff2b4159c90 2023-11-28T18:28:23.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b40ce370 con 0x7ff2b4153ee0 2023-11-28T18:28:23.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2ba2c8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b40c6b80 con 0x7ff2b415def0 2023-11-28T18:28:23.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b8ac5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4159c90 0x7ff2b40cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:23.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b8ac5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4159c90 0x7ff2b40cd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47520/0 (socket says 172.21.15.7:47520) 2023-11-28T18:28:23.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b8ac5640 1 -- 172.21.15.7:0/2236165666 learned_addr learned my addr 172.21.15.7:0/2236165666 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:23.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b92c6640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b4153ee0 0x7ff2b414a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:23.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b9ac7640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b415def0 0x7ff2b4152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:23.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b8ac5640 1 -- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b415def0 msgr2=0x7ff2b4152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:23.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b8ac5640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b415def0 0x7ff2b4152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b8ac5640 1 -- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b4153ee0 msgr2=0x7ff2b414a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:23.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b8ac5640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b4153ee0 0x7ff2b414a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b8ac5640 1 -- 172.21.15.7:0/2236165666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b00094d0 con 0x7ff2b4159c90 2023-11-28T18:28:23.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.316+0000 7ff2b92c6640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b4153ee0 0x7ff2b414a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:28:23.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2b8ac5640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4159c90 0x7ff2b40cd950 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400d800 tx=0x7ff2a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:23.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2a9ffb640 1 -- 172.21.15.7:0/2236165666 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a40096a0 con 0x7ff2b4159c90 2023-11-28T18:28:23.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2a9ffb640 1 -- 172.21.15.7:0/2236165666 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a4009800 con 0x7ff2b4159c90 2023-11-28T18:28:23.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2a9ffb640 1 -- 172.21.15.7:0/2236165666 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a4009990 con 0x7ff2b4159c90 2023-11-28T18:28:23.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b40c6e60 con 0x7ff2b4159c90 2023-11-28T18:28:23.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b40c7420 con 0x7ff2b4159c90 2023-11-28T18:28:23.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b404e890 con 0x7ff2b4159c90 2023-11-28T18:28:23.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2a9ffb640 1 -- 172.21.15.7:0/2236165666 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff2a4009af0 con 0x7ff2b4159c90 2023-11-28T18:28:23.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2a9ffb640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2980668d0 0x7ff298068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:23.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2a9ffb640 1 -- 172.21.15.7:0/2236165666 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7ff2a40848f0 con 0x7ff2b4159c90 2023-11-28T18:28:23.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.320+0000 7ff2b92c6640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2980668d0 0x7ff298068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:23.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.324+0000 7ff2b92c6640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2980668d0 0x7ff298068d90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00b2000 tx=0x7ff2b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:23.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.328+0000 7ff2a9ffb640 1 -- 172.21.15.7:0/2236165666 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2a4051ab0 con 0x7ff2b4159c90 2023-11-28T18:28:23.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.544+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2b4150a30 con 0x7ff2b4159c90 2023-11-28T18:28:23.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.544+0000 7ff2a9ffb640 1 -- 172.21.15.7:0/2236165666 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5763 (secure 0 0 0) 0x7ff2a40550d0 con 0x7ff2b4159c90 2023-11-28T18:28:23.549 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:23.549 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:23.186844+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:23.549 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 31 2023-11-28T18:28:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2980668d0 msgr2=0x7ff298068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2980668d0 0x7ff298068d90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00b2000 tx=0x7ff2b0009040 comp rx=0 tx=0).stop 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4159c90 msgr2=0x7ff2b40cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4159c90 0x7ff2b40cd950 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400d800 tx=0x7ff2a400dcc0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 shutdown_connections 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2980668d0 0x7ff298068d90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2b415def0 0x7ff2b4152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2b4159c90 0x7ff2b40cd950 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 --2- 172.21.15.7:0/2236165666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2b4153ee0 0x7ff2b414a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 >> 172.21.15.7:0/2236165666 conn(0x7ff2b4071e20 msgr2=0x7ff2b40ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:23.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 shutdown_connections 2023-11-28T18:28:23.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:23.548+0000 7ff2ba2c8640 1 -- 172.21.15.7:0/2236165666 wait complete. 2023-11-28T18:28:23.571 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:28:25.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.728+0000 7f18e18c8640 1 Processor -- start 2023-11-28T18:28:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e18c8640 1 -- start start 2023-11-28T18:28:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e18c8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18dc150f00 0x7f18dc0c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e18c8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc0c5d10 con 0x7f18dc0c8010 2023-11-28T18:28:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e18c8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc1569b0 con 0x7f18dc0c7c30 2023-11-28T18:28:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e18c8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18dc070670 con 0x7f18dc150f00 2023-11-28T18:28:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e10c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18dc150f00 0x7f18dc0c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e10c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18dc150f00 0x7f18dc0c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46064/0 (socket says 172.21.15.7:46064) 2023-11-28T18:28:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18dbfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f18dc0c8010 0x7f18dc1507f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43170/0 (socket says 172.21.15.7:43170) 2023-11-28T18:28:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e10c7640 1 -- 172.21.15.7:0/2896033986 learned_addr learned my addr 172.21.15.7:0/2896033986 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e10c7640 1 -- 172.21.15.7:0/2896033986 >> v1:172.21.15.193:6789/0 conn(0x7f18dc0c7c30 legacy=0x7f18dc0c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e10c7640 1 -- 172.21.15.7:0/2896033986 >> v1:172.21.15.7:6789/0 conn(0x7f18dc0c8010 legacy=0x7f18dc1507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18e10c7640 1 -- 172.21.15.7:0/2896033986 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc156c70 con 0x7f18dc150f00 2023-11-28T18:28:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.732+0000 7f18db7fe640 1 -- 172.21.15.7:0/2896033986 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439664822 0 0) 0x7f18dc0c5d10 con 0x7f18dc0c8010 2023-11-28T18:28:25.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18e10c7640 1 --2- 172.21.15.7:0/2896033986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc150f00 0x7f18dc0c61a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009e50 tx=0x7f18d401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6de65c92a6f23f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:25.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18db7fe640 1 -- 172.21.15.7:0/2896033986 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d40af400 con 0x7f18dc150f00 2023-11-28T18:28:25.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18db7fe640 1 -- 172.21.15.7:0/2896033986 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d40ab050 con 0x7f18dc150f00 2023-11-28T18:28:25.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18db7fe640 1 -- 172.21.15.7:0/2896033986 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d40af780 con 0x7f18dc150f00 2023-11-28T18:28:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18e18c8640 1 -- 172.21.15.7:0/2896033986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc150f00 msgr2=0x7f18dc0c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18e18c8640 1 --2- 172.21.15.7:0/2896033986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc150f00 0x7f18dc0c61a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009e50 tx=0x7f18d401f1f0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18e18c8640 1 -- 172.21.15.7:0/2896033986 shutdown_connections 2023-11-28T18:28:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18e18c8640 1 --2- 172.21.15.7:0/2896033986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc150f00 0x7f18dc0c61a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18e18c8640 1 -- 172.21.15.7:0/2896033986 >> 172.21.15.7:0/2896033986 conn(0x7f18dc06c370 msgr2=0x7f18dc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18e18c8640 1 -- 172.21.15.7:0/2896033986 shutdown_connections 2023-11-28T18:28:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.736+0000 7f18e18c8640 1 -- 172.21.15.7:0/2896033986 wait complete. 2023-11-28T18:28:25.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 Processor -- start 2023-11-28T18:28:25.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 -- start start 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc0c7c30 0x7f18dc14d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18dc0c8010 0x7f18dc1ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc150f00 0x7f18dc1efca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc1f2a60 con 0x7f18dc0c7c30 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc1f2cb0 con 0x7f18dc150f00 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc1f2f90 con 0x7f18dc0c8010 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc0c7c30 0x7f18dc14d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc0c7c30 0x7f18dc14d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46066/0 (socket says 172.21.15.7:46066) 2023-11-28T18:28:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 -- 172.21.15.7:0/3319289062 learned_addr learned my addr 172.21.15.7:0/3319289062 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 -- 172.21.15.7:0/3319289062 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18dc0c8010 msgr2=0x7f18dc1ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18dc0c8010 0x7f18dc1ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 -- 172.21.15.7:0/3319289062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc150f00 msgr2=0x7f18dc1efca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc150f00 0x7f18dc1efca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 -- 172.21.15.7:0/3319289062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4009b30 con 0x7f18dc0c7c30 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e08c6640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc0c7c30 0x7f18dc14d5f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f18d000c500 tx=0x7f18d000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18d97fa640 1 -- 172.21.15.7:0/3319289062 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0005220 con 0x7f18dc0c7c30 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18d97fa640 1 -- 172.21.15.7:0/3319289062 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d0005380 con 0x7f18dc0c7c30 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc1f3270 con 0x7f18dc0c7c30 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18d97fa640 1 -- 172.21.15.7:0/3319289062 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0007270 con 0x7f18dc0c7c30 2023-11-28T18:28:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.740+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc1f3830 con 0x7f18dc0c7c30 2023-11-28T18:28:25.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.744+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc1ef4f0 con 0x7f18dc0c7c30 2023-11-28T18:28:25.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.744+0000 7f18d97fa640 1 -- 172.21.15.7:0/3319289062 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f18d00073d0 con 0x7f18dc0c7c30 2023-11-28T18:28:25.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.744+0000 7f18d97fa640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c00668d0 0x7f18c0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:25.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.744+0000 7f18d97fa640 1 -- 172.21.15.7:0/3319289062 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f18d0083f50 con 0x7f18dc0c7c30 2023-11-28T18:28:25.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.744+0000 7f18dbfff640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c00668d0 0x7f18c0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:25.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.744+0000 7f18dbfff640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c00668d0 0x7f18c0068d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f18c8000c10 tx=0x7f18c8007910 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.748+0000 7f18d97fa640 1 -- 172.21.15.7:0/3319289062 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f18d0051080 con 0x7f18dc0c7c30 2023-11-28T18:28:25.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.972+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18dc04e890 con 0x7f18dc0c7c30 2023-11-28T18:28:25.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18d97fa640 1 -- 172.21.15.7:0/3319289062 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5763 (secure 0 0 0) 0x7f18d00546a0 con 0x7f18dc0c7c30 2023-11-28T18:28:25.979 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:25.979 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:23.186844+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 31 2023-11-28T18:28:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c00668d0 msgr2=0x7f18c0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c00668d0 0x7f18c0068d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f18c8000c10 tx=0x7f18c8007910 comp rx=0 tx=0).stop 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc0c7c30 msgr2=0x7f18dc14d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc0c7c30 0x7f18dc14d5f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f18d000c500 tx=0x7f18d000c9c0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 shutdown_connections 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c00668d0 0x7f18c0068d90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc150f00 0x7f18dc1efca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18dc0c8010 0x7f18dc1ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 --2- 172.21.15.7:0/3319289062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18dc0c7c30 0x7f18dc14d5f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 >> 172.21.15.7:0/3319289062 conn(0x7f18dc06c370 msgr2=0x7f18dc072750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 shutdown_connections 2023-11-28T18:28:25.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:25.976+0000 7f18e18c8640 1 -- 172.21.15.7:0/3319289062 wait complete. 2023-11-28T18:28:26.001 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 31, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4284": { "addr": "172.21.15.7:6835/1419029356", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1419029356, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1419029356, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4284, "incarnation": 13, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4285": { "addr": "172.21.15.193:6837/3911527535", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 3911527535, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 3911527535, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 11, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.7:6838/2690556675", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 2690556675, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 2690556675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:28:23.186844+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 4288, "mds_2": 4285, "mds_3": 4284, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:28:28.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:28.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.152+0000 7fd4a23f8640 1 Processor -- start 2023-11-28T18:28:28.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.152+0000 7fd4a23f8640 1 -- start start 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a23f8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd49c15ce10 0x7fd49c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a23f8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c15a3a0 con 0x7fd49c0cafe0 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a23f8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c15a620 con 0x7fd49c15ca30 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a23f8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c15a900 con 0x7fd49c15ce10 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a0bf5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd49c15ce10 0x7fd49c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a1bf7640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd49c0cafe0 0x7fd49c0cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43186/0 (socket says 172.21.15.7:43186) 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a1bf7640 1 -- 172.21.15.7:0/2747837931 learned_addr learned my addr 172.21.15.7:0/2747837931 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a0bf5640 1 -- 172.21.15.7:0/2747837931 >> v1:172.21.15.193:6789/0 conn(0x7fd49c15ca30 legacy=0x7fd49c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a0bf5640 1 -- 172.21.15.7:0/2747837931 >> v1:172.21.15.7:6789/0 conn(0x7fd49c0cafe0 legacy=0x7fd49c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:28.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd493fff640 1 -- 172.21.15.7:0/2747837931 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554886946 0 0) 0x7fd49c15a3a0 con 0x7fd49c0cafe0 2023-11-28T18:28:28.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.156+0000 7fd4a0bf5640 1 -- 172.21.15.7:0/2747837931 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c15abe0 con 0x7fd49c15ce10 2023-11-28T18:28:28.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a0bf5640 1 --2- 172.21.15.7:0/2747837931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ce10 0x7fd49c0ca1d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0097c0 tx=0x7fd48c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e7d6f8a2069ca9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd493fff640 1 -- 172.21.15.7:0/2747837931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0ad660 con 0x7fd49c15ce10 2023-11-28T18:28:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd493fff640 1 -- 172.21.15.7:0/2747837931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c0ad7c0 con 0x7fd49c15ce10 2023-11-28T18:28:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd493fff640 1 -- 172.21.15.7:0/2747837931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0adb50 con 0x7fd49c15ce10 2023-11-28T18:28:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 -- 172.21.15.7:0/2747837931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ce10 msgr2=0x7fd49c0ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 --2- 172.21.15.7:0/2747837931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ce10 0x7fd49c0ca1d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0097c0 tx=0x7fd48c01c450 comp rx=0 tx=0).stop 2023-11-28T18:28:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 -- 172.21.15.7:0/2747837931 shutdown_connections 2023-11-28T18:28:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 --2- 172.21.15.7:0/2747837931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ce10 0x7fd49c0ca1d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 -- 172.21.15.7:0/2747837931 >> 172.21.15.7:0/2747837931 conn(0x7fd49c06c060 msgr2=0x7fd49c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 -- 172.21.15.7:0/2747837931 shutdown_connections 2023-11-28T18:28:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 -- 172.21.15.7:0/2747837931 wait complete. 2023-11-28T18:28:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 Processor -- start 2023-11-28T18:28:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.160+0000 7fd4a23f8640 1 -- start start 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd49c0cafe0 0x7fd49c1f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ca30 0x7fd49c157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd49c15ce10 0x7fd49c1562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c1653c0 con 0x7fd49c15ca30 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c155e20 con 0x7fd49c15ce10 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c1f3b70 con 0x7fd49c0cafe0 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ca30 0x7fd49c157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ca30 0x7fd49c157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46088/0 (socket says 172.21.15.7:46088) 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 -- 172.21.15.7:0/3515291704 learned_addr learned my addr 172.21.15.7:0/3515291704 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a13f6640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd49c0cafe0 0x7fd49c1f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a1bf7640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd49c15ce10 0x7fd49c1562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 -- 172.21.15.7:0/3515291704 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd49c0cafe0 msgr2=0x7fd49c1f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd49c0cafe0 0x7fd49c1f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 -- 172.21.15.7:0/3515291704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd49c15ce10 msgr2=0x7fd49c1562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd49c15ce10 0x7fd49c1562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 -- 172.21.15.7:0/3515291704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c009420 con 0x7fd49c15ca30 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a0bf5640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ca30 0x7fd49c157d60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd48c019060 tx=0x7fd48c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd491ffb640 1 -- 172.21.15.7:0/3515291704 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0b9860 con 0x7fd49c15ca30 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd491ffb640 1 -- 172.21.15.7:0/3515291704 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c0b99e0 con 0x7fd49c15ca30 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd491ffb640 1 -- 172.21.15.7:0/3515291704 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0b9d90 con 0x7fd49c15ca30 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c1571a0 con 0x7fd49c15ca30 2023-11-28T18:28:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c1557d0 con 0x7fd49c15ca30 2023-11-28T18:28:28.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.164+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c155fd0 con 0x7fd49c15ca30 2023-11-28T18:28:28.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.168+0000 7fd491ffb640 1 -- 172.21.15.7:0/3515291704 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd48c0c4460 con 0x7fd49c15ca30 2023-11-28T18:28:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.168+0000 7fd491ffb640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd4800668d0 0x7fd480068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.168+0000 7fd491ffb640 1 -- 172.21.15.7:0/3515291704 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fd48c11f300 con 0x7fd49c15ca30 2023-11-28T18:28:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.168+0000 7fd4a13f6640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd4800668d0 0x7fd480068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.168+0000 7fd4a13f6640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd4800668d0 0x7fd480068d90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd498000f70 tx=0x7fd498009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.172+0000 7fd491ffb640 1 -- 172.21.15.7:0/3515291704 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd49c155fd0 con 0x7fd49c15ca30 2023-11-28T18:28:28.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.392+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd49c155fd0 con 0x7fd49c15ca30 2023-11-28T18:28:28.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.392+0000 7fd491ffb640 1 -- 172.21.15.7:0/3515291704 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5760 (secure 0 0 0) 0x7fd49c155fd0 con 0x7fd49c15ca30 2023-11-28T18:28:28.398 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:28.398 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:28.208572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:28.398 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 32 2023-11-28T18:28:28.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd4800668d0 msgr2=0x7fd480068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:28.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd4800668d0 0x7fd480068d90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd498000f70 tx=0x7fd498009040 comp rx=0 tx=0).stop 2023-11-28T18:28:28.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ca30 msgr2=0x7fd49c157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:28.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ca30 0x7fd49c157d60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd48c019060 tx=0x7fd48c01ac90 comp rx=0 tx=0).stop 2023-11-28T18:28:28.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 shutdown_connections 2023-11-28T18:28:28.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd4800668d0 0x7fd480068d90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:28.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd49c15ce10 0x7fd49c1562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:28.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd49c15ca30 0x7fd49c157d60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:28.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 --2- 172.21.15.7:0/3515291704 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd49c0cafe0 0x7fd49c1f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:28.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 >> 172.21.15.7:0/3515291704 conn(0x7fd49c06c060 msgr2=0x7fd49c15c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:28.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 shutdown_connections 2023-11-28T18:28:28.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:28.396+0000 7fd4a23f8640 1 -- 172.21.15.7:0/3515291704 wait complete. 2023-11-28T18:28:28.422 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:28:28.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:28:28.802 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:28:30.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:30.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd8989d7640 1 Processor -- start 2023-11-28T18:28:30.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd8989d7640 1 -- start start 2023-11-28T18:28:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd8989d7640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd8940ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd8989d7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd8940cafe0 0x7fd8940ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd8989d7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89415a3a0 con 0x7fd89415ca30 2023-11-28T18:28:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd8989d7640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89415a620 con 0x7fd8940cafe0 2023-11-28T18:28:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd8989d7640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89415a900 con 0x7fd89415ce10 2023-11-28T18:28:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd89359e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd8940cafe0 0x7fd8940ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd892d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd89415ca30 0x7fd894159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43202/0 (socket says 172.21.15.7:43202) 2023-11-28T18:28:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd892d9d640 1 -- 172.21.15.7:0/1311315977 learned_addr learned my addr 172.21.15.7:0/1311315977 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:28:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd88bfff640 1 --2- 172.21.15.7:0/1311315977 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd8940ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd89359e640 1 -- 172.21.15.7:0/1311315977 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 msgr2=0x7fd8940ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:30.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd89359e640 1 --2- 172.21.15.7:0/1311315977 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd8940ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd89359e640 1 -- 172.21.15.7:0/1311315977 >> v1:172.21.15.7:6789/0 conn(0x7fd89415ca30 legacy=0x7fd894159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:30.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd89359e640 1 -- 172.21.15.7:0/1311315977 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89415abe0 con 0x7fd8940cafe0 2023-11-28T18:28:30.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.576+0000 7fd88bfff640 1 --2- 172.21.15.7:0/1311315977 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd8940ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:28:30.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.580+0000 7fd89359e640 1 --2- 172.21.15.7:0/1311315977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940ca710 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009d90 tx=0x7fd88c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f4429c38b314e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:30.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd89259c640 1 -- 172.21.15.7:0/1311315977 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c0af600 con 0x7fd8940cafe0 2023-11-28T18:28:30.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd89259c640 1 -- 172.21.15.7:0/1311315977 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c0ab050 con 0x7fd8940cafe0 2023-11-28T18:28:30.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd89259c640 1 -- 172.21.15.7:0/1311315977 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c0af980 con 0x7fd8940cafe0 2023-11-28T18:28:30.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 -- 172.21.15.7:0/1311315977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 msgr2=0x7fd8940ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:30.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 --2- 172.21.15.7:0/1311315977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940ca710 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009d90 tx=0x7fd88c01f130 comp rx=0 tx=0).stop 2023-11-28T18:28:30.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 -- 172.21.15.7:0/1311315977 shutdown_connections 2023-11-28T18:28:30.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 --2- 172.21.15.7:0/1311315977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940ca710 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 --2- 172.21.15.7:0/1311315977 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd8940ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 -- 172.21.15.7:0/1311315977 >> 172.21.15.7:0/1311315977 conn(0x7fd89406c060 msgr2=0x7fd8940662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:30.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 -- 172.21.15.7:0/1311315977 shutdown_connections 2023-11-28T18:28:30.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 -- 172.21.15.7:0/1311315977 wait complete. 2023-11-28T18:28:30.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 Processor -- start 2023-11-28T18:28:30.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.584+0000 7fd8989d7640 1 -- start start 2023-11-28T18:28:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89415ca30 0x7fd89415b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd894156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd894154d20 con 0x7fd8940cafe0 2023-11-28T18:28:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8941568b0 con 0x7fd89415ca30 2023-11-28T18:28:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940c3cf0 con 0x7fd89415ce10 2023-11-28T18:28:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940c37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46110/0 (socket says 172.21.15.7:46110) 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 -- 172.21.15.7:0/2123692187 learned_addr learned my addr 172.21.15.7:0/2123692187 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd88bfff640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89415ca30 0x7fd89415b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd89359e640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd894156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 -- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 msgr2=0x7fd894156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd894156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 -- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89415ca30 msgr2=0x7fd89415b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89415ca30 0x7fd89415b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 -- 172.21.15.7:0/2123692187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c009a70 con 0x7fd8940cafe0 2023-11-28T18:28:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd88bfff640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89415ca30 0x7fd89415b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:28:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd892d9d640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940c37b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd88400c500 tx=0x7fd88400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd88b7fe640 1 -- 172.21.15.7:0/2123692187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884005cf0 con 0x7fd8940cafe0 2023-11-28T18:28:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd88b7fe640 1 -- 172.21.15.7:0/2123692187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd884005e50 con 0x7fd8940cafe0 2023-11-28T18:28:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8941f3350 con 0x7fd8940cafe0 2023-11-28T18:28:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd88b7fe640 1 -- 172.21.15.7:0/2123692187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8840072f0 con 0x7fd8940cafe0 2023-11-28T18:28:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8941524d0 con 0x7fd8940cafe0 2023-11-28T18:28:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.588+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89404e890 con 0x7fd8940cafe0 2023-11-28T18:28:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.592+0000 7fd88b7fe640 1 -- 172.21.15.7:0/2123692187 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd884007450 con 0x7fd8940cafe0 2023-11-28T18:28:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.592+0000 7fd88b7fe640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd874066950 0x7fd874068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.592+0000 7fd88b7fe640 1 -- 172.21.15.7:0/2123692187 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fd884083930 con 0x7fd8940cafe0 2023-11-28T18:28:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.592+0000 7fd88bfff640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd874066950 0x7fd874068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.592+0000 7fd88bfff640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd874066950 0x7fd874068e10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd87c006fd0 tx=0x7fd87c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.596+0000 7fd88b7fe640 1 -- 172.21.15.7:0/2123692187 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd884050c70 con 0x7fd8940cafe0 2023-11-28T18:28:30.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.820+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8940cad10 con 0x7fd8940cafe0 2023-11-28T18:28:30.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.820+0000 7fd88b7fe640 1 -- 172.21.15.7:0/2123692187 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5760 (secure 0 0 0) 0x7fd884054290 con 0x7fd8940cafe0 2023-11-28T18:28:30.826 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:30.826 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:28.208572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:30.826 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 32 2023-11-28T18:28:30.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd874066950 msgr2=0x7fd874068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:30.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd874066950 0x7fd874068e10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd87c006fd0 tx=0x7fd87c008040 comp rx=0 tx=0).stop 2023-11-28T18:28:30.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 msgr2=0x7fd8940c37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940c37b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd88400c500 tx=0x7fd88400c9c0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 shutdown_connections 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd874066950 0x7fd874068e10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd89415ce10 0x7fd894156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89415ca30 0x7fd89415b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 --2- 172.21.15.7:0/2123692187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8940cafe0 0x7fd8940c37b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 >> 172.21.15.7:0/2123692187 conn(0x7fd89406c060 msgr2=0x7fd8940cd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 shutdown_connections 2023-11-28T18:28:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:30.824+0000 7fd8989d7640 1 -- 172.21.15.7:0/2123692187 wait complete. 2023-11-28T18:28:30.852 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:28:32.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:33.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f028169c640 1 Processor -- start 2023-11-28T18:28:33.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f028169c640 1 -- start start 2023-11-28T18:28:33.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f028169c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 0x7f027c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:33.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f028169c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f027c15ce10 0x7f027c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f028169c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c15a620 con 0x7f027c0cafe0 2023-11-28T18:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f028169c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c15a900 con 0x7f027c15ce10 2023-11-28T18:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f028169c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c15abe0 con 0x7f027c15ca30 2023-11-28T18:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f027b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f027c15ce10 0x7f027c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f027b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f027c15ce10 0x7f027c0ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46122/0 (socket says 172.21.15.7:46122) 2023-11-28T18:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f027b7fe640 1 -- 172.21.15.7:0/695141315 learned_addr learned my addr 172.21.15.7:0/695141315 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f027bfff640 1 --2- 172.21.15.7:0/695141315 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 0x7f027c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:33.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f027b7fe640 1 -- 172.21.15.7:0/695141315 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 msgr2=0x7f027c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:33.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f027b7fe640 1 --2- 172.21.15.7:0/695141315 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 0x7f027c159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f027b7fe640 1 -- 172.21.15.7:0/695141315 >> v1:172.21.15.193:6789/0 conn(0x7f027c0cafe0 legacy=0x7f027c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:33.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.040+0000 7f027b7fe640 1 -- 172.21.15.7:0/695141315 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c0cbb10 con 0x7f027c15ce10 2023-11-28T18:28:33.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f027b7fe640 1 --2- 172.21.15.7:0/695141315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 0x7f027c0ca1d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0270009870 tx=0x7f027001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c90aa2c48e75be9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:33.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f027affd640 1 -- 172.21.15.7:0/695141315 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02700a8a30 con 0x7f027c15ce10 2023-11-28T18:28:33.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f027affd640 1 -- 172.21.15.7:0/695141315 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02700a8b90 con 0x7f027c15ce10 2023-11-28T18:28:33.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f027affd640 1 -- 172.21.15.7:0/695141315 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02700b0440 con 0x7f027c15ce10 2023-11-28T18:28:33.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f028169c640 1 -- 172.21.15.7:0/695141315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 msgr2=0x7f027c0ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:33.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f028169c640 1 --2- 172.21.15.7:0/695141315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 0x7f027c0ca1d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0270009870 tx=0x7f027001ed30 comp rx=0 tx=0).stop 2023-11-28T18:28:33.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f028169c640 1 -- 172.21.15.7:0/695141315 shutdown_connections 2023-11-28T18:28:33.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f028169c640 1 --2- 172.21.15.7:0/695141315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 0x7f027c0ca1d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f028169c640 1 --2- 172.21.15.7:0/695141315 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 0x7f027c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f028169c640 1 -- 172.21.15.7:0/695141315 >> 172.21.15.7:0/695141315 conn(0x7f027c06c060 msgr2=0x7f027c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:33.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f028169c640 1 -- 172.21.15.7:0/695141315 shutdown_connections 2023-11-28T18:28:33.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.044+0000 7f028169c640 1 -- 172.21.15.7:0/695141315 wait complete. 2023-11-28T18:28:33.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 Processor -- start 2023-11-28T18:28:33.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 -- start start 2023-11-28T18:28:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f027c0cafe0 0x7f027c154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 0x7f027c1553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 0x7f027c14e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c1e48e0 con 0x7f027c15ce10 2023-11-28T18:28:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0c87a0 con 0x7f027c0cafe0 2023-11-28T18:28:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c1e4ad0 con 0x7f027c15ca30 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f027bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f027c0cafe0 0x7f027c154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0280e9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 0x7f027c14e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f027bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f027c0cafe0 0x7f027c154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50238/0 (socket says 172.21.15.7:50238) 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f027bfff640 1 -- 172.21.15.7:0/3823473186 learned_addr learned my addr 172.21.15.7:0/3823473186 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f027b7fe640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 0x7f027c1553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0280e9b640 1 -- 172.21.15.7:0/3823473186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 msgr2=0x7f027c1553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0280e9b640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 0x7f027c1553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0280e9b640 1 -- 172.21.15.7:0/3823473186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f027c0cafe0 msgr2=0x7f027c154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0280e9b640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f027c0cafe0 0x7f027c154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0280e9b640 1 -- 172.21.15.7:0/3823473186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02700094d0 con 0x7f027c15ce10 2023-11-28T18:28:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0280e9b640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 0x7f027c14e870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f027c0cb490 tx=0x7f027400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0278ff9640 1 -- 172.21.15.7:0/3823473186 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274003b80 con 0x7f027c15ce10 2023-11-28T18:28:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0278ff9640 1 -- 172.21.15.7:0/3823473186 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0274003ce0 con 0x7f027c15ce10 2023-11-28T18:28:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c1e4db0 con 0x7f027c15ce10 2023-11-28T18:28:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f0278ff9640 1 -- 172.21.15.7:0/3823473186 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274005310 con 0x7f027c15ce10 2023-11-28T18:28:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.048+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c1e5370 con 0x7f027c15ce10 2023-11-28T18:28:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.052+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c04e890 con 0x7f027c15ce10 2023-11-28T18:28:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.052+0000 7f0278ff9640 1 -- 172.21.15.7:0/3823473186 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0274005470 con 0x7f027c15ce10 2023-11-28T18:28:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.052+0000 7f0278ff9640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0268066900 0x7f0268068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.052+0000 7f0278ff9640 1 -- 172.21.15.7:0/3823473186 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f0274084100 con 0x7f027c15ce10 2023-11-28T18:28:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.052+0000 7f027bfff640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0268066900 0x7f0268068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.052+0000 7f027bfff640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0268066900 0x7f0268068dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f026c007b10 tx=0x7f026c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.056+0000 7f0278ff9640 1 -- 172.21.15.7:0/3823473186 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f02740512c0 con 0x7f027c15ce10 2023-11-28T18:28:33.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.288+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f027c0cad10 con 0x7f027c15ce10 2023-11-28T18:28:33.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.288+0000 7f0278ff9640 1 -- 172.21.15.7:0/3823473186 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5760 (secure 0 0 0) 0x7f02740548e0 con 0x7f027c15ce10 2023-11-28T18:28:33.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:33.294 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:33.237815+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:33.296 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 33 2023-11-28T18:28:33.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0268066900 msgr2=0x7f0268068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:33.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0268066900 0x7f0268068dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f026c007b10 tx=0x7f026c008040 comp rx=0 tx=0).stop 2023-11-28T18:28:33.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 msgr2=0x7f027c14e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:33.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 0x7f027c14e870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f027c0cb490 tx=0x7f027400cb90 comp rx=0 tx=0).stop 2023-11-28T18:28:33.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 shutdown_connections 2023-11-28T18:28:33.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0268066900 0x7f0268068dc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f027c15ce10 0x7f027c14e870 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f027c15ca30 0x7f027c1553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 --2- 172.21.15.7:0/3823473186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f027c0cafe0 0x7f027c154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:33.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 >> 172.21.15.7:0/3823473186 conn(0x7f027c06c060 msgr2=0x7f027c15b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:33.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 shutdown_connections 2023-11-28T18:28:33.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:33.292+0000 7f028169c640 1 -- 172.21.15.7:0/3823473186 wait complete. 2023-11-28T18:28:33.317 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=4, replacing gid=4279 2023-11-28T18:28:33.317 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.f 2023-11-28T18:28:39.318 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-11-28T18:28:39.319 INFO:tasks.ceph.mds.f:Restarting daemon 2023-11-28T18:28:39.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-11-28T18:28:39.323 INFO:tasks.ceph.mds.f:Started 2023-11-28T18:28:39.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:39.411 INFO:tasks.ceph.mds.f.smithi193.stdout:starting mds.f at 2023-11-28T18:28:39.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc22bb640 1 Processor -- start 2023-11-28T18:28:39.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc22bb640 1 -- start start 2023-11-28T18:28:39.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc22bb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dbc15ca30 0x7f9dbc159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:39.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc22bb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9dbc15ce10 0x7f9dbc0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:39.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc22bb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc15a620 con 0x7f9dbc0cafe0 2023-11-28T18:28:39.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc22bb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dbc15a900 con 0x7f9dbc15ca30 2023-11-28T18:28:39.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc22bb640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dbc15abe0 con 0x7f9dbc15ce10 2023-11-28T18:28:39.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc12b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dbc15ca30 0x7f9dbc159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:39.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc12b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dbc15ca30 0x7f9dbc159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48764/0 (socket says 172.21.15.7:48764) 2023-11-28T18:28:39.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc12b9640 1 -- 172.21.15.7:0/2941610811 learned_addr learned my addr 172.21.15.7:0/2941610811 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:39.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc0ab8640 1 --2- 172.21.15.7:0/2941610811 >> v2:172.21.15.193:3300/0 conn(0x7f9dbc15ce10 0x7f9dbc0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:39.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc12b9640 1 -- 172.21.15.7:0/2941610811 >> v2:172.21.15.193:3300/0 conn(0x7f9dbc15ce10 msgr2=0x7f9dbc0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:39.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc12b9640 1 --2- 172.21.15.7:0/2941610811 >> v2:172.21.15.193:3300/0 conn(0x7f9dbc15ce10 0x7f9dbc0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc12b9640 1 -- 172.21.15.7:0/2941610811 >> v1:172.21.15.7:6789/0 conn(0x7f9dbc0cafe0 legacy=0x7f9dbc0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:39.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.472+0000 7f9dc12b9640 1 -- 172.21.15.7:0/2941610811 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc0cbb10 con 0x7f9dbc15ca30 2023-11-28T18:28:39.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc12b9640 1 --2- 172.21.15.7:0/2941610811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc159c90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0d0200 tx=0x7f9db801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f208193e1ede2d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:39.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9db3fff640 1 -- 172.21.15.7:0/2941610811 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db80ac640 con 0x7f9dbc15ca30 2023-11-28T18:28:39.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9db3fff640 1 -- 172.21.15.7:0/2941610811 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db80ac7a0 con 0x7f9dbc15ca30 2023-11-28T18:28:39.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9db3fff640 1 -- 172.21.15.7:0/2941610811 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db80acb30 con 0x7f9dbc15ca30 2023-11-28T18:28:39.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2941610811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 msgr2=0x7f9dbc159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:39.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2941610811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc159c90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0d0200 tx=0x7f9db801c4d0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2941610811 shutdown_connections 2023-11-28T18:28:39.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2941610811 >> v2:172.21.15.193:3300/0 conn(0x7f9dbc15ce10 0x7f9dbc0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2941610811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc159c90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2941610811 >> 172.21.15.7:0/2941610811 conn(0x7f9dbc06c060 msgr2=0x7f9dbc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:39.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2941610811 shutdown_connections 2023-11-28T18:28:39.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.476+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2941610811 wait complete. 2023-11-28T18:28:39.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc22bb640 1 Processor -- start 2023-11-28T18:28:39.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc22bb640 1 -- start start 2023-11-28T18:28:39.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc22bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc0cafe0 0x7f9dbc1f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:39.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc22bb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc0c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:39.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc22bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9dbc15ce10 0x7f9dbc15d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:39.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc22bb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc0c35c0 con 0x7f9dbc15ca30 2023-11-28T18:28:39.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc22bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc1572f0 con 0x7f9dbc0cafe0 2023-11-28T18:28:39.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc22bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc1f68f0 con 0x7f9dbc15ce10 2023-11-28T18:28:39.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc0c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:39.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc0c4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48776/0 (socket says 172.21.15.7:48776) 2023-11-28T18:28:39.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 -- 172.21.15.7:0/2901748680 learned_addr learned my addr 172.21.15.7:0/2901748680 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:39.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc12b9640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc0cafe0 0x7f9dbc1f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:39.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc1aba640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9dbc15ce10 0x7f9dbc15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:39.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 -- 172.21.15.7:0/2901748680 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9dbc15ce10 msgr2=0x7f9dbc15d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:39.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9dbc15ce10 0x7f9dbc15d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 -- 172.21.15.7:0/2901748680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc0cafe0 msgr2=0x7f9dbc1f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:39.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc0cafe0 0x7f9dbc1f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 -- 172.21.15.7:0/2901748680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db80094d0 con 0x7f9dbc15ca30 2023-11-28T18:28:39.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9dc0ab8640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc0c4700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00e960 tx=0x7f9dac00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:39.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.480+0000 7f9db1ffb640 1 -- 172.21.15.7:0/2901748680 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac009690 con 0x7f9dbc15ca30 2023-11-28T18:28:39.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.484+0000 7f9db1ffb640 1 -- 172.21.15.7:0/2901748680 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac00ca00 con 0x7f9dbc15ca30 2023-11-28T18:28:39.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.484+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc1f6bd0 con 0x7f9dbc15ca30 2023-11-28T18:28:39.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.484+0000 7f9db1ffb640 1 -- 172.21.15.7:0/2901748680 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac00cd70 con 0x7f9dbc15ca30 2023-11-28T18:28:39.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.484+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc0c32e0 con 0x7f9dbc15ca30 2023-11-28T18:28:39.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.484+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc157070 con 0x7f9dbc15ca30 2023-11-28T18:28:39.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.484+0000 7f9db1ffb640 1 -- 172.21.15.7:0/2901748680 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9dac013470 con 0x7f9dbc15ca30 2023-11-28T18:28:39.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.484+0000 7f9db1ffb640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9da00668d0 0x7f9da0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:39.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.488+0000 7f9db1ffb640 1 -- 172.21.15.7:0/2901748680 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f9dac0849f0 con 0x7f9dbc15ca30 2023-11-28T18:28:39.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.488+0000 7f9dc12b9640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9da00668d0 0x7f9da0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:39.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.488+0000 7f9dc12b9640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9da00668d0 0x7f9da0068d90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9db80ae000 tx=0x7f9db80a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.492+0000 7f9db1ffb640 1 -- 172.21.15.7:0/2901748680 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9dac051c50 con 0x7f9dbc15ca30 2023-11-28T18:28:39.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.740+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dbc04e890 con 0x7f9dbc15ca30 2023-11-28T18:28:39.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.740+0000 7f9db1ffb640 1 -- 172.21.15.7:0/2901748680 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5759 (secure 0 0 0) 0x7f9dac017090 con 0x7f9dbc15ca30 2023-11-28T18:28:39.744 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:39.744 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:39.744 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 34 2023-11-28T18:28:39.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9da00668d0 msgr2=0x7f9da0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:39.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9da00668d0 0x7f9da0068d90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9db80ae000 tx=0x7f9db80a7000 comp rx=0 tx=0).stop 2023-11-28T18:28:39.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 msgr2=0x7f9dbc0c4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:39.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc0c4700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00e960 tx=0x7f9dac00ee20 comp rx=0 tx=0).stop 2023-11-28T18:28:39.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 shutdown_connections 2023-11-28T18:28:39.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9da00668d0 0x7f9da0068d90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9dbc15ce10 0x7f9dbc15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dbc15ca30 0x7f9dbc0c4700 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 --2- 172.21.15.7:0/2901748680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc0cafe0 0x7f9dbc1f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:39.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 >> 172.21.15.7:0/2901748680 conn(0x7f9dbc06c060 msgr2=0x7f9dbc0cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:39.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 shutdown_connections 2023-11-28T18:28:39.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:39.744+0000 7f9dc22bb640 1 -- 172.21.15.7:0/2901748680 wait complete. 2023-11-28T18:28:39.767 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-11-28T18:28:41.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:28:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f41088640 1 Processor -- start 2023-11-28T18:28:41.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f41088640 1 -- start start 2023-11-28T18:28:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f41088640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f3c15ca30 0x7f1f3c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f41088640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1f3c15ce10 0x7f1f3c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f41088640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c15a620 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f41088640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c15a900 con 0x7f1f3c15ca30 2023-11-28T18:28:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f41088640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c15abe0 con 0x7f1f3c15ce10 2023-11-28T18:28:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f3bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f3c15ca30 0x7f1f3c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f3bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f3c15ca30 0x7f1f3c159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48788/0 (socket says 172.21.15.7:48788) 2023-11-28T18:28:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f3bfff640 1 -- 172.21.15.7:0/3578939461 learned_addr learned my addr 172.21.15.7:0/3578939461 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f3b7fe640 1 --2- 172.21.15.7:0/3578939461 >> v2:172.21.15.193:3300/0 conn(0x7f1f3c15ce10 0x7f1f3c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f3bfff640 1 -- 172.21.15.7:0/3578939461 >> v2:172.21.15.193:3300/0 conn(0x7f1f3c15ce10 msgr2=0x7f1f3c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:28:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f3bfff640 1 --2- 172.21.15.7:0/3578939461 >> v2:172.21.15.193:3300/0 conn(0x7f1f3c15ce10 0x7f1f3c0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f3bfff640 1 -- 172.21.15.7:0/3578939461 >> v1:172.21.15.193:6789/0 conn(0x7f1f3c0cafe0 legacy=0x7f1f3c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:41.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.936+0000 7f1f3bfff640 1 -- 172.21.15.7:0/3578939461 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c0cbb10 con 0x7f1f3c15ca30 2023-11-28T18:28:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f3bfff640 1 --2- 172.21.15.7:0/3578939461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ca30 0x7f1f3c159c90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0d0200 tx=0x7f1f2c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afcdf4f535e4ef57 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f3affd640 1 -- 172.21.15.7:0/3578939461 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c0ac640 con 0x7f1f3c15ca30 2023-11-28T18:28:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f3affd640 1 -- 172.21.15.7:0/3578939461 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c0ac7a0 con 0x7f1f3c15ca30 2023-11-28T18:28:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f3affd640 1 -- 172.21.15.7:0/3578939461 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c0acb30 con 0x7f1f3c15ca30 2023-11-28T18:28:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f41088640 1 -- 172.21.15.7:0/3578939461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ca30 msgr2=0x7f1f3c159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f41088640 1 --2- 172.21.15.7:0/3578939461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ca30 0x7f1f3c159c90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0d0200 tx=0x7f1f2c01c4d0 comp rx=0 tx=0).stop 2023-11-28T18:28:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f41088640 1 -- 172.21.15.7:0/3578939461 shutdown_connections 2023-11-28T18:28:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f41088640 1 --2- 172.21.15.7:0/3578939461 >> v2:172.21.15.193:3300/0 conn(0x7f1f3c15ce10 0x7f1f3c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f41088640 1 --2- 172.21.15.7:0/3578939461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ca30 0x7f1f3c159c90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f41088640 1 -- 172.21.15.7:0/3578939461 >> 172.21.15.7:0/3578939461 conn(0x7f1f3c06c060 msgr2=0x7f1f3c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f41088640 1 -- 172.21.15.7:0/3578939461 shutdown_connections 2023-11-28T18:28:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.940+0000 7f1f41088640 1 -- 172.21.15.7:0/3578939461 wait complete. 2023-11-28T18:28:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f41088640 1 Processor -- start 2023-11-28T18:28:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f41088640 1 -- start start 2023-11-28T18:28:41.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f41088640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f3c0cafe0 0x7f1f3c1639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f41088640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f3c15ca30 0x7f1f3c0c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f41088640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ce10 0x7f1f3c156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f41088640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c156070 con 0x7f1f3c15ce10 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f41088640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c155010 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f41088640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c164060 con 0x7f1f3c15ca30 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f40887640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ce10 0x7f1f3c156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f3bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f3c0cafe0 0x7f1f3c1639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f40887640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ce10 0x7f1f3c156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48792/0 (socket says 172.21.15.7:48792) 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f40887640 1 -- 172.21.15.7:0/1659999026 learned_addr learned my addr 172.21.15.7:0/1659999026 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:28:41.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f3b7fe640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f3c15ca30 0x7f1f3c0c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f3bfff640 1 -- 172.21.15.7:0/1659999026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f3c15ca30 msgr2=0x7f1f3c0c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f3bfff640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f3c15ca30 0x7f1f3c0c3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f3bfff640 1 -- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ce10 msgr2=0x7f1f3c156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f3bfff640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ce10 0x7f1f3c156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f3bfff640 1 -- 172.21.15.7:0/1659999026 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0094d0 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.944+0000 7f1f3bfff640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f3c0cafe0 0x7f1f3c1639b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0cd160 tx=0x7f1f2c01ad10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.948+0000 7f1f38ff9640 1 -- 172.21.15.7:0/1659999026 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c0b7650 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.948+0000 7f1f38ff9640 1 -- 172.21.15.7:0/1659999026 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c0b77b0 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.948+0000 7f1f38ff9640 1 -- 172.21.15.7:0/1659999026 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c0b7b60 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.948+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c1552b0 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.948+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c155870 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.948+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c04e890 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.948+0000 7f1f38ff9640 1 -- 172.21.15.7:0/1659999026 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1f2c0b3070 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.952+0000 7f1f38ff9640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f280668d0 0x7f1f28068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:28:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.952+0000 7f1f38ff9640 1 -- 172.21.15.7:0/1659999026 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f1f2c11f0c0 con 0x7f1f3c0cafe0 2023-11-28T18:28:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.952+0000 7f1f3b7fe640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f280668d0 0x7f1f28068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:28:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.952+0000 7f1f3b7fe640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f280668d0 0x7f1f28068d90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0c6610 tx=0x7f1f30009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:28:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:41.952+0000 7f1f38ff9640 1 -- 172.21.15.7:0/1659999026 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1f2c0ec1e0 con 0x7f1f3c0cafe0 2023-11-28T18:28:42.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.188+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f3c0cad10 con 0x7f1f3c0cafe0 2023-11-28T18:28:42.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.188+0000 7f1f38ff9640 1 -- 172.21.15.7:0/1659999026 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f1f2c0b1020 con 0x7f1f3c0cafe0 2023-11-28T18:28:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:28:42.193 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:28:42.194 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:28:42.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f280668d0 msgr2=0x7f1f28068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:42.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f280668d0 0x7f1f28068d90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0c6610 tx=0x7f1f30009190 comp rx=0 tx=0).stop 2023-11-28T18:28:42.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f3c0cafe0 msgr2=0x7f1f3c1639b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:28:42.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f3c0cafe0 0x7f1f3c1639b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0cd160 tx=0x7f1f2c01ad10 comp rx=0 tx=0).stop 2023-11-28T18:28:42.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 shutdown_connections 2023-11-28T18:28:42.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f280668d0 0x7f1f28068d90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:42.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c15ce10 0x7f1f3c156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:42.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f3c15ca30 0x7f1f3c0c3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:42.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 --2- 172.21.15.7:0/1659999026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f3c0cafe0 0x7f1f3c1639b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:28:42.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 >> 172.21.15.7:0/1659999026 conn(0x7f1f3c06c060 msgr2=0x7f1f3c0c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:28:42.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 shutdown_connections 2023-11-28T18:28:42.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:28:42.192+0000 7f1f41088640 1 -- 172.21.15.7:0/1659999026 wait complete. 2023-11-28T18:28:42.217 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-11-28T18:28:42.217 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 64 secs before thrashing 2023-11-28T18:28:58.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:28:58.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:29:28.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:29:28.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:29:46.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:29:46.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476359e640 1 Processor -- start 2023-11-28T18:29:46.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476359e640 1 -- start start 2023-11-28T18:29:46.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476359e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764153ee0 0x7f47640cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476359e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4764159c90 0x7f476415d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476359e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476415d240 con 0x7f476415def0 2023-11-28T18:29:46.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476359e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f476415d520 con 0x7f4764159c90 2023-11-28T18:29:46.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476359e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47640c4bc0 con 0x7f4764153ee0 2023-11-28T18:29:46.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f4761d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4764159c90 0x7f476415d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f4762d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f476415def0 0x7f47640ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40132/0 (socket says 172.21.15.7:40132) 2023-11-28T18:29:46.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f4762d9d640 1 -- 172.21.15.7:0/1811838963 learned_addr learned my addr 172.21.15.7:0/1811838963 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:29:46.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476259c640 1 --2- 172.21.15.7:0/1811838963 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764153ee0 0x7f47640cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f4761d9b640 1 -- 172.21.15.7:0/1811838963 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764153ee0 msgr2=0x7f47640cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:29:46.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f4761d9b640 1 --2- 172.21.15.7:0/1811838963 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764153ee0 0x7f47640cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f4761d9b640 1 -- 172.21.15.7:0/1811838963 >> v1:172.21.15.7:6789/0 conn(0x7f476415def0 legacy=0x7f47640ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f4761d9b640 1 -- 172.21.15.7:0/1811838963 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476415a240 con 0x7f4764159c90 2023-11-28T18:29:46.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.367+0000 7f476159a640 1 -- 172.21.15.7:0/1811838963 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716114492 0 0) 0x7f476415d240 con 0x7f476415def0 2023-11-28T18:29:46.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.371+0000 7f4761d9b640 1 --2- 172.21.15.7:0/1811838963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764159c90 0x7f476415d9b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f47580097c0 tx=0x7f475801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19a7c87e466a99 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:46.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.371+0000 7f476159a640 1 -- 172.21.15.7:0/1811838963 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47580a8970 con 0x7f4764159c90 2023-11-28T18:29:46.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.371+0000 7f476159a640 1 -- 172.21.15.7:0/1811838963 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47580a8ad0 con 0x7f4764159c90 2023-11-28T18:29:46.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.371+0000 7f476159a640 1 -- 172.21.15.7:0/1811838963 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47580a8e60 con 0x7f4764159c90 2023-11-28T18:29:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 -- 172.21.15.7:0/1811838963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764159c90 msgr2=0x7f476415d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 --2- 172.21.15.7:0/1811838963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764159c90 0x7f476415d9b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f47580097c0 tx=0x7f475801ec70 comp rx=0 tx=0).stop 2023-11-28T18:29:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 -- 172.21.15.7:0/1811838963 shutdown_connections 2023-11-28T18:29:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 --2- 172.21.15.7:0/1811838963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764159c90 0x7f476415d9b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 --2- 172.21.15.7:0/1811838963 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764153ee0 0x7f47640cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 -- 172.21.15.7:0/1811838963 >> 172.21.15.7:0/1811838963 conn(0x7f4764071e20 msgr2=0x7f4764072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 -- 172.21.15.7:0/1811838963 shutdown_connections 2023-11-28T18:29:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 -- 172.21.15.7:0/1811838963 wait complete. 2023-11-28T18:29:46.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 Processor -- start 2023-11-28T18:29:46.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.375+0000 7f476359e640 1 -- start start 2023-11-28T18:29:46.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4764153ee0 0x7f4764157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764159c90 0x7f47640cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f476415def0 0x7f47640cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47640b52b0 con 0x7f476415def0 2023-11-28T18:29:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47641f3f90 con 0x7f4764153ee0 2023-11-28T18:29:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47640b54e0 con 0x7f4764159c90 2023-11-28T18:29:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f476415def0 0x7f47640cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f476415def0 0x7f47640cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49428/0 (socket says 172.21.15.7:49428) 2023-11-28T18:29:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 -- 172.21.15.7:0/2343916873 learned_addr learned my addr 172.21.15.7:0/2343916873 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4761d9b640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764159c90 0x7f47640cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476259c640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4764153ee0 0x7f4764157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 -- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764159c90 msgr2=0x7f47640cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764159c90 0x7f47640cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 -- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4764153ee0 msgr2=0x7f4764157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4764153ee0 0x7f4764157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 -- 172.21.15.7:0/2343916873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4758009420 con 0x7f476415def0 2023-11-28T18:29:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4761d9b640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764159c90 0x7f47640cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:29:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4762d9d640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f476415def0 0x7f47640cdec0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f475c001570 tx=0x7f475c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4752ffd640 1 -- 172.21.15.7:0/2343916873 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c00a640 con 0x7f476415def0 2023-11-28T18:29:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4752ffd640 1 -- 172.21.15.7:0/2343916873 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475c00a7a0 con 0x7f476415def0 2023-11-28T18:29:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f4752ffd640 1 -- 172.21.15.7:0/2343916873 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c00ab90 con 0x7f476415def0 2023-11-28T18:29:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47641f3380 con 0x7f476415def0 2023-11-28T18:29:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47641f3940 con 0x7f476415def0 2023-11-28T18:29:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.379+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476404e890 con 0x7f476415def0 2023-11-28T18:29:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.383+0000 7f4752ffd640 1 -- 172.21.15.7:0/2343916873 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f475c00acf0 con 0x7f476415def0 2023-11-28T18:29:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.383+0000 7f4752ffd640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4744066900 0x7f4744068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.383+0000 7f4752ffd640 1 -- 172.21.15.7:0/2343916873 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f475c083a90 con 0x7f476415def0 2023-11-28T18:29:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.383+0000 7f476259c640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4744066900 0x7f4744068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.383+0000 7f476259c640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4744066900 0x7f4744068dc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4754009ac0 tx=0x7f4754009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.387+0000 7f4752ffd640 1 -- 172.21.15.7:0/2343916873 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f475c050bd0 con 0x7f476415def0 2023-11-28T18:29:46.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.579+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47641f40d0 con 0x7f476415def0 2023-11-28T18:29:46.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.579+0000 7f4752ffd640 1 -- 172.21.15.7:0/2343916873 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f475c018030 con 0x7f476415def0 2023-11-28T18:29:46.584 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:29:46.584 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:29:46.584 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.579+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4744066900 msgr2=0x7f4744068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.579+0000 7f476359e640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4744066900 0x7f4744068dc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4754009ac0 tx=0x7f4754009190 comp rx=0 tx=0).stop 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.579+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f476415def0 msgr2=0x7f47640cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.579+0000 7f476359e640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f476415def0 0x7f47640cdec0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f475c001570 tx=0x7f475c00cc70 comp rx=0 tx=0).stop 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.583+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 shutdown_connections 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.583+0000 7f476359e640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4744066900 0x7f4744068dc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.583+0000 7f476359e640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f476415def0 0x7f47640cdec0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.583+0000 7f476359e640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4764159c90 0x7f47640cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.583+0000 7f476359e640 1 --2- 172.21.15.7:0/2343916873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4764153ee0 0x7f4764157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.583+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 >> 172.21.15.7:0/2343916873 conn(0x7f4764071e20 msgr2=0x7f47640c5560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:46.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.583+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 shutdown_connections 2023-11-28T18:29:46.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.583+0000 7f476359e640 1 -- 172.21.15.7:0/2343916873 wait complete. 2023-11-28T18:29:46.609 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=3) 2023-11-28T18:29:46.609 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T18:29:46.609 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:29:46.624 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T18:29:46.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:29:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f2305796640 1 Processor -- start 2023-11-28T18:29:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f2305796640 1 -- start start 2023-11-28T18:29:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f2305796640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2300153ee0 0x7f23000cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f2305796640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2300159c90 0x7f230015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f2305796640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230015d240 con 0x7f230015def0 2023-11-28T18:29:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f2305796640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230015d520 con 0x7f2300159c90 2023-11-28T18:29:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f2305796640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23000c4bc0 con 0x7f2300153ee0 2023-11-28T18:29:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f22ff7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2300159c90 0x7f230015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f22ff7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2300159c90 0x7f230015d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49442/0 (socket says 172.21.15.7:49442) 2023-11-28T18:29:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f22ff7fe640 1 -- 172.21.15.7:0/914080047 learned_addr learned my addr 172.21.15.7:0/914080047 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f22fffff640 1 --2- 172.21.15.7:0/914080047 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2300153ee0 0x7f23000cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f22ff7fe640 1 -- 172.21.15.7:0/914080047 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2300153ee0 msgr2=0x7f23000cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:29:46.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f22ff7fe640 1 --2- 172.21.15.7:0/914080047 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2300153ee0 0x7f23000cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f22ff7fe640 1 -- 172.21.15.7:0/914080047 >> v1:172.21.15.193:6789/0 conn(0x7f230015def0 legacy=0x7f23000ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.839+0000 7f22ff7fe640 1 -- 172.21.15.7:0/914080047 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230015a240 con 0x7f2300159c90 2023-11-28T18:29:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f22ff7fe640 1 --2- 172.21.15.7:0/914080047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f230015d9b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009870 tx=0x7f22f401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba76fcc8c9f5800 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f22feffd640 1 -- 172.21.15.7:0/914080047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f40a8a30 con 0x7f2300159c90 2023-11-28T18:29:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f22feffd640 1 -- 172.21.15.7:0/914080047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f40a8b90 con 0x7f2300159c90 2023-11-28T18:29:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f22feffd640 1 -- 172.21.15.7:0/914080047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f40b0440 con 0x7f2300159c90 2023-11-28T18:29:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f2305796640 1 -- 172.21.15.7:0/914080047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 msgr2=0x7f230015d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f2305796640 1 --2- 172.21.15.7:0/914080047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f230015d9b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009870 tx=0x7f22f401ed30 comp rx=0 tx=0).stop 2023-11-28T18:29:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f2305796640 1 -- 172.21.15.7:0/914080047 shutdown_connections 2023-11-28T18:29:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f2305796640 1 --2- 172.21.15.7:0/914080047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f230015d9b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f2305796640 1 --2- 172.21.15.7:0/914080047 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2300153ee0 0x7f23000cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f2305796640 1 -- 172.21.15.7:0/914080047 >> 172.21.15.7:0/914080047 conn(0x7f2300071e20 msgr2=0x7f2300072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f2305796640 1 -- 172.21.15.7:0/914080047 shutdown_connections 2023-11-28T18:29:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.843+0000 7f2305796640 1 -- 172.21.15.7:0/914080047 wait complete. 2023-11-28T18:29:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2305796640 1 Processor -- start 2023-11-28T18:29:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2305796640 1 -- start start 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2305796640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2300153ee0 0x7f23000b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2305796640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f23001f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2305796640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f230015def0 0x7f23001f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2305796640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23001f2f70 con 0x7f2300159c90 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2305796640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000ce290 con 0x7f2300153ee0 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2305796640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300151660 con 0x7f230015def0 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f23001f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f23001f3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49458/0 (socket says 172.21.15.7:49458) 2023-11-28T18:29:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f22ff7fe640 1 -- 172.21.15.7:0/1294109998 learned_addr learned my addr 172.21.15.7:0/1294109998 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f22fffff640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2300153ee0 0x7f23000b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f2304f95640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f230015def0 0x7f23001f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.847+0000 7f22ff7fe640 1 -- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f230015def0 msgr2=0x7f23001f3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22ff7fe640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f230015def0 0x7f23001f3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22ff7fe640 1 -- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2300153ee0 msgr2=0x7f23000b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22ff7fe640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2300153ee0 0x7f23000b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22ff7fe640 1 -- 172.21.15.7:0/1294109998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f40094d0 con 0x7f2300159c90 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f2304f95640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f230015def0 0x7f23001f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22fffff640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2300153ee0 0x7f23000b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22ff7fe640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f23001f3400 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f22f401d3e0 tx=0x7f22f401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22fcff9640 1 -- 172.21.15.7:0/1294109998 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f40b0dd0 con 0x7f2300159c90 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22fcff9640 1 -- 172.21.15.7:0/1294109998 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f40bb460 con 0x7f2300159c90 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2300151940 con 0x7f2300159c90 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22fcff9640 1 -- 172.21.15.7:0/1294109998 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f40bb810 con 0x7f2300159c90 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2300151f00 con 0x7f2300159c90 2023-11-28T18:29:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230004e890 con 0x7f2300159c90 2023-11-28T18:29:46.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.851+0000 7f22fcff9640 1 -- 172.21.15.7:0/1294109998 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f22f40b83f0 con 0x7f2300159c90 2023-11-28T18:29:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.855+0000 7f22fcff9640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22ec0668d0 0x7f22ec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.855+0000 7f22fcff9640 1 -- 172.21.15.7:0/1294109998 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f22f4121ef0 con 0x7f2300159c90 2023-11-28T18:29:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.855+0000 7f22fffff640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22ec0668d0 0x7f22ec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:46.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.855+0000 7f22fffff640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22ec0668d0 0x7f22ec068d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009a40 tx=0x7f22f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:46.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:46.859+0000 7f22fcff9640 1 -- 172.21.15.7:0/1294109998 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f22f40ef010 con 0x7f2300159c90 2023-11-28T18:29:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23000c53e0 con 0x7f2300159c90 2023-11-28T18:29:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f22fcff9640 1 -- 172.21.15.7:0/1294109998 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f22f40c0090 con 0x7f2300159c90 2023-11-28T18:29:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:29:47.068 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:29:47.068 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22ec0668d0 msgr2=0x7f22ec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22ec0668d0 0x7f22ec068d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009a40 tx=0x7f22f0008040 comp rx=0 tx=0).stop 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 msgr2=0x7f23001f3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f23001f3400 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f22f401d3e0 tx=0x7f22f401bad0 comp rx=0 tx=0).stop 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 shutdown_connections 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22ec0668d0 0x7f22ec068d90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f230015def0 0x7f23001f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2300159c90 0x7f23001f3400 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 --2- 172.21.15.7:0/1294109998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2300153ee0 0x7f23000b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.063+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 >> 172.21.15.7:0/1294109998 conn(0x7f2300071e20 msgr2=0x7f2300149820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.067+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 shutdown_connections 2023-11-28T18:29:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:47.067+0000 7f2305796640 1 -- 172.21.15.7:0/1294109998 wait complete. 2023-11-28T18:29:47.092 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:29:49.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:29:49.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910dead640 1 Processor -- start 2023-11-28T18:29:49.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910dead640 1 -- start start 2023-11-28T18:29:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910dead640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91080cafe0 0x7f91080cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910dead640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910815a3a0 con 0x7f910815ce10 2023-11-28T18:29:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910dead640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910815a620 con 0x7f910815ca30 2023-11-28T18:29:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910dead640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f910815a900 con 0x7f91080cafe0 2023-11-28T18:29:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910d6ac640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91080cafe0 0x7f91080cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:49.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f9107fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f910815ce10 0x7f91080ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40142/0 (socket says 172.21.15.7:40142) 2023-11-28T18:29:49.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f9107fff640 1 -- 172.21.15.7:0/2047308587 learned_addr learned my addr 172.21.15.7:0/2047308587 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:29:49.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910d6ac640 1 -- 172.21.15.7:0/2047308587 >> v1:172.21.15.193:6789/0 conn(0x7f910815ca30 legacy=0x7f9108159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:49.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f910d6ac640 1 -- 172.21.15.7:0/2047308587 >> v1:172.21.15.7:6789/0 conn(0x7f910815ce10 legacy=0x7f91080ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:49.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.247+0000 7f91077fe640 1 -- 172.21.15.7:0/2047308587 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594761393 0 0) 0x7f910815a3a0 con 0x7f910815ce10 2023-11-28T18:29:49.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.251+0000 7f910d6ac640 1 -- 172.21.15.7:0/2047308587 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910815abe0 con 0x7f91080cafe0 2023-11-28T18:29:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.251+0000 7f910d6ac640 1 --2- 172.21.15.7:0/2047308587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f91080cc5f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9100009d90 tx=0x7f910001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97de52e42ab14ae0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f91077fe640 1 -- 172.21.15.7:0/2047308587 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91000a58f0 con 0x7f91080cafe0 2023-11-28T18:29:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f91077fe640 1 -- 172.21.15.7:0/2047308587 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91000a5a50 con 0x7f91080cafe0 2023-11-28T18:29:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f91077fe640 1 -- 172.21.15.7:0/2047308587 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91000a5de0 con 0x7f91080cafe0 2023-11-28T18:29:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 -- 172.21.15.7:0/2047308587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 msgr2=0x7f91080cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 --2- 172.21.15.7:0/2047308587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f91080cc5f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9100009d90 tx=0x7f910001c920 comp rx=0 tx=0).stop 2023-11-28T18:29:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 -- 172.21.15.7:0/2047308587 shutdown_connections 2023-11-28T18:29:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 --2- 172.21.15.7:0/2047308587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f91080cc5f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 -- 172.21.15.7:0/2047308587 >> 172.21.15.7:0/2047308587 conn(0x7f910806c060 msgr2=0x7f91080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 -- 172.21.15.7:0/2047308587 shutdown_connections 2023-11-28T18:29:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 -- 172.21.15.7:0/2047308587 wait complete. 2023-11-28T18:29:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 Processor -- start 2023-11-28T18:29:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.255+0000 7f910dead640 1 -- start start 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f910814ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f910815ca30 0x7f91080c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f910815ce10 0x7f91081567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91081f65a0 con 0x7f91080cafe0 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91080c4ed0 con 0x7f910815ca30 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91081f5990 con 0x7f910815ce10 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f910814ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f910814ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49466/0 (socket says 172.21.15.7:49466) 2023-11-28T18:29:49.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 -- 172.21.15.7:0/480082773 learned_addr learned my addr 172.21.15.7:0/480082773 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f9107fff640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f910815ca30 0x7f91080c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910d6ac640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f910815ce10 0x7f91081567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 -- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f910815ce10 msgr2=0x7f91081567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f910815ce10 0x7f91081567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 -- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f910815ca30 msgr2=0x7f91080c45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f910815ca30 0x7f91080c45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 -- 172.21.15.7:0/480082773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9100009a70 con 0x7f91080cafe0 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910d6ac640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f910815ce10 0x7f91081567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:29:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910ceab640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f910814ef20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00c500 tx=0x7f90fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:49.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f91057fa640 1 -- 172.21.15.7:0/480082773 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc005220 con 0x7f91080cafe0 2023-11-28T18:29:49.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f91057fa640 1 -- 172.21.15.7:0/480082773 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc005380 con 0x7f91080cafe0 2023-11-28T18:29:49.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91081f5c70 con 0x7f91080cafe0 2023-11-28T18:29:49.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f91057fa640 1 -- 172.21.15.7:0/480082773 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc007270 con 0x7f91080cafe0 2023-11-28T18:29:49.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91081f6230 con 0x7f91080cafe0 2023-11-28T18:29:49.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.259+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910804e890 con 0x7f91080cafe0 2023-11-28T18:29:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.263+0000 7f91057fa640 1 -- 172.21.15.7:0/480082773 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f90fc0073d0 con 0x7f91080cafe0 2023-11-28T18:29:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.263+0000 7f91057fa640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f90ec0668d0 0x7f90ec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.263+0000 7f91057fa640 1 -- 172.21.15.7:0/480082773 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f90fc083d70 con 0x7f91080cafe0 2023-11-28T18:29:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.263+0000 7f9107fff640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f90ec0668d0 0x7f90ec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.263+0000 7f9107fff640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f90ec0668d0 0x7f90ec068d90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f91080cf690 tx=0x7f90f40079e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:49.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.267+0000 7f91057fa640 1 -- 172.21.15.7:0/480082773 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f90fc050ea0 con 0x7f91080cafe0 2023-11-28T18:29:49.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.503+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91080cad10 con 0x7f91080cafe0 2023-11-28T18:29:49.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.503+0000 7f91057fa640 1 -- 172.21.15.7:0/480082773 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f90fc0544c0 con 0x7f91080cafe0 2023-11-28T18:29:49.510 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:29:49.510 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:29:49.511 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:29:49.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f90ec0668d0 msgr2=0x7f90ec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f90ec0668d0 0x7f90ec068d90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f91080cf690 tx=0x7f90f40079e0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 msgr2=0x7f910814ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f910814ef20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00c500 tx=0x7f90fc00c9c0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 shutdown_connections 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f90ec0668d0 0x7f90ec068d90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f910815ce10 0x7f91081567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f910815ca30 0x7f91080c45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 --2- 172.21.15.7:0/480082773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91080cafe0 0x7f910814ef20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 >> 172.21.15.7:0/480082773 conn(0x7f910806c060 msgr2=0x7f910814fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 shutdown_connections 2023-11-28T18:29:49.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:49.507+0000 7f910dead640 1 -- 172.21.15.7:0/480082773 wait complete. 2023-11-28T18:29:49.533 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:29:51.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:29:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.679+0000 7faa0efb8640 1 Processor -- start 2023-11-28T18:29:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.679+0000 7faa0efb8640 1 -- start start 2023-11-28T18:29:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.679+0000 7faa0efb8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa0815d350 0x7faa0806a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.679+0000 7faa0efb8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa080c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.679+0000 7faa0efb8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080c8750 con 0x7faa081540a0 2023-11-28T18:29:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.679+0000 7faa0efb8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa080c8a30 con 0x7faa0815d350 2023-11-28T18:29:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.679+0000 7faa0efb8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa080731c0 con 0x7faa0806ab30 2023-11-28T18:29:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.679+0000 7faa07fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa0815d350 0x7faa0806a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa07fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa0815d350 0x7faa0806a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49476/0 (socket says 172.21.15.7:49476) 2023-11-28T18:29:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa07fff640 1 -- 172.21.15.7:0/1240370504 learned_addr learned my addr 172.21.15.7:0/1240370504 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa0d555640 1 --2- 172.21.15.7:0/1240370504 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa080c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa07fff640 1 -- 172.21.15.7:0/1240370504 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 msgr2=0x7faa080c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:29:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa07fff640 1 --2- 172.21.15.7:0/1240370504 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa080c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa07fff640 1 -- 172.21.15.7:0/1240370504 >> v1:172.21.15.193:6789/0 conn(0x7faa081540a0 legacy=0x7faa08072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa07fff640 1 -- 172.21.15.7:0/1240370504 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0815d840 con 0x7faa0815d350 2023-11-28T18:29:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa0d555640 1 --2- 172.21.15.7:0/1240370504 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa080c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:29:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa07fff640 1 --2- 172.21.15.7:0/1240370504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa0815d350 0x7faa0806a5f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009870 tx=0x7fa9f401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2daa2c88c7fb6b18 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa077fe640 1 -- 172.21.15.7:0/1240370504 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40ab600 con 0x7faa0815d350 2023-11-28T18:29:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa077fe640 1 -- 172.21.15.7:0/1240370504 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f40a7050 con 0x7faa0815d350 2023-11-28T18:29:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.683+0000 7faa077fe640 1 -- 172.21.15.7:0/1240370504 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40ab980 con 0x7faa0815d350 2023-11-28T18:29:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- 172.21.15.7:0/1240370504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa0815d350 msgr2=0x7faa0806a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1240370504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa0815d350 0x7faa0806a5f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009870 tx=0x7fa9f401c540 comp rx=0 tx=0).stop 2023-11-28T18:29:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- 172.21.15.7:0/1240370504 shutdown_connections 2023-11-28T18:29:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1240370504 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa080c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1240370504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa0815d350 0x7faa0806a5f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- 172.21.15.7:0/1240370504 >> 172.21.15.7:0/1240370504 conn(0x7faa08073cd0 msgr2=0x7faa080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- 172.21.15.7:0/1240370504 shutdown_connections 2023-11-28T18:29:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- 172.21.15.7:0/1240370504 wait complete. 2023-11-28T18:29:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 Processor -- start 2023-11-28T18:29:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- start start 2023-11-28T18:29:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa0814c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa081540a0 0x7faa08156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa0815d350 0x7faa08156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa081537c0 con 0x7faa081540a0 2023-11-28T18:29:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0815af20 con 0x7faa0815d350 2023-11-28T18:29:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.687+0000 7faa0efb8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0814e6f0 con 0x7faa0806ab30 2023-11-28T18:29:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa081540a0 0x7faa08156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa081540a0 0x7faa08156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49488/0 (socket says 172.21.15.7:49488) 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 -- 172.21.15.7:0/1680414154 learned_addr learned my addr 172.21.15.7:0/1680414154 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa0cd54640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa0814c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa0d555640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa0815d350 0x7faa08156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 -- 172.21.15.7:0/1680414154 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 msgr2=0x7faa0814c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa0814c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 -- 172.21.15.7:0/1680414154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa0815d350 msgr2=0x7faa08156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa0815d350 0x7faa08156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 -- 172.21.15.7:0/1680414154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f40094d0 con 0x7faa081540a0 2023-11-28T18:29:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa07fff640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa081540a0 0x7faa08156620 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa9f401a170 tx=0x7fa9f40192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa057fa640 1 -- 172.21.15.7:0/1680414154 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40b4840 con 0x7faa081540a0 2023-11-28T18:29:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa057fa640 1 -- 172.21.15.7:0/1680414154 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f40b49c0 con 0x7faa081540a0 2023-11-28T18:29:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa08153200 con 0x7faa081540a0 2023-11-28T18:29:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa057fa640 1 -- 172.21.15.7:0/1680414154 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40b4d70 con 0x7faa081540a0 2023-11-28T18:29:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa08156100 con 0x7faa081540a0 2023-11-28T18:29:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa08155f40 con 0x7faa081540a0 2023-11-28T18:29:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.691+0000 7faa057fa640 1 -- 172.21.15.7:0/1680414154 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa9f40b2040 con 0x7faa081540a0 2023-11-28T18:29:51.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.695+0000 7faa057fa640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9ec0668d0 0x7fa9ec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:51.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.695+0000 7faa057fa640 1 -- 172.21.15.7:0/1680414154 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fa9f411e610 con 0x7faa081540a0 2023-11-28T18:29:51.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.695+0000 7faa0cd54640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9ec0668d0 0x7fa9ec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:51.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.695+0000 7faa0cd54640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9ec0668d0 0x7fa9ec068d90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc007fd0 tx=0x7fa9fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:51.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.699+0000 7faa057fa640 1 -- 172.21.15.7:0/1680414154 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa9f40eb7c0 con 0x7faa081540a0 2023-11-28T18:29:51.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa0804e890 con 0x7faa081540a0 2023-11-28T18:29:51.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa057fa640 1 -- 172.21.15.7:0/1680414154 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7fa9f40b0090 con 0x7faa081540a0 2023-11-28T18:29:51.951 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:29:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:29:51.951 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:29:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9ec0668d0 msgr2=0x7fa9ec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9ec0668d0 0x7fa9ec068d90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc007fd0 tx=0x7fa9fc009040 comp rx=0 tx=0).stop 2023-11-28T18:29:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa081540a0 msgr2=0x7faa08156620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa081540a0 0x7faa08156620 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa9f401a170 tx=0x7fa9f40192e0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 shutdown_connections 2023-11-28T18:29:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9ec0668d0 0x7fa9ec068d90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa0815d350 0x7faa08156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa081540a0 0x7faa08156620 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 --2- 172.21.15.7:0/1680414154 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa0806ab30 0x7faa0814c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:51.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 >> 172.21.15.7:0/1680414154 conn(0x7faa08073cd0 msgr2=0x7faa0815c090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:51.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.947+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 shutdown_connections 2023-11-28T18:29:51.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:51.951+0000 7faa0efb8640 1 -- 172.21.15.7:0/1680414154 wait complete. 2023-11-28T18:29:51.974 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:29:53.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:29:54.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.131+0000 7ff955daa640 1 Processor -- start 2023-11-28T18:29:54.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.131+0000 7ff955daa640 1 -- start start 2023-11-28T18:29:54.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff955daa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff9501542e0 0x7ff95015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:54.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff955daa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9500cbb70 0x7ff9500cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:54.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff955daa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9500cc810 0x7ff950072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:54.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff955daa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff950071d50 con 0x7ff9500cbb70 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff955daa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff950072030 con 0x7ff9501542e0 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff955daa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500ca920 con 0x7ff9500cc810 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9500cbb70 0x7ff9500cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9500cbb70 0x7ff9500cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47166/0 (socket says 172.21.15.7:47166) 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94effd640 1 -- 172.21.15.7:0/1598304312 learned_addr learned my addr 172.21.15.7:0/1598304312 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94f7fe640 1 --2- 172.21.15.7:0/1598304312 >> v2:172.21.15.193:3300/0 conn(0x7ff9501542e0 0x7ff95015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94ffff640 1 --2- 172.21.15.7:0/1598304312 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9500cc810 0x7ff950072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94effd640 1 -- 172.21.15.7:0/1598304312 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9500cc810 msgr2=0x7ff950072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:29:54.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94effd640 1 --2- 172.21.15.7:0/1598304312 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9500cc810 0x7ff950072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94effd640 1 -- 172.21.15.7:0/1598304312 >> v2:172.21.15.193:3300/0 conn(0x7ff9501542e0 msgr2=0x7ff95015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:29:54.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94effd640 1 --2- 172.21.15.7:0/1598304312 >> v2:172.21.15.193:3300/0 conn(0x7ff9501542e0 0x7ff95015e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.135+0000 7ff94effd640 1 -- 172.21.15.7:0/1598304312 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95015e550 con 0x7ff9500cbb70 2023-11-28T18:29:54.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff94effd640 1 --2- 172.21.15.7:0/1598304312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cc2d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff950071d20 tx=0x7ff94401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77bdf0064ebc7873 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:54.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff94e7fc640 1 -- 172.21.15.7:0/1598304312 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440ac630 con 0x7ff9500cbb70 2023-11-28T18:29:54.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff94e7fc640 1 -- 172.21.15.7:0/1598304312 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440ac790 con 0x7ff9500cbb70 2023-11-28T18:29:54.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff94e7fc640 1 -- 172.21.15.7:0/1598304312 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440acb20 con 0x7ff9500cbb70 2023-11-28T18:29:54.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 -- 172.21.15.7:0/1598304312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 msgr2=0x7ff9500cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:54.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 --2- 172.21.15.7:0/1598304312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cc2d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff950071d20 tx=0x7ff94401c500 comp rx=0 tx=0).stop 2023-11-28T18:29:54.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 -- 172.21.15.7:0/1598304312 shutdown_connections 2023-11-28T18:29:54.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 --2- 172.21.15.7:0/1598304312 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9500cc810 0x7ff950072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 --2- 172.21.15.7:0/1598304312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cc2d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 --2- 172.21.15.7:0/1598304312 >> v2:172.21.15.193:3300/0 conn(0x7ff9501542e0 0x7ff95015e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 -- 172.21.15.7:0/1598304312 >> 172.21.15.7:0/1598304312 conn(0x7ff9500717e0 msgr2=0x7ff9500662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:54.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 -- 172.21.15.7:0/1598304312 shutdown_connections 2023-11-28T18:29:54.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.139+0000 7ff955daa640 1 -- 172.21.15.7:0/1598304312 wait complete. 2023-11-28T18:29:54.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 Processor -- start 2023-11-28T18:29:54.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 -- start start 2023-11-28T18:29:54.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:54.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9500cc810 0x7ff9500ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:54.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9501542e0 0x7ff9500cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9501f2bb0 con 0x7ff9500cbb70 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9501f3a10 con 0x7ff9500cc810 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9501f2e00 con 0x7ff9501542e0 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47178/0 (socket says 172.21.15.7:47178) 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 -- 172.21.15.7:0/2315884368 learned_addr learned my addr 172.21.15.7:0/2315884368 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94effd640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9500cc810 0x7ff9500ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94ffff640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9501542e0 0x7ff9500cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 -- 172.21.15.7:0/2315884368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9501542e0 msgr2=0x7ff9500cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9501542e0 0x7ff9500cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 -- 172.21.15.7:0/2315884368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9500cc810 msgr2=0x7ff9500ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:54.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9500cc810 0x7ff9500ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 -- 172.21.15.7:0/2315884368 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440094d0 con 0x7ff9500cbb70 2023-11-28T18:29:54.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff94f7fe640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cf910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff94000d5f0 tx=0x7ff94000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:54.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff937fff640 1 -- 172.21.15.7:0/2315884368 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9400125f0 con 0x7ff9500cbb70 2023-11-28T18:29:54.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff937fff640 1 -- 172.21.15.7:0/2315884368 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff940012750 con 0x7ff9500cbb70 2023-11-28T18:29:54.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9501f30e0 con 0x7ff9500cbb70 2023-11-28T18:29:54.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff937fff640 1 -- 172.21.15.7:0/2315884368 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff940012ae0 con 0x7ff9500cbb70 2023-11-28T18:29:54.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.143+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9501f36a0 con 0x7ff9500cbb70 2023-11-28T18:29:54.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.147+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95006c670 con 0x7ff9500cbb70 2023-11-28T18:29:54.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.147+0000 7ff937fff640 1 -- 172.21.15.7:0/2315884368 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff940012c40 con 0x7ff9500cbb70 2023-11-28T18:29:54.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.147+0000 7ff937fff640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff93c0668d0 0x7ff93c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:54.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.147+0000 7ff937fff640 1 -- 172.21.15.7:0/2315884368 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7ff9400848f0 con 0x7ff9500cbb70 2023-11-28T18:29:54.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.147+0000 7ff94effd640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff93c0668d0 0x7ff93c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:54.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.147+0000 7ff94effd640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff93c0668d0 0x7ff93c068d90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff9440ae000 tx=0x7ff944008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:54.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.151+0000 7ff937fff640 1 -- 172.21.15.7:0/2315884368 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff940051ac0 con 0x7ff9500cbb70 2023-11-28T18:29:54.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.347+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff95004e890 con 0x7ff9500cbb70 2023-11-28T18:29:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.347+0000 7ff937fff640 1 -- 172.21.15.7:0/2315884368 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7ff940007380 con 0x7ff9500cbb70 2023-11-28T18:29:54.354 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:29:54.354 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:29:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:29:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff93c0668d0 msgr2=0x7ff93c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff93c0668d0 0x7ff93c068d90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff9440ae000 tx=0x7ff944008040 comp rx=0 tx=0).stop 2023-11-28T18:29:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 msgr2=0x7ff9500cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cf910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff94000d5f0 tx=0x7ff94000dab0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 shutdown_connections 2023-11-28T18:29:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff93c0668d0 0x7ff93c068d90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff9501542e0 0x7ff9500cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9500cc810 0x7ff9500ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 --2- 172.21.15.7:0/2315884368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9500cbb70 0x7ff9500cf910 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 >> 172.21.15.7:0/2315884368 conn(0x7ff9500717e0 msgr2=0x7ff9500c5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 shutdown_connections 2023-11-28T18:29:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:54.351+0000 7ff955daa640 1 -- 172.21.15.7:0/2315884368 wait complete. 2023-11-28T18:29:54.378 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:29:56.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:29:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f5095417640 1 Processor -- start 2023-11-28T18:29:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f5095417640 1 -- start start 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f5095417640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f509015ca30 0x7f5090159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f5095417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f5095417640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f509015a620 con 0x7f509015ce10 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f5095417640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509015a900 con 0x7f509015ca30 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f5095417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509015abe0 con 0x7f50900cafe0 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f508ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f509015ca30 0x7f5090159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f508ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f509015ca30 0x7f5090159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47192/0 (socket says 172.21.15.7:47192) 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f508ffff640 1 -- 172.21.15.7:0/601954826 learned_addr learned my addr 172.21.15.7:0/601954826 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.539+0000 7f5094c16640 1 --2- 172.21.15.7:0/601954826 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.543+0000 7f508ffff640 1 -- 172.21.15.7:0/601954826 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 msgr2=0x7f50900cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:29:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.543+0000 7f508ffff640 1 --2- 172.21.15.7:0/601954826 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.543+0000 7f508ffff640 1 -- 172.21.15.7:0/601954826 >> v1:172.21.15.193:6789/0 conn(0x7f509015ce10 legacy=0x7f50900ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.543+0000 7f508ffff640 1 -- 172.21.15.7:0/601954826 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50900cbb10 con 0x7f509015ca30 2023-11-28T18:29:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.543+0000 7f508ffff640 1 --2- 172.21.15.7:0/601954826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ca30 0x7f5090159c90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5080009870 tx=0x7f508001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abef47052484d3ff server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f508effd640 1 -- 172.21.15.7:0/601954826 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50800a8a30 con 0x7f509015ca30 2023-11-28T18:29:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f508effd640 1 -- 172.21.15.7:0/601954826 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50800a8b90 con 0x7f509015ca30 2023-11-28T18:29:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f508effd640 1 -- 172.21.15.7:0/601954826 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50800b0440 con 0x7f509015ca30 2023-11-28T18:29:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 -- 172.21.15.7:0/601954826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ca30 msgr2=0x7f5090159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 --2- 172.21.15.7:0/601954826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ca30 0x7f5090159c90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5080009870 tx=0x7f508001ed30 comp rx=0 tx=0).stop 2023-11-28T18:29:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 -- 172.21.15.7:0/601954826 shutdown_connections 2023-11-28T18:29:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 --2- 172.21.15.7:0/601954826 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 --2- 172.21.15.7:0/601954826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ca30 0x7f5090159c90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 -- 172.21.15.7:0/601954826 >> 172.21.15.7:0/601954826 conn(0x7f509006c060 msgr2=0x7f50900662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 -- 172.21.15.7:0/601954826 shutdown_connections 2023-11-28T18:29:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 -- 172.21.15.7:0/601954826 wait complete. 2023-11-28T18:29:56.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 Processor -- start 2023-11-28T18:29:56.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.547+0000 7f5095417640 1 -- start start 2023-11-28T18:29:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f509015ca30 0x7f50900c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ce10 0x7f50900c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50900c3200 con 0x7f509015ce10 2023-11-28T18:29:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50901f0080 con 0x7f509015ca30 2023-11-28T18:29:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50901539d0 con 0x7f50900cafe0 2023-11-28T18:29:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5094c16640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ce10 0x7f50900c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5094c16640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ce10 0x7f50900c3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47194/0 (socket says 172.21.15.7:47194) 2023-11-28T18:29:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5094c16640 1 -- 172.21.15.7:0/3105556142 learned_addr learned my addr 172.21.15.7:0/3105556142 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508f7fe640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f509015ca30 0x7f50900c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508ffff640 1 -- 172.21.15.7:0/3105556142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f509015ca30 msgr2=0x7f50900c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508ffff640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f509015ca30 0x7f50900c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508ffff640 1 -- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ce10 msgr2=0x7f50900c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508ffff640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ce10 0x7f50900c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508ffff640 1 -- 172.21.15.7:0/3105556142 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800094d0 con 0x7f50900cafe0 2023-11-28T18:29:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5094c16640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ce10 0x7f50900c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:29:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508ffff640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900c50f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f50800af000 tx=0x7f508001bad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508cff9640 1 -- 172.21.15.7:0/3105556142 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50800be070 con 0x7f50900cafe0 2023-11-28T18:29:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508cff9640 1 -- 172.21.15.7:0/3105556142 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50800ba400 con 0x7f50900cafe0 2023-11-28T18:29:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f508cff9640 1 -- 172.21.15.7:0/3105556142 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50800ba7d0 con 0x7f50900cafe0 2023-11-28T18:29:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50900c2e90 con 0x7f50900cafe0 2023-11-28T18:29:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50901f0340 con 0x7f50900cafe0 2023-11-28T18:29:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.551+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50900c3020 con 0x7f50900cafe0 2023-11-28T18:29:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.555+0000 7f508cff9640 1 -- 172.21.15.7:0/3105556142 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f50800c3410 con 0x7f50900cafe0 2023-11-28T18:29:56.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.555+0000 7f508cff9640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f507c0668d0 0x7f507c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:56.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.555+0000 7f508cff9640 1 -- 172.21.15.7:0/3105556142 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f5080121f20 con 0x7f50900cafe0 2023-11-28T18:29:56.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.555+0000 7f508f7fe640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f507c0668d0 0x7f507c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:56.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.555+0000 7f508f7fe640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f507c0668d0 0x7f507c068d90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50900c2a90 tx=0x7f5084009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:56.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.559+0000 7f508cff9640 1 -- 172.21.15.7:0/3105556142 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f50800ef0e0 con 0x7f50900cafe0 2023-11-28T18:29:56.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.755+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f509004e890 con 0x7f50900cafe0 2023-11-28T18:29:56.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f508cff9640 1 -- 172.21.15.7:0/3105556142 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f50800f2700 con 0x7f50900cafe0 2023-11-28T18:29:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:29:56.762 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:29:56.762 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:29:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f507c0668d0 msgr2=0x7f507c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f507c0668d0 0x7f507c068d90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50900c2a90 tx=0x7f5084009040 comp rx=0 tx=0).stop 2023-11-28T18:29:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 msgr2=0x7f50900c50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900c50f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f50800af000 tx=0x7f508001bad0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 shutdown_connections 2023-11-28T18:29:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f507c0668d0 0x7f507c068d90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f509015ce10 0x7f50900c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f509015ca30 0x7f50900c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 --2- 172.21.15.7:0/3105556142 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50900cafe0 0x7f50900c50f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 >> 172.21.15.7:0/3105556142 conn(0x7f509006c060 msgr2=0x7f50901597f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 shutdown_connections 2023-11-28T18:29:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:56.759+0000 7f5095417640 1 -- 172.21.15.7:0/3105556142 wait complete. 2023-11-28T18:29:56.790 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:29:58.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:29:58.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f006169c640 1 Processor -- start 2023-11-28T18:29:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f006169c640 1 -- start start 2023-11-28T18:29:58.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f006169c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c15ca30 0x7f005c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:58.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f006169c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f005c15ce10 0x7f005c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:58.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f006169c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c15a620 con 0x7f005c0cafe0 2023-11-28T18:29:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f006169c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005c15a900 con 0x7f005c15ce10 2023-11-28T18:29:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f006169c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c15abe0 con 0x7f005c15ca30 2023-11-28T18:29:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f00537fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f005c15ce10 0x7f005c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f00537fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f005c15ce10 0x7f005c0ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47206/0 (socket says 172.21.15.7:47206) 2023-11-28T18:29:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f0060e9b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f005c0cafe0 0x7f005c0cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39286/0 (socket says 172.21.15.7:39286) 2023-11-28T18:29:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f00537fe640 1 -- 172.21.15.7:0/2298974173 learned_addr learned my addr 172.21.15.7:0/2298974173 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f005bfff640 1 --2- 172.21.15.7:0/2298974173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c15ca30 0x7f005c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f00537fe640 1 -- 172.21.15.7:0/2298974173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c15ca30 msgr2=0x7f005c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:29:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f00537fe640 1 --2- 172.21.15.7:0/2298974173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c15ca30 0x7f005c159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f00537fe640 1 -- 172.21.15.7:0/2298974173 >> v1:172.21.15.7:6789/0 conn(0x7f005c0cafe0 legacy=0x7f005c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f00537fe640 1 -- 172.21.15.7:0/2298974173 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c0cbb10 con 0x7f005c15ce10 2023-11-28T18:29:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f005b7fe640 1 -- 172.21.15.7:0/2298974173 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335310819 0 0) 0x7f005c15a620 con 0x7f005c0cafe0 2023-11-28T18:29:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.935+0000 7f005bfff640 1 --2- 172.21.15.7:0/2298974173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c15ca30 0x7f005c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:29:58.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.939+0000 7f00537fe640 1 --2- 172.21.15.7:0/2298974173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ce10 0x7f005c0ca1d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0044009870 tx=0x7f004401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498be015561ad71b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:58.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.939+0000 7f005b7fe640 1 -- 172.21.15.7:0/2298974173 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00440a8a30 con 0x7f005c15ce10 2023-11-28T18:29:58.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.939+0000 7f005b7fe640 1 -- 172.21.15.7:0/2298974173 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00440a8b90 con 0x7f005c15ce10 2023-11-28T18:29:58.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.939+0000 7f005b7fe640 1 -- 172.21.15.7:0/2298974173 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00440b0440 con 0x7f005c15ce10 2023-11-28T18:29:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.939+0000 7f006169c640 1 -- 172.21.15.7:0/2298974173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ce10 msgr2=0x7f005c0ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.939+0000 7f006169c640 1 --2- 172.21.15.7:0/2298974173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ce10 0x7f005c0ca1d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0044009870 tx=0x7f004401ed30 comp rx=0 tx=0).stop 2023-11-28T18:29:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.943+0000 7f006169c640 1 -- 172.21.15.7:0/2298974173 shutdown_connections 2023-11-28T18:29:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.943+0000 7f006169c640 1 --2- 172.21.15.7:0/2298974173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ce10 0x7f005c0ca1d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.943+0000 7f006169c640 1 --2- 172.21.15.7:0/2298974173 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c15ca30 0x7f005c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.943+0000 7f006169c640 1 -- 172.21.15.7:0/2298974173 >> 172.21.15.7:0/2298974173 conn(0x7f005c06c060 msgr2=0x7f005c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.943+0000 7f006169c640 1 -- 172.21.15.7:0/2298974173 shutdown_connections 2023-11-28T18:29:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.943+0000 7f006169c640 1 -- 172.21.15.7:0/2298974173 wait complete. 2023-11-28T18:29:58.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.943+0000 7f006169c640 1 Processor -- start 2023-11-28T18:29:58.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.943+0000 7f006169c640 1 -- start start 2023-11-28T18:29:58.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f006169c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c0cafe0 0x7f005c1f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:58.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f006169c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ca30 0x7f005c150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:58.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f006169c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f005c15ce10 0x7f005c1554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:58.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f006169c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c156de0 con 0x7f005c15ca30 2023-11-28T18:29:58.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f006169c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c1efaa0 con 0x7f005c15ce10 2023-11-28T18:29:58.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f006169c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c154cb0 con 0x7f005c0cafe0 2023-11-28T18:29:58.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ca30 0x7f005c150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:58.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ca30 0x7f005c150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47216/0 (socket says 172.21.15.7:47216) 2023-11-28T18:29:58.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 -- 172.21.15.7:0/3094328714 learned_addr learned my addr 172.21.15.7:0/3094328714 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:29:58.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f005bfff640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c0cafe0 0x7f005c1f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f0060e9b640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f005c15ce10 0x7f005c1554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 -- 172.21.15.7:0/3094328714 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c0cafe0 msgr2=0x7f005c1f04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c0cafe0 0x7f005c1f04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 -- 172.21.15.7:0/3094328714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f005c15ce10 msgr2=0x7f005c1554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f005c15ce10 0x7f005c1554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 -- 172.21.15.7:0/3094328714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00440094d0 con 0x7f005c15ca30 2023-11-28T18:29:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00537fe640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ca30 0x7f005c150110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f00440b3040 tx=0x7f004401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00597fa640 1 -- 172.21.15.7:0/3094328714 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00440be070 con 0x7f005c15ca30 2023-11-28T18:29:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00597fa640 1 -- 172.21.15.7:0/3094328714 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00440ba400 con 0x7f005c15ca30 2023-11-28T18:29:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c1f3f10 con 0x7f005c15ca30 2023-11-28T18:29:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f00597fa640 1 -- 172.21.15.7:0/3094328714 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00440ba7d0 con 0x7f005c15ca30 2023-11-28T18:29:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.947+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c1f09f0 con 0x7f005c15ca30 2023-11-28T18:29:58.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.951+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005c1f01d0 con 0x7f005c15ca30 2023-11-28T18:29:58.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.951+0000 7f00597fa640 1 -- 172.21.15.7:0/3094328714 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f00440c3410 con 0x7f005c15ca30 2023-11-28T18:29:58.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.951+0000 7f00597fa640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f003c0668d0 0x7f003c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:29:58.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.951+0000 7f00597fa640 1 -- 172.21.15.7:0/3094328714 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f0044121fa0 con 0x7f005c15ca30 2023-11-28T18:29:58.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.955+0000 7f005bfff640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f003c0668d0 0x7f003c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:29:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.955+0000 7f005bfff640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f003c0668d0 0x7f003c068d90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f004c003b90 tx=0x7f004c009890 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:29:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:58.955+0000 7f00597fa640 1 -- 172.21.15.7:0/3094328714 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f00440ef150 con 0x7f005c15ca30 2023-11-28T18:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f005c067b50 con 0x7f005c15ca30 2023-11-28T18:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f00597fa640 1 -- 172.21.15.7:0/3094328714 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f00440f2770 con 0x7f005c15ca30 2023-11-28T18:29:59.167 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:29:59.167 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:29:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f003c0668d0 msgr2=0x7f003c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f003c0668d0 0x7f003c068d90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f004c003b90 tx=0x7f004c009890 comp rx=0 tx=0).stop 2023-11-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ca30 msgr2=0x7f005c150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ca30 0x7f005c150110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f00440b3040 tx=0x7f004401bad0 comp rx=0 tx=0).stop 2023-11-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 shutdown_connections 2023-11-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f003c0668d0 0x7f003c068d90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f005c15ce10 0x7f005c1554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f005c15ca30 0x7f005c150110 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:59.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 --2- 172.21.15.7:0/3094328714 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f005c0cafe0 0x7f005c1f04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:29:59.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 >> 172.21.15.7:0/3094328714 conn(0x7f005c06c060 msgr2=0x7f005c15bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:29:59.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 shutdown_connections 2023-11-28T18:29:59.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:29:59.163+0000 7f006169c640 1 -- 172.21.15.7:0/3094328714 wait complete. 2023-11-28T18:29:59.190 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:30:00.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:30:00.024 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:30:01.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:01.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.347+0000 7eff996fe640 1 Processor -- start 2023-11-28T18:30:01.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.347+0000 7eff996fe640 1 -- start start 2023-11-28T18:30:01.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.347+0000 7eff996fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7eff941542e0 0x7eff9415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:01.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.347+0000 7eff996fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff940cbb70 0x7eff940cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:01.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff996fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff940ca920 con 0x7eff940cc810 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff996fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff94071d50 con 0x7eff940cbb70 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff996fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff94072030 con 0x7eff941542e0 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff927fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff940cbb70 0x7eff940cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff927fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff940cbb70 0x7eff940cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47226/0 (socket says 172.21.15.7:47226) 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff927fc640 1 -- 172.21.15.7:0/1068879744 learned_addr learned my addr 172.21.15.7:0/1068879744 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff92ffd640 1 --2- 172.21.15.7:0/1068879744 >> v2:172.21.15.193:3300/0 conn(0x7eff941542e0 0x7eff9415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff927fc640 1 -- 172.21.15.7:0/1068879744 >> v2:172.21.15.193:3300/0 conn(0x7eff941542e0 msgr2=0x7eff9415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff927fc640 1 --2- 172.21.15.7:0/1068879744 >> v2:172.21.15.193:3300/0 conn(0x7eff941542e0 0x7eff9415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff927fc640 1 -- 172.21.15.7:0/1068879744 >> v1:172.21.15.193:6789/0 conn(0x7eff940cc810 legacy=0x7eff94072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff927fc640 1 -- 172.21.15.7:0/1068879744 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9415e550 con 0x7eff940cbb70 2023-11-28T18:30:01.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff92ffd640 1 --2- 172.21.15.7:0/1068879744 >> v2:172.21.15.193:3300/0 conn(0x7eff941542e0 0x7eff9415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:30:01.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.351+0000 7eff927fc640 1 --2- 172.21.15.7:0/1068879744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff940cc2d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7eff94071d20 tx=0x7eff8801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afbbdb57eda0a7a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:01.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff91ffb640 1 -- 172.21.15.7:0/1068879744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff880ab610 con 0x7eff940cbb70 2023-11-28T18:30:01.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff91ffb640 1 -- 172.21.15.7:0/1068879744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff880ab770 con 0x7eff940cbb70 2023-11-28T18:30:01.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff91ffb640 1 -- 172.21.15.7:0/1068879744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff880abb00 con 0x7eff940cbb70 2023-11-28T18:30:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 -- 172.21.15.7:0/1068879744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 msgr2=0x7eff940cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 --2- 172.21.15.7:0/1068879744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff940cc2d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7eff94071d20 tx=0x7eff8801c500 comp rx=0 tx=0).stop 2023-11-28T18:30:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 -- 172.21.15.7:0/1068879744 shutdown_connections 2023-11-28T18:30:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 --2- 172.21.15.7:0/1068879744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff940cc2d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 --2- 172.21.15.7:0/1068879744 >> v2:172.21.15.193:3300/0 conn(0x7eff941542e0 0x7eff9415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 -- 172.21.15.7:0/1068879744 >> 172.21.15.7:0/1068879744 conn(0x7eff940717e0 msgr2=0x7eff940662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 -- 172.21.15.7:0/1068879744 shutdown_connections 2023-11-28T18:30:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 -- 172.21.15.7:0/1068879744 wait complete. 2023-11-28T18:30:01.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 Processor -- start 2023-11-28T18:30:01.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.355+0000 7eff996fe640 1 -- start start 2023-11-28T18:30:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff94153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff940cc810 0x7eff94147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff941542e0 0x7eff941f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9414a880 con 0x7eff940cbb70 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff941492b0 con 0x7eff940cc810 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff941483c0 con 0x7eff941542e0 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff94153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff94153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47240/0 (socket says 172.21.15.7:47240) 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 -- 172.21.15.7:0/184788116 learned_addr learned my addr 172.21.15.7:0/184788116 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff937fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff941542e0 0x7eff941f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff927fc640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff940cc810 0x7eff94147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 -- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff941542e0 msgr2=0x7eff941f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff941542e0 0x7eff941f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 -- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff940cc810 msgr2=0x7eff94147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff940cc810 0x7eff94147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 -- 172.21.15.7:0/184788116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff880094d0 con 0x7eff940cbb70 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff937fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff941542e0 0x7eff941f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff92ffd640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff94153310 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7eff8400e810 tx=0x7eff8400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff7b7fe640 1 -- 172.21.15.7:0/184788116 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400c8d0 con 0x7eff940cbb70 2023-11-28T18:30:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff7b7fe640 1 -- 172.21.15.7:0/184788116 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8400ca30 con 0x7eff940cbb70 2023-11-28T18:30:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff7b7fe640 1 -- 172.21.15.7:0/184788116 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400cdc0 con 0x7eff940cbb70 2023-11-28T18:30:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff941486a0 con 0x7eff940cbb70 2023-11-28T18:30:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff94148c60 con 0x7eff940cbb70 2023-11-28T18:30:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.359+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff9406c670 con 0x7eff940cbb70 2023-11-28T18:30:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.363+0000 7eff7b7fe640 1 -- 172.21.15.7:0/184788116 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7eff84013410 con 0x7eff940cbb70 2023-11-28T18:30:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.363+0000 7eff7b7fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff80066900 0x7eff80068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.363+0000 7eff7b7fe640 1 -- 172.21.15.7:0/184788116 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5997+0+0 (secure 0 0 0) 0x7eff84084260 con 0x7eff940cbb70 2023-11-28T18:30:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.363+0000 7eff927fc640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff80066900 0x7eff80068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.363+0000 7eff927fc640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff80066900 0x7eff80068dc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7eff880af000 tx=0x7eff880b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.367+0000 7eff7b7fe640 1 -- 172.21.15.7:0/184788116 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7eff840514c0 con 0x7eff940cbb70 2023-11-28T18:30:01.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff9404e890 con 0x7eff940cbb70 2023-11-28T18:30:01.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff7b7fe640 1 -- 172.21.15.7:0/184788116 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6549 (secure 0 0 0) 0x7eff84054ae0 con 0x7eff940cbb70 2023-11-28T18:30:01.603 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:01.603 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5086,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:28:38.708410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":4284,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"a","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.7:6835/1419029356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1419029356},{"type":"v1","addr":"172.21.15.7:6835","nonce":1419029356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:01.603 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 35 2023-11-28T18:30:01.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff80066900 msgr2=0x7eff80068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:01.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff80066900 0x7eff80068dc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7eff880af000 tx=0x7eff880b6040 comp rx=0 tx=0).stop 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 msgr2=0x7eff94153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff94153310 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7eff8400e810 tx=0x7eff8400ecd0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 shutdown_connections 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff80066900 0x7eff80068dc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff941542e0 0x7eff941f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff940cc810 0x7eff94147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 --2- 172.21.15.7:0/184788116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff940cbb70 0x7eff94153310 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 >> 172.21.15.7:0/184788116 conn(0x7eff940717e0 msgr2=0x7eff940c8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 shutdown_connections 2023-11-28T18:30:01.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:01.599+0000 7eff996fe640 1 -- 172.21.15.7:0/184788116 wait complete. 2023-11-28T18:30:01.628 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:30:03.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:03.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc73609640 1 Processor -- start 2023-11-28T18:30:03.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc73609640 1 -- start start 2023-11-28T18:30:03.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc73609640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbc6c1540e0 0x7fbc6c0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:03.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc73609640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c15e380 con 0x7fbc6c15a8a0 2023-11-28T18:30:03.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc73609640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c15d7e0 con 0x7fbc6c0cf6c0 2023-11-28T18:30:03.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc73609640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6c0cab80 con 0x7fbc6c1540e0 2023-11-28T18:30:03.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc70ba4640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbc6c15a8a0 0x7fbc6c15dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49874/0 (socket says 172.21.15.7:49874) 2023-11-28T18:30:03.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc70ba4640 1 -- 172.21.15.7:0/1237206690 learned_addr learned my addr 172.21.15.7:0/1237206690 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:03.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.783+0000 7fbc713a5640 1 --2- 172.21.15.7:0/1237206690 >> v2:172.21.15.193:3300/0 conn(0x7fbc6c1540e0 0x7fbc6c0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:03.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.787+0000 7fbc63fff640 1 -- 172.21.15.7:0/1237206690 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365720875 0 0) 0x7fbc5c001120 con 0x7fbc6c15a8a0 2023-11-28T18:30:03.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.787+0000 7fbc713a5640 1 -- 172.21.15.7:0/1237206690 >> v1:172.21.15.193:6789/0 conn(0x7fbc6c0cf6c0 legacy=0x7fbc6c164760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:03.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.787+0000 7fbc713a5640 1 -- 172.21.15.7:0/1237206690 >> v1:172.21.15.7:6789/0 conn(0x7fbc6c15a8a0 legacy=0x7fbc6c15dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:03.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.787+0000 7fbc713a5640 1 -- 172.21.15.7:0/1237206690 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c0cb850 con 0x7fbc6c1540e0 2023-11-28T18:30:03.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.787+0000 7fbc713a5640 1 --2- 172.21.15.7:0/1237206690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0ca4d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009740 tx=0x7fbc6801c2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46eb70637a92b69b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:03.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc63fff640 1 -- 172.21.15.7:0/1237206690 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc680ab600 con 0x7fbc6c1540e0 2023-11-28T18:30:03.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc63fff640 1 -- 172.21.15.7:0/1237206690 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc680a7050 con 0x7fbc6c1540e0 2023-11-28T18:30:03.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc63fff640 1 -- 172.21.15.7:0/1237206690 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc680ab980 con 0x7fbc6c1540e0 2023-11-28T18:30:03.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- 172.21.15.7:0/1237206690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c1540e0 msgr2=0x7fbc6c0ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 --2- 172.21.15.7:0/1237206690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0ca4d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009740 tx=0x7fbc6801c2b0 comp rx=0 tx=0).stop 2023-11-28T18:30:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- 172.21.15.7:0/1237206690 shutdown_connections 2023-11-28T18:30:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 --2- 172.21.15.7:0/1237206690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0ca4d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- 172.21.15.7:0/1237206690 >> 172.21.15.7:0/1237206690 conn(0x7fbc6c0662e0 msgr2=0x7fbc6c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- 172.21.15.7:0/1237206690 shutdown_connections 2023-11-28T18:30:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- 172.21.15.7:0/1237206690 wait complete. 2023-11-28T18:30:03.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 Processor -- start 2023-11-28T18:30:03.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- start start 2023-11-28T18:30:03.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbc6c0cf6c0 0x7fbc6c158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:03.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:03.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c15a8a0 0x7fbc6c0c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:03.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c0c7660 con 0x7fbc6c1540e0 2023-11-28T18:30:03.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c0c7c20 con 0x7fbc6c15a8a0 2023-11-28T18:30:03.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.791+0000 7fbc73609640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c158b40 con 0x7fbc6c0cf6c0 2023-11-28T18:30:03.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:03.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53314/0 (socket says 172.21.15.7:53314) 2023-11-28T18:30:03.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 -- 172.21.15.7:0/1758472779 learned_addr learned my addr 172.21.15.7:0/1758472779 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc713a5640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbc6c0cf6c0 0x7fbc6c158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc71ba6640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c15a8a0 0x7fbc6c0c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 -- 172.21.15.7:0/1758472779 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbc6c0cf6c0 msgr2=0x7fbc6c158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbc6c0cf6c0 0x7fbc6c158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 -- 172.21.15.7:0/1758472779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c15a8a0 msgr2=0x7fbc6c0c80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c15a8a0 0x7fbc6c0c80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 -- 172.21.15.7:0/1758472779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc68009420 con 0x7fbc6c1540e0 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc70ba4640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0b49a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00c7b0 tx=0x7fbc5c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc61ffb640 1 -- 172.21.15.7:0/1758472779 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0072d0 con 0x7fbc6c1540e0 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc61ffb640 1 -- 172.21.15.7:0/1758472779 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c007430 con 0x7fbc6c1540e0 2023-11-28T18:30:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6c0c7920 con 0x7fbc6c1540e0 2023-11-28T18:30:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc61ffb640 1 -- 172.21.15.7:0/1758472779 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c001d20 con 0x7fbc6c1540e0 2023-11-28T18:30:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6c159890 con 0x7fbc6c1540e0 2023-11-28T18:30:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6c04e890 con 0x7fbc6c1540e0 2023-11-28T18:30:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.795+0000 7fbc61ffb640 1 -- 172.21.15.7:0/1758472779 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbc5c0047f0 con 0x7fbc6c1540e0 2023-11-28T18:30:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.799+0000 7fbc61ffb640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbc500668d0 0x7fbc50068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.799+0000 7fbc61ffb640 1 -- 172.21.15.7:0/1758472779 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fbc5c0838d0 con 0x7fbc6c1540e0 2023-11-28T18:30:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.799+0000 7fbc713a5640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbc500668d0 0x7fbc50068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.799+0000 7fbc713a5640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbc500668d0 0x7fbc50068d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbc680ad000 tx=0x7fbc680a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:03.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:03.803+0000 7fbc61ffb640 1 -- 172.21.15.7:0/1758472779 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbc5c050ad0 con 0x7fbc6c1540e0 2023-11-28T18:30:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.023+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc6c15e380 con 0x7fbc6c1540e0 2023-11-28T18:30:04.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.023+0000 7fbc61ffb640 1 -- 172.21.15.7:0/1758472779 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5762 (secure 0 0 0) 0x7fbc6c15e380 con 0x7fbc6c1540e0 2023-11-28T18:30:04.028 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:04.028 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:02.197526+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:04.028 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 36 2023-11-28T18:30:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbc500668d0 msgr2=0x7fbc50068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbc500668d0 0x7fbc50068d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbc680ad000 tx=0x7fbc680a6000 comp rx=0 tx=0).stop 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6c1540e0 msgr2=0x7fbc6c0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0b49a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00c7b0 tx=0x7fbc5c00cc70 comp rx=0 tx=0).stop 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 shutdown_connections 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbc500668d0 0x7fbc50068d90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc6c15a8a0 0x7fbc6c0c80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6c1540e0 0x7fbc6c0b49a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 --2- 172.21.15.7:0/1758472779 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbc6c0cf6c0 0x7fbc6c158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 >> 172.21.15.7:0/1758472779 conn(0x7fbc6c0662e0 msgr2=0x7fbc6c146db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 shutdown_connections 2023-11-28T18:30:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.027+0000 7fbc73609640 1 -- 172.21.15.7:0/1758472779 wait complete. 2023-11-28T18:30:04.055 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-11-28T18:30:04.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:30:04.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9dd03640 1 Processor -- start 2023-11-28T18:30:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9dd03640 1 -- start start 2023-11-28T18:30:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9dd03640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3c9815ce10 0x7f3c980ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9dd03640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3c980cafe0 0x7f3c980ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9dd03640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9815a3a0 con 0x7f3c9815ca30 2023-11-28T18:30:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9dd03640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9815a620 con 0x7f3c980cafe0 2023-11-28T18:30:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9dd03640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9815a900 con 0x7f3c9815ce10 2023-11-28T18:30:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9d502640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3c980cafe0 0x7f3c980ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9d502640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3c980cafe0 0x7f3c980ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53320/0 (socket says 172.21.15.7:53320) 2023-11-28T18:30:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9d502640 1 -- 172.21.15.7:0/2140828413 learned_addr learned my addr 172.21.15.7:0/2140828413 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c97fff640 1 --2- 172.21.15.7:0/2140828413 >> v2:172.21.15.193:3300/0 conn(0x7f3c9815ce10 0x7f3c980ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9d502640 1 -- 172.21.15.7:0/2140828413 >> v2:172.21.15.193:3300/0 conn(0x7f3c9815ce10 msgr2=0x7f3c980ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9d502640 1 --2- 172.21.15.7:0/2140828413 >> v2:172.21.15.193:3300/0 conn(0x7f3c9815ce10 0x7f3c980ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9d502640 1 -- 172.21.15.7:0/2140828413 >> v1:172.21.15.193:6789/0 conn(0x7f3c9815ca30 legacy=0x7f3c98159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.219+0000 7f3c9d502640 1 -- 172.21.15.7:0/2140828413 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9815abe0 con 0x7f3c980cafe0 2023-11-28T18:30:04.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.223+0000 7f3c9d502640 1 --2- 172.21.15.7:0/2140828413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c980cafe0 0x7f3c980ca710 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3c980d0200 tx=0x7f3c9001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca48c9e9b887f98 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:04.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.223+0000 7f3c977fe640 1 -- 172.21.15.7:0/2140828413 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c900a5860 con 0x7f3c980cafe0 2023-11-28T18:30:04.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.223+0000 7f3c977fe640 1 -- 172.21.15.7:0/2140828413 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c900a59c0 con 0x7f3c980cafe0 2023-11-28T18:30:04.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.223+0000 7f3c977fe640 1 -- 172.21.15.7:0/2140828413 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c900a5d50 con 0x7f3c980cafe0 2023-11-28T18:30:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- 172.21.15.7:0/2140828413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c980cafe0 msgr2=0x7f3c980ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/2140828413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c980cafe0 0x7f3c980ca710 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3c980d0200 tx=0x7f3c9001c9a0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- 172.21.15.7:0/2140828413 shutdown_connections 2023-11-28T18:30:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/2140828413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c980cafe0 0x7f3c980ca710 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/2140828413 >> v2:172.21.15.193:3300/0 conn(0x7f3c9815ce10 0x7f3c980ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- 172.21.15.7:0/2140828413 >> 172.21.15.7:0/2140828413 conn(0x7f3c9806c060 msgr2=0x7f3c980662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- 172.21.15.7:0/2140828413 shutdown_connections 2023-11-28T18:30:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- 172.21.15.7:0/2140828413 wait complete. 2023-11-28T18:30:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 Processor -- start 2023-11-28T18:30:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- start start 2023-11-28T18:30:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c980cafe0 0x7f3c98152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9815ca30 0x7f3c981f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c9815ce10 0x7f3c981f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98154980 con 0x7f3c9815ca30 2023-11-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98155d80 con 0x7f3c980cafe0 2023-11-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9dd03640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c981554d0 con 0x7f3c9815ce10 2023-11-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9cd01640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c980cafe0 0x7f3c98152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.227+0000 7f3c9cd01640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c980cafe0 0x7f3c98152f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:44240/0 (socket says 172.21.15.7:44240) 2023-11-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9cd01640 1 -- 172.21.15.7:0/3226230609 learned_addr learned my addr 172.21.15.7:0/3226230609 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9d502640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c9815ce10 0x7f3c981f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c97fff640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9815ca30 0x7f3c981f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9d502640 1 -- 172.21.15.7:0/3226230609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c980cafe0 msgr2=0x7f3c98152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9d502640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c980cafe0 0x7f3c98152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9d502640 1 -- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9815ca30 msgr2=0x7f3c981f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9d502640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9815ca30 0x7f3c981f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9d502640 1 -- 172.21.15.7:0/3226230609 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90009b30 con 0x7f3c9815ce10 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9d502640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c9815ce10 0x7f3c981f3f10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c9815a550 tx=0x7f3c90019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c957fa640 1 -- 172.21.15.7:0/3226230609 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c900bd070 con 0x7f3c9815ce10 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c957fa640 1 -- 172.21.15.7:0/3226230609 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c900b8470 con 0x7f3c9815ce10 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c957fa640 1 -- 172.21.15.7:0/3226230609 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c900b8840 con 0x7f3c9815ce10 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c98150ff0 con 0x7f3c9815ce10 2023-11-28T18:30:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c981546a0 con 0x7f3c9815ce10 2023-11-28T18:30:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.231+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9804e890 con 0x7f3c9815ce10 2023-11-28T18:30:04.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.235+0000 7f3c957fa640 1 -- 172.21.15.7:0/3226230609 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3c980cce00 con 0x7f3c9815ce10 2023-11-28T18:30:04.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.235+0000 7f3c957fa640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c7c0668d0 0x7f3c7c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:04.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.235+0000 7f3c957fa640 1 -- 172.21.15.7:0/3226230609 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f3c9011ef80 con 0x7f3c9815ce10 2023-11-28T18:30:04.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.235+0000 7f3c9cd01640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c7c0668d0 0x7f3c7c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:04.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.235+0000 7f3c9cd01640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c7c0668d0 0x7f3c7c068d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0018b0 tx=0x7f3c8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:04.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.239+0000 7f3c957fa640 1 -- 172.21.15.7:0/3226230609 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3c900ec170 con 0x7f3c9815ce10 2023-11-28T18:30:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.491+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c980cad10 con 0x7f3c9815ce10 2023-11-28T18:30:04.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.491+0000 7f3c957fa640 1 -- 172.21.15.7:0/3226230609 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5762 (secure 0 0 0) 0x7f3c900ef790 con 0x7f3c9815ce10 2023-11-28T18:30:04.495 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:04.496 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:02.197526+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 36 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.491+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c7c0668d0 msgr2=0x7f3c7c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.491+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c7c0668d0 0x7f3c7c068d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0018b0 tx=0x7f3c8c009040 comp rx=0 tx=0).stop 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.491+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c9815ce10 msgr2=0x7f3c981f3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.491+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c9815ce10 0x7f3c981f3f10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c9815a550 tx=0x7f3c90019740 comp rx=0 tx=0).stop 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.495+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 shutdown_connections 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.495+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3c7c0668d0 0x7f3c7c068d90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.495+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c9815ce10 0x7f3c981f3f10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.495+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9815ca30 0x7f3c981f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.495+0000 7f3c9dd03640 1 --2- 172.21.15.7:0/3226230609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c980cafe0 0x7f3c98152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.495+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 >> 172.21.15.7:0/3226230609 conn(0x7f3c9806c060 msgr2=0x7f3c98158170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.495+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 shutdown_connections 2023-11-28T18:30:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:04.495+0000 7f3c9dd03640 1 -- 172.21.15.7:0/3226230609 wait complete. 2023-11-28T18:30:04.520 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 36, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4285": { "addr": "172.21.15.193:6837/3911527535", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 3911527535, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 3911527535, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 11, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.7:6838/2690556675", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 2690556675, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 2690556675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:30:02.197526+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 4288, "mds_2": 4285, "mds_3": 5086, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:30:06.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:06.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.675+0000 7ff411d2a640 1 Processor -- start 2023-11-28T18:30:06.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.675+0000 7ff411d2a640 1 -- start start 2023-11-28T18:30:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff411d2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c0cbb70 0x7ff40c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff411d2a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff40c0cc810 0x7ff40c072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff411d2a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40c0ca920 con 0x7ff40c1542e0 2023-11-28T18:30:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff411d2a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40c071d50 con 0x7ff40c0cc810 2023-11-28T18:30:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff411d2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c072030 con 0x7ff40c0cbb70 2023-11-28T18:30:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff40c1542e0 0x7ff40c15e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49884/0 (socket says 172.21.15.7:49884) 2023-11-28T18:30:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40b7fe640 1 -- 172.21.15.7:0/39554867 learned_addr learned my addr 172.21.15.7:0/39554867 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40bfff640 1 --2- 172.21.15.7:0/39554867 >> v2:172.21.15.193:3300/0 conn(0x7ff40c0cc810 0x7ff40c072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40affd640 1 --2- 172.21.15.7:0/39554867 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c0cbb70 0x7ff40c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40a7fc640 1 -- 172.21.15.7:0/39554867 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318217537 0 0) 0x7ff40c0ca920 con 0x7ff40c1542e0 2023-11-28T18:30:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40a7fc640 1 -- 172.21.15.7:0/39554867 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f4003530 con 0x7ff40c1542e0 2023-11-28T18:30:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40bfff640 1 -- 172.21.15.7:0/39554867 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c0cbb70 msgr2=0x7ff40c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40bfff640 1 --2- 172.21.15.7:0/39554867 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c0cbb70 0x7ff40c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40bfff640 1 -- 172.21.15.7:0/39554867 >> v1:172.21.15.7:6789/0 conn(0x7ff40c1542e0 legacy=0x7ff40c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40a7fc640 1 -- 172.21.15.7:0/39554867 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2326277501 0 0) 0x7ff3f4003530 con 0x7ff40c1542e0 2023-11-28T18:30:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.679+0000 7ff40bfff640 1 -- 172.21.15.7:0/39554867 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c0c4cd0 con 0x7ff40c0cc810 2023-11-28T18:30:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff40bfff640 1 --2- 172.21.15.7:0/39554867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 0x7ff40c072450 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff404009d90 tx=0x7ff40401c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=705aeb7680c76f3f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:06.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff40a7fc640 1 -- 172.21.15.7:0/39554867 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc001680 con 0x7ff40c0cc810 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff40a7fc640 1 -- 172.21.15.7:0/39554867 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4040a5580 con 0x7ff40c0cc810 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff40a7fc640 1 -- 172.21.15.7:0/39554867 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040a5910 con 0x7ff40c0cc810 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff411d2a640 1 -- 172.21.15.7:0/39554867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 msgr2=0x7ff40c072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff411d2a640 1 --2- 172.21.15.7:0/39554867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 0x7ff40c072450 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff404009d90 tx=0x7ff40401c870 comp rx=0 tx=0).stop 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff411d2a640 1 -- 172.21.15.7:0/39554867 shutdown_connections 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff411d2a640 1 --2- 172.21.15.7:0/39554867 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c0cbb70 0x7ff40c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff411d2a640 1 --2- 172.21.15.7:0/39554867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 0x7ff40c072450 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff411d2a640 1 -- 172.21.15.7:0/39554867 >> 172.21.15.7:0/39554867 conn(0x7ff40c0717e0 msgr2=0x7ff40c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff411d2a640 1 -- 172.21.15.7:0/39554867 shutdown_connections 2023-11-28T18:30:06.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.683+0000 7ff411d2a640 1 -- 172.21.15.7:0/39554867 wait complete. 2023-11-28T18:30:06.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 Processor -- start 2023-11-28T18:30:06.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 -- start start 2023-11-28T18:30:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff40c0cbb70 0x7ff40c1537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 0x7ff40c14ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c1542e0 0x7ff40c144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c1f6b90 con 0x7ff40c0cbb70 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c1f2670 con 0x7ff40c0cc810 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c1f68b0 con 0x7ff40c1542e0 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff40c0cbb70 0x7ff40c1537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff40c0cbb70 0x7ff40c1537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53340/0 (socket says 172.21.15.7:53340) 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 -- 172.21.15.7:0/626682324 learned_addr learned my addr 172.21.15.7:0/626682324 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40affd640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 0x7ff40c14ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40bfff640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c1542e0 0x7ff40c144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 -- 172.21.15.7:0/626682324 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c1542e0 msgr2=0x7ff40c144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c1542e0 0x7ff40c144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 -- 172.21.15.7:0/626682324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 msgr2=0x7ff40c14ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 0x7ff40c14ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 -- 172.21.15.7:0/626682324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff404009a70 con 0x7ff40c0cbb70 2023-11-28T18:30:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff40b7fe640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff40c0cbb70 0x7ff40c1537f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00c750 tx=0x7ff3fc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff3f3fff640 1 -- 172.21.15.7:0/626682324 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc005ae0 con 0x7ff40c0cbb70 2023-11-28T18:30:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff3f3fff640 1 -- 172.21.15.7:0/626682324 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc005c40 con 0x7ff40c0cbb70 2023-11-28T18:30:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff3f3fff640 1 -- 172.21.15.7:0/626682324 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc007070 con 0x7ff40c0cbb70 2023-11-28T18:30:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40c1f2880 con 0x7ff40c0cbb70 2023-11-28T18:30:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.687+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40c1f2e40 con 0x7ff40c0cbb70 2023-11-28T18:30:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.691+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40c06c670 con 0x7ff40c0cbb70 2023-11-28T18:30:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.691+0000 7ff3f3fff640 1 -- 172.21.15.7:0/626682324 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff3fc0071f0 con 0x7ff40c0cbb70 2023-11-28T18:30:06.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.691+0000 7ff3f3fff640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3f8066900 0x7ff3f8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:06.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.691+0000 7ff3f3fff640 1 -- 172.21.15.7:0/626682324 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7ff3fc0842e0 con 0x7ff40c0cbb70 2023-11-28T18:30:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.691+0000 7ff40affd640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3f8066900 0x7ff3f8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.695+0000 7ff40affd640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3f8066900 0x7ff3f8068dc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0cf2a0 tx=0x7ff400009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.695+0000 7ff3f3fff640 1 -- 172.21.15.7:0/626682324 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff3fc0513b0 con 0x7ff40c0cbb70 2023-11-28T18:30:06.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.915+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff40c04e890 con 0x7ff40c0cbb70 2023-11-28T18:30:06.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.915+0000 7ff3f3fff640 1 -- 172.21.15.7:0/626682324 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5762 (secure 0 0 0) 0x7ff3fc0549d0 con 0x7ff40c0cbb70 2023-11-28T18:30:06.921 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:06.922 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:02.197526+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 36 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.915+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3f8066900 msgr2=0x7ff3f8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.915+0000 7ff411d2a640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3f8066900 0x7ff3f8068dc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0cf2a0 tx=0x7ff400009190 comp rx=0 tx=0).stop 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.915+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff40c0cbb70 msgr2=0x7ff40c1537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.915+0000 7ff411d2a640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff40c0cbb70 0x7ff40c1537f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00c750 tx=0x7ff3fc00cc10 comp rx=0 tx=0).stop 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.915+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 shutdown_connections 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.915+0000 7ff411d2a640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3f8066900 0x7ff3f8068dc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.919+0000 7ff411d2a640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff40c1542e0 0x7ff40c144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.919+0000 7ff411d2a640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff40c0cc810 0x7ff40c14ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.919+0000 7ff411d2a640 1 --2- 172.21.15.7:0/626682324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff40c0cbb70 0x7ff40c1537f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.919+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 >> 172.21.15.7:0/626682324 conn(0x7ff40c0717e0 msgr2=0x7ff40c157eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:06.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.919+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 shutdown_connections 2023-11-28T18:30:06.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:06.919+0000 7ff411d2a640 1 -- 172.21.15.7:0/626682324 wait complete. 2023-11-28T18:30:06.945 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:08.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:09.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f36591b8640 1 Processor -- start 2023-11-28T18:30:09.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f36591b8640 1 -- start start 2023-11-28T18:30:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f36591b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 0x7f36540ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f36591b8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f365415a8a0 0x7f365415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f36591b8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365415d7e0 con 0x7f36540cf6c0 2023-11-28T18:30:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f36591b8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36540cab80 con 0x7f365415a8a0 2023-11-28T18:30:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f36591b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36540cb850 con 0x7f36541540e0 2023-11-28T18:30:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f365259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f365415a8a0 0x7f365415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f365259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f365415a8a0 0x7f365415dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53354/0 (socket says 172.21.15.7:53354) 2023-11-28T18:30:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f365259c640 1 -- 172.21.15.7:0/344525450 learned_addr learned my addr 172.21.15.7:0/344525450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.111+0000 7f3652d9d640 1 --2- 172.21.15.7:0/344525450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 0x7f36540ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.115+0000 7f365259c640 1 -- 172.21.15.7:0/344525450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 msgr2=0x7f36540ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.115+0000 7f365259c640 1 --2- 172.21.15.7:0/344525450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 0x7f36540ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.115+0000 7f365259c640 1 -- 172.21.15.7:0/344525450 >> v1:172.21.15.193:6789/0 conn(0x7f36540cf6c0 legacy=0x7f36540cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:09.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.115+0000 7f365259c640 1 -- 172.21.15.7:0/344525450 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36540cbb30 con 0x7f365415a8a0 2023-11-28T18:30:09.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f365259c640 1 --2- 172.21.15.7:0/344525450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f365415dc70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3648009870 tx=0x7f364801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42505a7f498a7054 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:09.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f3651d9b640 1 -- 172.21.15.7:0/344525450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480ab650 con 0x7f365415a8a0 2023-11-28T18:30:09.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f3651d9b640 1 -- 172.21.15.7:0/344525450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36480ab7b0 con 0x7f365415a8a0 2023-11-28T18:30:09.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f3651d9b640 1 -- 172.21.15.7:0/344525450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480abb40 con 0x7f365415a8a0 2023-11-28T18:30:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 -- 172.21.15.7:0/344525450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 msgr2=0x7f365415dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 --2- 172.21.15.7:0/344525450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f365415dc70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3648009870 tx=0x7f364801c540 comp rx=0 tx=0).stop 2023-11-28T18:30:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 -- 172.21.15.7:0/344525450 shutdown_connections 2023-11-28T18:30:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 --2- 172.21.15.7:0/344525450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f365415dc70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 --2- 172.21.15.7:0/344525450 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 0x7f36540ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 -- 172.21.15.7:0/344525450 >> 172.21.15.7:0/344525450 conn(0x7f36540662e0 msgr2=0x7f36540666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 -- 172.21.15.7:0/344525450 shutdown_connections 2023-11-28T18:30:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 -- 172.21.15.7:0/344525450 wait complete. 2023-11-28T18:30:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 Processor -- start 2023-11-28T18:30:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.119+0000 7f36591b8640 1 -- start start 2023-11-28T18:30:09.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f36591b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36540cf6c0 0x7f365415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:09.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f36591b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 0x7f36541596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:09.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f36591b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f36541f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:09.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f36591b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3654159d00 con 0x7f365415a8a0 2023-11-28T18:30:09.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f36591b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3654158b20 con 0x7f36540cf6c0 2023-11-28T18:30:09.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f36591b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3654158e00 con 0x7f36541540e0 2023-11-28T18:30:09.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f36541f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:09.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365259c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 0x7f36541596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:09.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f36541f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53360/0 (socket says 172.21.15.7:53360) 2023-11-28T18:30:09.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 -- 172.21.15.7:0/2352752150 learned_addr learned my addr 172.21.15.7:0/2352752150 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:09.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f3652d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36540cf6c0 0x7f365415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:09.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 -- 172.21.15.7:0/2352752150 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 msgr2=0x7f36541596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:09.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 0x7f36541596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 -- 172.21.15.7:0/2352752150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36540cf6c0 msgr2=0x7f365415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:09.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36540cf6c0 0x7f365415ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 -- 172.21.15.7:0/2352752150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36480094d0 con 0x7f365415a8a0 2023-11-28T18:30:09.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f365359e640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f36541f2950 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f364c00c770 tx=0x7f364c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:09.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f363b7fe640 1 -- 172.21.15.7:0/2352752150 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c003b80 con 0x7f365415a8a0 2023-11-28T18:30:09.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f363b7fe640 1 -- 172.21.15.7:0/2352752150 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364c003ce0 con 0x7f365415a8a0 2023-11-28T18:30:09.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f363b7fe640 1 -- 172.21.15.7:0/2352752150 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c00aa90 con 0x7f365415a8a0 2023-11-28T18:30:09.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36540cea50 con 0x7f365415a8a0 2023-11-28T18:30:09.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.123+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36541590e0 con 0x7f365415a8a0 2023-11-28T18:30:09.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.127+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365404e890 con 0x7f365415a8a0 2023-11-28T18:30:09.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.127+0000 7f363b7fe640 1 -- 172.21.15.7:0/2352752150 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f364c00abf0 con 0x7f365415a8a0 2023-11-28T18:30:09.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.127+0000 7f363b7fe640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36400668d0 0x7f3640068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:09.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.127+0000 7f363b7fe640 1 -- 172.21.15.7:0/2352752150 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f364c0842b0 con 0x7f365415a8a0 2023-11-28T18:30:09.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.127+0000 7f3652d9d640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36400668d0 0x7f3640068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:09.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.127+0000 7f3652d9d640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36400668d0 0x7f3640068d90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3644006aa0 tx=0x7f3644008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.131+0000 7f363b7fe640 1 -- 172.21.15.7:0/2352752150 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f364c051420 con 0x7f365415a8a0 2023-11-28T18:30:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.335+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36540cef00 con 0x7f365415a8a0 2023-11-28T18:30:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.335+0000 7f363b7fe640 1 -- 172.21.15.7:0/2352752150 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5764 (secure 0 0 0) 0x7f364c054a40 con 0x7f365415a8a0 2023-11-28T18:30:09.340 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:09.340 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:07.218717+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:resolve","state_seq":23,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 37 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36400668d0 msgr2=0x7f3640068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36400668d0 0x7f3640068d90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3644006aa0 tx=0x7f3644008040 comp rx=0 tx=0).stop 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 msgr2=0x7f36541f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f36541f2950 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f364c00c770 tx=0x7f364c00cc30 comp rx=0 tx=0).stop 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 shutdown_connections 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36400668d0 0x7f3640068d90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f365415a8a0 0x7f36541f2950 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36541540e0 0x7f36541596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 --2- 172.21.15.7:0/2352752150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36540cf6c0 0x7f365415ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:09.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 >> 172.21.15.7:0/2352752150 conn(0x7f36540662e0 msgr2=0x7f365414ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:09.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 shutdown_connections 2023-11-28T18:30:09.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:09.339+0000 7f36591b8640 1 -- 172.21.15.7:0/2352752150 wait complete. 2023-11-28T18:30:09.366 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:11.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.527+0000 7f9572cd6640 1 Processor -- start 2023-11-28T18:30:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.527+0000 7f9572cd6640 1 -- start start 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9572cd6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 0x7f956c159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9572cd6640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f956c15dc50 0x7f956c0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9572cd6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f956c15e090 0x7f956c0cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9572cd6640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956c15d490 con 0x7f956c15e090 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9572cd6640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956c15a480 con 0x7f956c15dc50 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9572cd6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c15a760 con 0x7f956c154300 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f956c15e090 0x7f956c0cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f956c15e090 0x7f956c0cb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53376/0 (socket says 172.21.15.7:53376) 2023-11-28T18:30:11.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 -- 172.21.15.7:0/1022364910 learned_addr learned my addr 172.21.15.7:0/1022364910 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:11.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9570a72640 1 --2- 172.21.15.7:0/1022364910 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 0x7f956c159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:11.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f956bfff640 1 --2- 172.21.15.7:0/1022364910 >> v2:172.21.15.193:3300/0 conn(0x7f956c15dc50 0x7f956c0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:11.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 -- 172.21.15.7:0/1022364910 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 msgr2=0x7f956c159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:11.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 --2- 172.21.15.7:0/1022364910 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 0x7f956c159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 -- 172.21.15.7:0/1022364910 >> v2:172.21.15.193:3300/0 conn(0x7f956c15dc50 msgr2=0x7f956c0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:11.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 --2- 172.21.15.7:0/1022364910 >> v2:172.21.15.193:3300/0 conn(0x7f956c15dc50 0x7f956c0cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 -- 172.21.15.7:0/1022364910 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c15aa40 con 0x7f956c15e090 2023-11-28T18:30:11.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.531+0000 7f9571273640 1 --2- 172.21.15.7:0/1022364910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15e090 0x7f956c0cb6b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f956c0747d0 tx=0x7f956401d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7820de3f8bc908 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:11.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f956b7fe640 1 -- 172.21.15.7:0/1022364910 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95640a5840 con 0x7f956c15e090 2023-11-28T18:30:11.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f956b7fe640 1 -- 172.21.15.7:0/1022364910 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95640a59a0 con 0x7f956c15e090 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f956b7fe640 1 -- 172.21.15.7:0/1022364910 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95640a5d30 con 0x7f956c15e090 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- 172.21.15.7:0/1022364910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15e090 msgr2=0x7f956c0cb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 --2- 172.21.15.7:0/1022364910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15e090 0x7f956c0cb6b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f956c0747d0 tx=0x7f956401d400 comp rx=0 tx=0).stop 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- 172.21.15.7:0/1022364910 shutdown_connections 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 --2- 172.21.15.7:0/1022364910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15e090 0x7f956c0cb6b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 --2- 172.21.15.7:0/1022364910 >> v2:172.21.15.193:3300/0 conn(0x7f956c15dc50 0x7f956c0cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 --2- 172.21.15.7:0/1022364910 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 0x7f956c159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- 172.21.15.7:0/1022364910 >> 172.21.15.7:0/1022364910 conn(0x7f956c0662e0 msgr2=0x7f956c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- 172.21.15.7:0/1022364910 shutdown_connections 2023-11-28T18:30:11.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- 172.21.15.7:0/1022364910 wait complete. 2023-11-28T18:30:11.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 Processor -- start 2023-11-28T18:30:11.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- start start 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 0x7f956c1f2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15dc50 0x7f956c15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c15e090 0x7f956c15b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c146d10 con 0x7f956c15dc50 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c0b5080 con 0x7f956c15e090 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.535+0000 7f9572cd6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c150590 con 0x7f956c154300 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15dc50 0x7f956c15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15dc50 0x7f956c15b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53380/0 (socket says 172.21.15.7:53380) 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 -- 172.21.15.7:0/4101237655 learned_addr learned my addr 172.21.15.7:0/4101237655 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f9571273640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c15e090 0x7f956c15b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f9570a72640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 0x7f956c1f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 -- 172.21.15.7:0/4101237655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 msgr2=0x7f956c1f2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 0x7f956c1f2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 -- 172.21.15.7:0/4101237655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c15e090 msgr2=0x7f956c15b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c15e090 0x7f956c15b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 -- 172.21.15.7:0/4101237655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564009b30 con 0x7f956c15dc50 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f956bfff640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15dc50 0x7f956c15b220 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f955800d7d0 tx=0x7f955800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f95697fa640 1 -- 172.21.15.7:0/4101237655 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9558018070 con 0x7f956c15dc50 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f95697fa640 1 -- 172.21.15.7:0/4101237655 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558004020 con 0x7f956c15dc50 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f95697fa640 1 -- 172.21.15.7:0/4101237655 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95580041d0 con 0x7f956c15dc50 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c147200 con 0x7f956c15dc50 2023-11-28T18:30:11.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c146f20 con 0x7f956c15dc50 2023-11-28T18:30:11.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.539+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c04e890 con 0x7f956c15dc50 2023-11-28T18:30:11.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.543+0000 7f95697fa640 1 -- 172.21.15.7:0/4101237655 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9558004330 con 0x7f956c15dc50 2023-11-28T18:30:11.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.543+0000 7f95697fa640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f95500668d0 0x7f9550068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:11.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.543+0000 7f95697fa640 1 -- 172.21.15.7:0/4101237655 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f95580843a0 con 0x7f956c15dc50 2023-11-28T18:30:11.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.543+0000 7f9570a72640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f95500668d0 0x7f9550068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:11.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.543+0000 7f9570a72640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f95500668d0 0x7f9550068d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f95600045f0 tx=0x7f9560008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:11.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.547+0000 7f95697fa640 1 -- 172.21.15.7:0/4101237655 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9558051480 con 0x7f956c15dc50 2023-11-28T18:30:11.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.775+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f956c0caf50 con 0x7f956c15dc50 2023-11-28T18:30:11.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.775+0000 7f95697fa640 1 -- 172.21.15.7:0/4101237655 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5764 (secure 0 0 0) 0x7f9558054aa0 con 0x7f956c15dc50 2023-11-28T18:30:11.782 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:07.218717+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:resolve","state_seq":23,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:11.782 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 37 2023-11-28T18:30:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f95500668d0 msgr2=0x7f9550068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f95500668d0 0x7f9550068d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f95600045f0 tx=0x7f9560008040 comp rx=0 tx=0).stop 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15dc50 msgr2=0x7f956c15b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15dc50 0x7f956c15b220 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f955800d7d0 tx=0x7f955800dc90 comp rx=0 tx=0).stop 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 shutdown_connections 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f95500668d0 0x7f9550068d90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c15e090 0x7f956c15b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956c15dc50 0x7f956c15b220 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 --2- 172.21.15.7:0/4101237655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f956c154300 0x7f956c1f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 >> 172.21.15.7:0/4101237655 conn(0x7f956c0662e0 msgr2=0x7f956c0cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 shutdown_connections 2023-11-28T18:30:11.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:11.779+0000 7f9572cd6640 1 -- 172.21.15.7:0/4101237655 wait complete. 2023-11-28T18:30:11.807 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:13.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:13.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.955+0000 7f186b49c640 1 Processor -- start 2023-11-28T18:30:13.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.955+0000 7f186b49c640 1 -- start start 2023-11-28T18:30:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186b49c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f186415ce10 0x7f18640ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186b49c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18640cafe0 0x7f18640ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186b49c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186415a3a0 con 0x7f186415ca30 2023-11-28T18:30:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186b49c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186415a620 con 0x7f18640cafe0 2023-11-28T18:30:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186b49c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186415a900 con 0x7f186415ce10 2023-11-28T18:30:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186a49a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f186415ca30 0x7f1864159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53700/0 (socket says 172.21.15.7:53700) 2023-11-28T18:30:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186a49a640 1 -- 172.21.15.7:0/929045562 learned_addr learned my addr 172.21.15.7:0/929045562 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:13.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186ac9b640 1 --2- 172.21.15.7:0/929045562 >> v2:172.21.15.7:3300/0 conn(0x7f18640cafe0 0x7f18640ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:13.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f1869c99640 1 --2- 172.21.15.7:0/929045562 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f186415ce10 0x7f18640ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:13.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f1869498640 1 -- 172.21.15.7:0/929045562 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762093153 0 0) 0x7f186415a3a0 con 0x7f186415ca30 2023-11-28T18:30:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f1869498640 1 -- 172.21.15.7:0/929045562 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1850003530 con 0x7f186415ca30 2023-11-28T18:30:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186ac9b640 1 -- 172.21.15.7:0/929045562 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f186415ce10 msgr2=0x7f18640ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186ac9b640 1 --2- 172.21.15.7:0/929045562 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f186415ce10 0x7f18640ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186ac9b640 1 -- 172.21.15.7:0/929045562 >> v1:172.21.15.7:6789/0 conn(0x7f186415ca30 legacy=0x7f1864159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.959+0000 7f186ac9b640 1 -- 172.21.15.7:0/929045562 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186415abe0 con 0x7f18640cafe0 2023-11-28T18:30:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186ac9b640 1 --2- 172.21.15.7:0/929045562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18640cafe0 0x7f18640ca710 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f185c009d90 tx=0x7f185c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7361aa07d43efce9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f1869498640 1 -- 172.21.15.7:0/929045562 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0af600 con 0x7f18640cafe0 2023-11-28T18:30:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f1869498640 1 -- 172.21.15.7:0/929045562 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c0ab050 con 0x7f18640cafe0 2023-11-28T18:30:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f1869498640 1 -- 172.21.15.7:0/929045562 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0af980 con 0x7f18640cafe0 2023-11-28T18:30:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 -- 172.21.15.7:0/929045562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18640cafe0 msgr2=0x7f18640ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 --2- 172.21.15.7:0/929045562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18640cafe0 0x7f18640ca710 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f185c009d90 tx=0x7f185c01f130 comp rx=0 tx=0).stop 2023-11-28T18:30:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 -- 172.21.15.7:0/929045562 shutdown_connections 2023-11-28T18:30:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 --2- 172.21.15.7:0/929045562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18640cafe0 0x7f18640ca710 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 --2- 172.21.15.7:0/929045562 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f186415ce10 0x7f18640ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 -- 172.21.15.7:0/929045562 >> 172.21.15.7:0/929045562 conn(0x7f186406c060 msgr2=0x7f18640662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 -- 172.21.15.7:0/929045562 shutdown_connections 2023-11-28T18:30:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 -- 172.21.15.7:0/929045562 wait complete. 2023-11-28T18:30:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 Processor -- start 2023-11-28T18:30:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.963+0000 7f186b49c640 1 -- start start 2023-11-28T18:30:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18640cafe0 0x7f18640c3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f186415ca30 0x7f18640c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f186415ce10 0x7f18640c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640c4b20 con 0x7f186415ca30 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186414fac0 con 0x7f186415ce10 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186414fda0 con 0x7f18640cafe0 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f186415ca30 0x7f18640c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f186415ca30 0x7f18640c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53844/0 (socket says 172.21.15.7:53844) 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 -- 172.21.15.7:0/3645707995 learned_addr learned my addr 172.21.15.7:0/3645707995 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186a49a640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18640cafe0 0x7f18640c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186ac9b640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f186415ce10 0x7f18640c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 -- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18640cafe0 msgr2=0x7f18640c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18640cafe0 0x7f18640c3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 -- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f186415ce10 msgr2=0x7f18640c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f186415ce10 0x7f18640c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 -- 172.21.15.7:0/3645707995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c009a70 con 0x7f186415ca30 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186ac9b640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f186415ce10 0x7f18640c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186a49a640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18640cafe0 0x7f18640c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f1869c99640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f186415ca30 0x7f18640c54c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f185400e930 tx=0x7f185400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f185affd640 1 -- 172.21.15.7:0/3645707995 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854009690 con 0x7f186415ca30 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f185affd640 1 -- 172.21.15.7:0/3645707995 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185400ca00 con 0x7f186415ca30 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1864150080 con 0x7f186415ca30 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f185affd640 1 -- 172.21.15.7:0/3645707995 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185400cd70 con 0x7f186415ca30 2023-11-28T18:30:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18640c4fa0 con 0x7f186415ca30 2023-11-28T18:30:13.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.967+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18641505a0 con 0x7f186415ca30 2023-11-28T18:30:13.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.971+0000 7f185affd640 1 -- 172.21.15.7:0/3645707995 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1854013470 con 0x7f186415ca30 2023-11-28T18:30:13.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.971+0000 7f185affd640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18480668d0 0x7f1848068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:13.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.971+0000 7f185affd640 1 -- 172.21.15.7:0/3645707995 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f1854084820 con 0x7f186415ca30 2023-11-28T18:30:13.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.971+0000 7f186a49a640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18480668d0 0x7f1848068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:13.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.971+0000 7f186a49a640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18480668d0 0x7f1848068d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f18600011f0 tx=0x7f1860009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:13.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:13.975+0000 7f185affd640 1 -- 172.21.15.7:0/3645707995 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1854051990 con 0x7f186415ca30 2023-11-28T18:30:14.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.219+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18640c4de0 con 0x7f186415ca30 2023-11-28T18:30:14.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f185affd640 1 -- 172.21.15.7:0/3645707995 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5766 (secure 0 0 0) 0x7f1854054fb0 con 0x7f186415ca30 2023-11-28T18:30:14.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:14.226 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:13.207949+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:14.227 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 38 2023-11-28T18:30:14.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18480668d0 msgr2=0x7f1848068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:14.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18480668d0 0x7f1848068d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f18600011f0 tx=0x7f1860009040 comp rx=0 tx=0).stop 2023-11-28T18:30:14.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f186415ca30 msgr2=0x7f18640c54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:14.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f186415ca30 0x7f18640c54c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f185400e930 tx=0x7f185400edf0 comp rx=0 tx=0).stop 2023-11-28T18:30:14.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 shutdown_connections 2023-11-28T18:30:14.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18480668d0 0x7f1848068d90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:14.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f186415ce10 0x7f18640c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:14.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f186415ca30 0x7f18640c54c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:14.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 --2- 172.21.15.7:0/3645707995 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18640cafe0 0x7f18640c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:14.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 >> 172.21.15.7:0/3645707995 conn(0x7f186406c060 msgr2=0x7f18640c94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:14.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 shutdown_connections 2023-11-28T18:30:14.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:14.223+0000 7f186b49c640 1 -- 172.21.15.7:0/3645707995 wait complete. 2023-11-28T18:30:14.254 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:16.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:16.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.403+0000 7f921d180640 1 Processor -- start 2023-11-28T18:30:16.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.403+0000 7f921d180640 1 -- start start 2023-11-28T18:30:16.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921d180640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f92181542e0 0x7f921815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:16.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921d180640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9218072450 0x7f92180728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:16.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921d180640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92180ca920 con 0x7f92180cbb70 2023-11-28T18:30:16.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921d180640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9218071d50 con 0x7f9218072450 2023-11-28T18:30:16.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921d180640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9218072030 con 0x7f92181542e0 2023-11-28T18:30:16.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921659c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f92180cbb70 0x7f92180cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53712/0 (socket says 172.21.15.7:53712) 2023-11-28T18:30:16.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921659c640 1 -- 172.21.15.7:0/3669421313 learned_addr learned my addr 172.21.15.7:0/3669421313 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f9216d9d640 1 --2- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f92181542e0 0x7f921815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921759e640 1 --2- 172.21.15.7:0/3669421313 >> v2:172.21.15.193:3300/0 conn(0x7f9218072450 0x7f92180728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f9215d9b640 1 -- 172.21.15.7:0/3669421313 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861646772 0 0) 0x7f92180ca920 con 0x7f92180cbb70 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f9215d9b640 1 -- 172.21.15.7:0/3669421313 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9200003530 con 0x7f92180cbb70 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f9215d9b640 1 -- 172.21.15.7:0/3669421313 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136325559 0 0) 0x7f9200003530 con 0x7f92180cbb70 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921759e640 1 -- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f92181542e0 msgr2=0x7f921815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921759e640 1 --2- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f92181542e0 0x7f921815e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921759e640 1 -- 172.21.15.7:0/3669421313 >> v1:172.21.15.7:6789/0 conn(0x7f92180cbb70 legacy=0x7f92180cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f921759e640 1 -- 172.21.15.7:0/3669421313 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921815e550 con 0x7f9218072450 2023-11-28T18:30:16.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.407+0000 7f9216d9d640 1 --2- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f92181542e0 0x7f921815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921759e640 1 --2- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9218072450 0x7f92180728e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9210009d90 tx=0x7f921001c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3219bb61a656979 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:16.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f9215d9b640 1 -- 172.21.15.7:0/3669421313 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c001680 con 0x7f9218072450 2023-11-28T18:30:16.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f9215d9b640 1 -- 172.21.15.7:0/3669421313 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92100a5670 con 0x7f9218072450 2023-11-28T18:30:16.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f9215d9b640 1 -- 172.21.15.7:0/3669421313 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92100a5a00 con 0x7f9218072450 2023-11-28T18:30:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921d180640 1 -- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9218072450 msgr2=0x7f92180728e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921d180640 1 --2- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9218072450 0x7f92180728e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9210009d90 tx=0x7f921001c870 comp rx=0 tx=0).stop 2023-11-28T18:30:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921d180640 1 -- 172.21.15.7:0/3669421313 shutdown_connections 2023-11-28T18:30:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921d180640 1 --2- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9218072450 0x7f92180728e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921d180640 1 --2- 172.21.15.7:0/3669421313 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f92181542e0 0x7f921815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921d180640 1 -- 172.21.15.7:0/3669421313 >> 172.21.15.7:0/3669421313 conn(0x7f92180717e0 msgr2=0x7f92180662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921d180640 1 -- 172.21.15.7:0/3669421313 shutdown_connections 2023-11-28T18:30:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.411+0000 7f921d180640 1 -- 172.21.15.7:0/3669421313 wait complete. 2023-11-28T18:30:16.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921d180640 1 Processor -- start 2023-11-28T18:30:16.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921d180640 1 -- start start 2023-11-28T18:30:16.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921d180640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9218072450 0x7f92180c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:16.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921d180640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92180cbb70 0x7f92180c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:16.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921d180640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92181542e0 0x7f921815f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:16.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921d180640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9218160b50 con 0x7f92180cbb70 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921d180640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92181f33f0 con 0x7f92181542e0 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921d180640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92181f2e70 con 0x7f9218072450 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92180cbb70 0x7f92180c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f9216d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9218072450 0x7f92180c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f9216d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9218072450 0x7f92180c8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:46732/0 (socket says 172.21.15.7:46732) 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f9216d9d640 1 -- 172.21.15.7:0/3644871942 learned_addr learned my addr 172.21.15.7:0/3644871942 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f921759e640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92181542e0 0x7f921815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f9216d9d640 1 -- 172.21.15.7:0/3644871942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92181542e0 msgr2=0x7f921815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f9216d9d640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92181542e0 0x7f921815f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f9216d9d640 1 -- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92180cbb70 msgr2=0x7f92180c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f9216d9d640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92180cbb70 0x7f92180c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.415+0000 7f9216d9d640 1 -- 172.21.15.7:0/3644871942 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9208009420 con 0x7f9218072450 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.419+0000 7f9216d9d640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9218072450 0x7f92180c8930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f920800ebe0 tx=0x7f920800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.419+0000 7f91ff7fe640 1 -- 172.21.15.7:0/3644871942 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920800eee0 con 0x7f9218072450 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.419+0000 7f91ff7fe640 1 -- 172.21.15.7:0/3644871942 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920800cc30 con 0x7f9218072450 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.419+0000 7f91ff7fe640 1 -- 172.21.15.7:0/3644871942 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9208013470 con 0x7f9218072450 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.419+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9210009a70 con 0x7f9218072450 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.419+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921815e8b0 con 0x7f9218072450 2023-11-28T18:30:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.419+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921806c670 con 0x7f9218072450 2023-11-28T18:30:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.419+0000 7f91ff7fe640 1 -- 172.21.15.7:0/3644871942 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f92080059a0 con 0x7f9218072450 2023-11-28T18:30:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.423+0000 7f91ff7fe640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f92040668d0 0x7f9204068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.423+0000 7f91ff7fe640 1 -- 172.21.15.7:0/3644871942 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f9208084960 con 0x7f9218072450 2023-11-28T18:30:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.423+0000 7f921659c640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f92040668d0 0x7f9204068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.423+0000 7f921659c640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f92040668d0 0x7f9204068d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f920c001190 tx=0x7f920c001120 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.423+0000 7f91ff7fe640 1 -- 172.21.15.7:0/3644871942 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9208051a20 con 0x7f9218072450 2023-11-28T18:30:16.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.655+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f921804e890 con 0x7f9218072450 2023-11-28T18:30:16.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.655+0000 7f91ff7fe640 1 -- 172.21.15.7:0/3644871942 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5766 (secure 0 0 0) 0x7f9208021090 con 0x7f9218072450 2023-11-28T18:30:16.661 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:16.661 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:13.207949+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:16.661 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 38 2023-11-28T18:30:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f92040668d0 msgr2=0x7f9204068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f92040668d0 0x7f9204068d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f920c001190 tx=0x7f920c001120 comp rx=0 tx=0).stop 2023-11-28T18:30:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9218072450 msgr2=0x7f92180c8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9218072450 0x7f92180c8930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f920800ebe0 tx=0x7f920800c520 comp rx=0 tx=0).stop 2023-11-28T18:30:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 shutdown_connections 2023-11-28T18:30:16.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f92040668d0 0x7f9204068d90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92181542e0 0x7f921815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92180cbb70 0x7f92180c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 --2- 172.21.15.7:0/3644871942 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9218072450 0x7f92180c8930 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 >> 172.21.15.7:0/3644871942 conn(0x7f92180717e0 msgr2=0x7f92180c5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 shutdown_connections 2023-11-28T18:30:16.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:16.659+0000 7f921d180640 1 -- 172.21.15.7:0/3644871942 wait complete. 2023-11-28T18:30:16.688 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4285": { "addr": "172.21.15.193:6837/3911527535", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 3911527535, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 3911527535, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 11, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.7:6838/2690556675", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 2690556675, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 2690556675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:reconnect", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:30:13.207949+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 4288, "mds_2": 4285, "mds_3": 5086, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:30:18.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:18.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3ab7c4640 1 Processor -- start 2023-11-28T18:30:18.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3ab7c4640 1 -- start start 2023-11-28T18:30:18.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3ab7c4640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe3a40ca4d0 0x7fe3a415a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3ab7c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a40cf6c0 0x7fe3a40cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3ab7c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a415e380 con 0x7fe3a41540e0 2023-11-28T18:30:18.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3ab7c4640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a415ade0 con 0x7fe3a40ca4d0 2023-11-28T18:30:18.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3ab7c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415d7e0 con 0x7fe3a40cf6c0 2023-11-28T18:30:18.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3a9560640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe3a41540e0 0x7fe3a415dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53714/0 (socket says 172.21.15.7:53714) 2023-11-28T18:30:18.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3a9560640 1 -- 172.21.15.7:0/2272020739 learned_addr learned my addr 172.21.15.7:0/2272020739 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3a8d5f640 1 --2- 172.21.15.7:0/2272020739 >> v2:172.21.15.193:3300/0 conn(0x7fe3a40ca4d0 0x7fe3a415a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3a9d61640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a40cf6c0 0x7fe3a40cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213267564 0 0) 0x7fe3a0001120 con 0x7fe3a41540e0 2023-11-28T18:30:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe390003530 con 0x7fe3a41540e0 2023-11-28T18:30:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784869323 0 0) 0x7fe390003530 con 0x7fe3a41540e0 2023-11-28T18:30:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a40cf6c0 msgr2=0x7fe3a40cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a40cf6c0 0x7fe3a40cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 >> v2:172.21.15.193:3300/0 conn(0x7fe3a40ca4d0 msgr2=0x7fe3a415a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 --2- 172.21.15.7:0/2272020739 >> v2:172.21.15.193:3300/0 conn(0x7fe3a40ca4d0 0x7fe3a415a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40ca8f0 con 0x7fe3a41540e0 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fe3a0001810 con 0x7fe3a41540e0 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 >> v1:172.21.15.7:6789/0 conn(0x7fe3a41540e0 legacy=0x7fe3a415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a40cf6c0 0x7fe390003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe390004d10 0x7fe390005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415ade0 con 0x7fe390003f50 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415d7e0 con 0x7fe390004d10 2023-11-28T18:30:18.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39000b990 con 0x7fe3a40cf6c0 2023-11-28T18:30:18.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3a8d5f640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3a9d61640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe390004d10 0x7fe390005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.839+0000 7fe3a9560640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a40cf6c0 0x7fe390003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.843+0000 7fe3a8d5f640 1 -- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a40cf6c0 msgr2=0x7fe390003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:18.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.843+0000 7fe3a8d5f640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a40cf6c0 0x7fe390003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:18.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.843+0000 7fe3a8d5f640 1 -- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe390004d10 msgr2=0x7fe390005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:18.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.843+0000 7fe3a8d5f640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe390004d10 0x7fe390005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:18.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.843+0000 7fe3a8d5f640 1 -- 172.21.15.7:0/2272020739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe390003530 con 0x7fe390003f50 2023-11-28T18:30:18.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.843+0000 7fe3a8d5f640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3900047d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe394009b10 tx=0x7fe39401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e1003ee02ab8be server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:18.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.843+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3940a5680 con 0x7fe390003f50 2023-11-28T18:30:18.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.843+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3940a57e0 con 0x7fe390003f50 2023-11-28T18:30:18.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe39bfff640 1 -- 172.21.15.7:0/2272020739 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3940a5b90 con 0x7fe390003f50 2023-11-28T18:30:18.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/2272020739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 msgr2=0x7fe3900047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:18.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3ab7c4640 1 --2- 172.21.15.7:0/2272020739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3900047d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe394009b10 tx=0x7fe39401ca90 comp rx=0 tx=0).stop 2023-11-28T18:30:18.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3a9d61640 1 -- 172.21.15.7:0/2272020739 reap_dead start 2023-11-28T18:30:18.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/2272020739 shutdown_connections 2023-11-28T18:30:18.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/2272020739 >> 172.21.15.7:0/2272020739 conn(0x7fe3a40662e0 msgr2=0x7fe3a40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:18.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/2272020739 shutdown_connections 2023-11-28T18:30:18.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/2272020739 wait complete. 2023-11-28T18:30:18.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3ab7c4640 1 Processor -- start 2023-11-28T18:30:18.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.847+0000 7fe3ab7c4640 1 -- start start 2023-11-28T18:30:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3a415bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3a41e3230 0x7fe3a40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a415a920 0x7fe3a41e5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415b940 con 0x7fe390003f50 2023-11-28T18:30:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415ca10 con 0x7fe3a41e3230 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415cc40 con 0x7fe3a415a920 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3a415bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3a415bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53876/0 (socket says 172.21.15.7:53876) 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 -- 172.21.15.7:0/1588153503 learned_addr learned my addr 172.21.15.7:0/1588153503 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a8d5f640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3a41e3230 0x7fe3a40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9d61640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a415a920 0x7fe3a41e5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 -- 172.21.15.7:0/1588153503 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a415a920 msgr2=0x7fe3a41e5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a415a920 0x7fe3a41e5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 -- 172.21.15.7:0/1588153503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3a41e3230 msgr2=0x7fe3a40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3a41e3230 0x7fe3a40b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 -- 172.21.15.7:0/1588153503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a415cf20 con 0x7fe390003f50 2023-11-28T18:30:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3a9560640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3a415bd40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0011810 tx=0x7fe3a0011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe399ffb640 1 -- 172.21.15.7:0/1588153503 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a000fb60 con 0x7fe390003f50 2023-11-28T18:30:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe399ffb640 1 -- 172.21.15.7:0/1588153503 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a000fd00 con 0x7fe390003f50 2023-11-28T18:30:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a415afa0 con 0x7fe390003f50 2023-11-28T18:30:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe399ffb640 1 -- 172.21.15.7:0/1588153503 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0015650 con 0x7fe390003f50 2023-11-28T18:30:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a415b560 con 0x7fe390003f50 2023-11-28T18:30:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.851+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a404e890 con 0x7fe390003f50 2023-11-28T18:30:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.855+0000 7fe399ffb640 1 -- 172.21.15.7:0/1588153503 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe3a00157b0 con 0x7fe390003f50 2023-11-28T18:30:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.855+0000 7fe399ffb640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3880668d0 0x7fe388068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.855+0000 7fe399ffb640 1 -- 172.21.15.7:0/1588153503 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fe3a0087480 con 0x7fe390003f50 2023-11-28T18:30:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.855+0000 7fe3a8d5f640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3880668d0 0x7fe388068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.855+0000 7fe3a8d5f640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3880668d0 0x7fe388068d90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe3940af000 tx=0x7fe3940046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:18.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:18.859+0000 7fe399ffb640 1 -- 172.21.15.7:0/1588153503 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe3a0054680 con 0x7fe390003f50 2023-11-28T18:30:19.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.095+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3a415e380 con 0x7fe390003f50 2023-11-28T18:30:19.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.095+0000 7fe399ffb640 1 -- 172.21.15.7:0/1588153503 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5763 (secure 0 0 0) 0x7fe3a415e380 con 0x7fe390003f50 2023-11-28T18:30:19.100 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:19.101 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:18.224674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 39 2023-11-28T18:30:19.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3880668d0 msgr2=0x7fe388068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:19.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3880668d0 0x7fe388068d90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe3940af000 tx=0x7fe3940046d0 comp rx=0 tx=0).stop 2023-11-28T18:30:19.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 msgr2=0x7fe3a415bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:19.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3a415bd40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0011810 tx=0x7fe3a0011cd0 comp rx=0 tx=0).stop 2023-11-28T18:30:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 shutdown_connections 2023-11-28T18:30:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3880668d0 0x7fe388068d90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3a415a920 0x7fe3a41e5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3a41e3230 0x7fe3a40b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 --2- 172.21.15.7:0/1588153503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe390003f50 0x7fe3a415bd40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 >> 172.21.15.7:0/1588153503 conn(0x7fe3a40662e0 msgr2=0x7fe3a41e64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 shutdown_connections 2023-11-28T18:30:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:19.099+0000 7fe3ab7c4640 1 -- 172.21.15.7:0/1588153503 wait complete. 2023-11-28T18:30:19.126 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:21.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff99a334640 1 Processor -- start 2023-11-28T18:30:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff99a334640 1 -- start start 2023-11-28T18:30:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff99a334640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff994159c90 0x7ff99415d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff99a334640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff99415def0 0x7ff9940ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff99a334640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99415d240 con 0x7ff994153ee0 2023-11-28T18:30:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff99a334640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99415d520 con 0x7ff994159c90 2023-11-28T18:30:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff99a334640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940cbd10 con 0x7ff99415def0 2023-11-28T18:30:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff999332640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff994153ee0 0x7ff9940cb600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53726/0 (socket says 172.21.15.7:53726) 2023-11-28T18:30:21.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff999332640 1 -- 172.21.15.7:0/348262959 learned_addr learned my addr 172.21.15.7:0/348262959 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff998b31640 1 --2- 172.21.15.7:0/348262959 >> v2:172.21.15.193:3300/0 conn(0x7ff994159c90 0x7ff99415d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.287+0000 7ff999b33640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff99415def0 0x7ff9940ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369696097 0 0) 0x7ff99415d240 con 0x7ff994153ee0 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff980003530 con 0x7ff994153ee0 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1184279652 0 0) 0x7ff980003530 con 0x7ff994153ee0 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff99415def0 msgr2=0x7ff9940ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff99415def0 0x7ff9940ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 >> v2:172.21.15.193:3300/0 conn(0x7ff994159c90 msgr2=0x7ff99415d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 --2- 172.21.15.7:0/348262959 >> v2:172.21.15.193:3300/0 conn(0x7ff994159c90 0x7ff99415d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9940c4bc0 con 0x7ff994153ee0 2023-11-28T18:30:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7ff990001680 con 0x7ff994153ee0 2023-11-28T18:30:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 >> v1:172.21.15.7:6789/0 conn(0x7ff994153ee0 legacy=0x7ff9940cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980003980 0x7ff980003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff99415def0 0x7ff9800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff980004d10 0x7ff980005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99415d520 con 0x7ff980004d10 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940cbd10 con 0x7ff980003980 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000b990 con 0x7ff99415def0 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff998b31640 1 --2- 172.21.15.7:0/348262959 >> v2:172.21.15.193:3300/0 conn(0x7ff994159c90 0x7ff99415d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff999332640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980003980 0x7ff980003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff999b33640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff980004d10 0x7ff980005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff998b31640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff99415def0 0x7ff9800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff999b33640 1 -- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff99415def0 msgr2=0x7ff9800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff999b33640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff99415def0 0x7ff9800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff999b33640 1 -- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980003980 msgr2=0x7ff980003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff999b33640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980003980 0x7ff980003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff999b33640 1 -- 172.21.15.7:0/348262959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff980003530 con 0x7ff980004d10 2023-11-28T18:30:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.291+0000 7ff999332640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980003980 0x7ff980003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff999b33640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff980004d10 0x7ff980005190 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009f90 tx=0x7ff98c01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df8ee954c0c3a95 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0a5c30 con 0x7ff980004d10 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00f040 con 0x7ff980004d10 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff98bfff640 1 -- 172.21.15.7:0/348262959 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0ad440 con 0x7ff980004d10 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff99a334640 1 -- 172.21.15.7:0/348262959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff980004d10 msgr2=0x7ff980005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff99a334640 1 --2- 172.21.15.7:0/348262959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff980004d10 0x7ff980005190 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009f90 tx=0x7ff98c01cc80 comp rx=0 tx=0).stop 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff999b33640 1 -- 172.21.15.7:0/348262959 reap_dead start 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff99a334640 1 -- 172.21.15.7:0/348262959 shutdown_connections 2023-11-28T18:30:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff99a334640 1 -- 172.21.15.7:0/348262959 >> 172.21.15.7:0/348262959 conn(0x7ff994071e20 msgr2=0x7ff994072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff99a334640 1 -- 172.21.15.7:0/348262959 shutdown_connections 2023-11-28T18:30:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff99a334640 1 -- 172.21.15.7:0/348262959 wait complete. 2023-11-28T18:30:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff99a334640 1 Processor -- start 2023-11-28T18:30:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.295+0000 7ff99a334640 1 -- start start 2023-11-28T18:30:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980004d10 0x7ff994154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff99415c400 0x7ff994145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff994145cd0 0x7ff9940ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff994146150 con 0x7ff99415c400 2023-11-28T18:30:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940c6bf0 con 0x7ff980004d10 2023-11-28T18:30:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940c6910 con 0x7ff994145cd0 2023-11-28T18:30:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff99415c400 0x7ff994145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff99415c400 0x7ff994145790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53904/0 (socket says 172.21.15.7:53904) 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 -- 172.21.15.7:0/1214969020 learned_addr learned my addr 172.21.15.7:0/1214969020 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff999332640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980004d10 0x7ff994154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff999b33640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff994145cd0 0x7ff9940ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 -- 172.21.15.7:0/1214969020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff994145cd0 msgr2=0x7ff9940ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff994145cd0 0x7ff9940ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 -- 172.21.15.7:0/1214969020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980004d10 msgr2=0x7ff994154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980004d10 0x7ff994154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 -- 172.21.15.7:0/1214969020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9940c7230 con 0x7ff99415c400 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff998b31640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff99415c400 0x7ff994145790 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff984011cd0 tx=0x7ff98400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff989ffb640 1 -- 172.21.15.7:0/1214969020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984011e80 con 0x7ff99415c400 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff989ffb640 1 -- 172.21.15.7:0/1214969020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984004020 con 0x7ff99415c400 2023-11-28T18:30:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9940c7510 con 0x7ff99415c400 2023-11-28T18:30:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff989ffb640 1 -- 172.21.15.7:0/1214969020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840043b0 con 0x7ff99415c400 2023-11-28T18:30:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9940c7ad0 con 0x7ff99415c400 2023-11-28T18:30:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.299+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9940c7c90 con 0x7ff99415c400 2023-11-28T18:30:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.303+0000 7ff989ffb640 1 -- 172.21.15.7:0/1214969020 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff984003620 con 0x7ff99415c400 2023-11-28T18:30:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.303+0000 7ff989ffb640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff978066900 0x7ff978068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.303+0000 7ff989ffb640 1 -- 172.21.15.7:0/1214969020 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7ff984084f00 con 0x7ff99415c400 2023-11-28T18:30:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.303+0000 7ff999332640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff978066900 0x7ff978068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.303+0000 7ff999332640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff978066900 0x7ff978068dc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff990001040 tx=0x7ff990007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:21.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.307+0000 7ff989ffb640 1 -- 172.21.15.7:0/1214969020 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff984051fe0 con 0x7ff99415c400 2023-11-28T18:30:21.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.535+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff994072c80 con 0x7ff99415c400 2023-11-28T18:30:21.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.535+0000 7ff989ffb640 1 -- 172.21.15.7:0/1214969020 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5763 (secure 0 0 0) 0x7ff984022030 con 0x7ff99415c400 2023-11-28T18:30:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:21.540 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:18.224674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:21.540 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 39 2023-11-28T18:30:21.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff978066900 msgr2=0x7ff978068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:21.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff978066900 0x7ff978068dc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff990001040 tx=0x7ff990007900 comp rx=0 tx=0).stop 2023-11-28T18:30:21.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff99415c400 msgr2=0x7ff994145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:21.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff99415c400 0x7ff994145790 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff984011cd0 tx=0x7ff98400f610 comp rx=0 tx=0).stop 2023-11-28T18:30:21.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 shutdown_connections 2023-11-28T18:30:21.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff978066900 0x7ff978068dc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff994145cd0 0x7ff9940ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff99415c400 0x7ff994145790 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 --2- 172.21.15.7:0/1214969020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff980004d10 0x7ff994154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:21.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 >> 172.21.15.7:0/1214969020 conn(0x7ff994071e20 msgr2=0x7ff994158780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:21.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 shutdown_connections 2023-11-28T18:30:21.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:21.539+0000 7ff99a334640 1 -- 172.21.15.7:0/1214969020 wait complete. 2023-11-28T18:30:21.562 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:23.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.723+0000 7fc626e09640 1 Processor -- start 2023-11-28T18:30:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.723+0000 7fc626e09640 1 -- start start 2023-11-28T18:30:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc626e09640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cae40 0x7fc6200ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc626e09640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6200cf9f0 0x7fc6200cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc626e09640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62015d4e0 con 0x7fc6200cb220 2023-11-28T18:30:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc626e09640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6200d03c0 con 0x7fc6200cf9f0 2023-11-28T18:30:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc626e09640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200717f0 con 0x7fc6200cae40 2023-11-28T18:30:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc6253a6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6200cf9f0 0x7fc6200cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc6253a6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6200cf9f0 0x7fc6200cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55902/0 (socket says 172.21.15.7:55902) 2023-11-28T18:30:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc6253a6640 1 -- 172.21.15.7:0/923357864 learned_addr learned my addr 172.21.15.7:0/923357864 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc624ba5640 1 --2- 172.21.15.7:0/923357864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cae40 0x7fc6200ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc6253a6640 1 -- 172.21.15.7:0/923357864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cae40 msgr2=0x7fc6200ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc6253a6640 1 --2- 172.21.15.7:0/923357864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cae40 0x7fc6200ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc6253a6640 1 -- 172.21.15.7:0/923357864 >> v1:172.21.15.7:6789/0 conn(0x7fc6200cb220 legacy=0x7fc620070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.727+0000 7fc6253a6640 1 -- 172.21.15.7:0/923357864 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62015d730 con 0x7fc6200cf9f0 2023-11-28T18:30:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc6253a6640 1 --2- 172.21.15.7:0/923357864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cf9f0 0x7fc6200cfe80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc618009e50 tx=0x7fc61801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4755061520b1d0d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc61f7fe640 1 -- 172.21.15.7:0/923357864 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6180af400 con 0x7fc6200cf9f0 2023-11-28T18:30:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc61f7fe640 1 -- 172.21.15.7:0/923357864 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6180ab050 con 0x7fc6200cf9f0 2023-11-28T18:30:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc61f7fe640 1 -- 172.21.15.7:0/923357864 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6180af780 con 0x7fc6200cf9f0 2023-11-28T18:30:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc626e09640 1 -- 172.21.15.7:0/923357864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cf9f0 msgr2=0x7fc6200cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc626e09640 1 --2- 172.21.15.7:0/923357864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cf9f0 0x7fc6200cfe80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc618009e50 tx=0x7fc61801f1f0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc626e09640 1 -- 172.21.15.7:0/923357864 shutdown_connections 2023-11-28T18:30:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc626e09640 1 --2- 172.21.15.7:0/923357864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cf9f0 0x7fc6200cfe80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc626e09640 1 --2- 172.21.15.7:0/923357864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cae40 0x7fc6200ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc626e09640 1 -- 172.21.15.7:0/923357864 >> 172.21.15.7:0/923357864 conn(0x7fc620072310 msgr2=0x7fc620072720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc626e09640 1 -- 172.21.15.7:0/923357864 shutdown_connections 2023-11-28T18:30:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.731+0000 7fc626e09640 1 -- 172.21.15.7:0/923357864 wait complete. 2023-11-28T18:30:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 Processor -- start 2023-11-28T18:30:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 -- start start 2023-11-28T18:30:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6200cae40 0x7fc6201571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cb220 0x7fc6201f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cf9f0 0x7fc6201f1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc620157700 con 0x7fc6200cb220 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200cde40 con 0x7fc6200cae40 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6201e70c0 con 0x7fc6200cf9f0 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cb220 0x7fc6201f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cb220 0x7fc6201f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55904/0 (socket says 172.21.15.7:55904) 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 -- 172.21.15.7:0/3378920243 learned_addr learned my addr 172.21.15.7:0/3378920243 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc624ba5640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6200cae40 0x7fc6201571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc6253a6640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cf9f0 0x7fc6201f1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 -- 172.21.15.7:0/3378920243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cf9f0 msgr2=0x7fc6201f1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cf9f0 0x7fc6201f1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 -- 172.21.15.7:0/3378920243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6200cae40 msgr2=0x7fc6201571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6200cae40 0x7fc6201571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 -- 172.21.15.7:0/3378920243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc618009b30 con 0x7fc6200cb220 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61ffff640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cb220 0x7fc6201f2da0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00c530 tx=0x7fc60c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61d7fa640 1 -- 172.21.15.7:0/3378920243 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c005220 con 0x7fc6200cb220 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61d7fa640 1 -- 172.21.15.7:0/3378920243 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c005380 con 0x7fc6200cb220 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6201f3a30 con 0x7fc6200cb220 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.735+0000 7fc61d7fa640 1 -- 172.21.15.7:0/3378920243 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c007270 con 0x7fc6200cb220 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.739+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6201f3750 con 0x7fc6200cb220 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.739+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc62004e890 con 0x7fc6200cb220 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.739+0000 7fc61d7fa640 1 -- 172.21.15.7:0/3378920243 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc60c0073d0 con 0x7fc6200cb220 2023-11-28T18:30:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.739+0000 7fc61d7fa640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6040668d0 0x7fc604068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.739+0000 7fc61d7fa640 1 -- 172.21.15.7:0/3378920243 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fc60c083da0 con 0x7fc6200cb220 2023-11-28T18:30:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.739+0000 7fc624ba5640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6040668d0 0x7fc604068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.743+0000 7fc624ba5640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6040668d0 0x7fc604068d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc6140079c0 tx=0x7fc614008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:23.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.743+0000 7fc61d7fa640 1 -- 172.21.15.7:0/3378920243 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc60c050e80 con 0x7fc6200cb220 2023-11-28T18:30:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.951+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6200caa50 con 0x7fc6200cb220 2023-11-28T18:30:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.951+0000 7fc61d7fa640 1 -- 172.21.15.7:0/3378920243 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5763 (secure 0 0 0) 0x7fc60c0544a0 con 0x7fc6200cb220 2023-11-28T18:30:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:23.956 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:18.224674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:23.956 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 39 2023-11-28T18:30:23.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6040668d0 msgr2=0x7fc604068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:23.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6040668d0 0x7fc604068d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc6140079c0 tx=0x7fc614008040 comp rx=0 tx=0).stop 2023-11-28T18:30:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cb220 msgr2=0x7fc6201f2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cb220 0x7fc6201f2da0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00c530 tx=0x7fc60c00c9f0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 shutdown_connections 2023-11-28T18:30:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6040668d0 0x7fc604068d90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6200cf9f0 0x7fc6201f1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6200cb220 0x7fc6201f2da0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 --2- 172.21.15.7:0/3378920243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6200cae40 0x7fc6201571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 >> 172.21.15.7:0/3378920243 conn(0x7fc620072310 msgr2=0x7fc620159d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 shutdown_connections 2023-11-28T18:30:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:23.955+0000 7fc626e09640 1 -- 172.21.15.7:0/3378920243 wait complete. 2023-11-28T18:30:23.982 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:25.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:26.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.143+0000 7f26f189c640 1 Processor -- start 2023-11-28T18:30:26.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.143+0000 7f26f189c640 1 -- start start 2023-11-28T18:30:26.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.143+0000 7f26f189c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f26ec15ca30 0x7f26ec159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:26.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.143+0000 7f26f189c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec15a620 con 0x7f26ec15ce10 2023-11-28T18:30:26.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.143+0000 7f26f189c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec15a900 con 0x7f26ec0cafe0 2023-11-28T18:30:26.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.143+0000 7f26f189c640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec15abe0 con 0x7f26ec15ca30 2023-11-28T18:30:26.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26ebfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f26ec15ce10 0x7f26ec0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51166/0 (socket says 172.21.15.7:51166) 2023-11-28T18:30:26.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26ebfff640 1 -- 172.21.15.7:0/1468598137 learned_addr learned my addr 172.21.15.7:0/1468598137 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:26.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f089a640 1 --2- 172.21.15.7:0/1468598137 >> v2:172.21.15.193:3300/0 conn(0x7f26ec15ca30 0x7f26ec159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:26.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80355656 0 0) 0x7f26ec15a620 con 0x7f26ec15ce10 2023-11-28T18:30:26.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc003530 con 0x7f26ec15ce10 2023-11-28T18:30:26.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897044429 0 0) 0x7f26dc003530 con 0x7f26ec15ce10 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 >> v2:172.21.15.193:3300/0 conn(0x7f26ec15ca30 msgr2=0x7f26ec159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 --2- 172.21.15.7:0/1468598137 >> v2:172.21.15.193:3300/0 conn(0x7f26ec15ca30 0x7f26ec159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 >> v1:172.21.15.193:6789/0 conn(0x7f26ec0cafe0 legacy=0x7f26ec0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec0cbb10 con 0x7f26ec15ce10 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f26d8001680 con 0x7f26ec15ce10 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 >> v1:172.21.15.7:6789/0 conn(0x7f26ec15ce10 legacy=0x7f26ec0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc003980 0x7f26dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26dc0047d0 0x7f26dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26dc005190 0x7f26dc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec15abe0 con 0x7f26dc005190 2023-11-28T18:30:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc00c030 con 0x7f26dc003980 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc00c1f0 con 0x7f26dc0047d0 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f089a640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc003980 0x7f26dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f109b640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26dc005190 0x7f26dc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26ebfff640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26dc0047d0 0x7f26dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f109b640 1 -- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26dc0047d0 msgr2=0x7f26dc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f109b640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26dc0047d0 0x7f26dc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f109b640 1 -- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc003980 msgr2=0x7f26dc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f109b640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc003980 0x7f26dc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f109b640 1 -- 172.21.15.7:0/1468598137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc003530 con 0x7f26dc005190 2023-11-28T18:30:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.147+0000 7f26f089a640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc003980 0x7f26dc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:26.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f109b640 1 -- 172.21.15.7:0/1468598137 reap_dead start 2023-11-28T18:30:26.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f109b640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26dc005190 0x7f26dc00b990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f26e4001e70 tx=0x7f26e401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d05625c250ba19 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:26.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4003b80 con 0x7f26dc005190 2023-11-28T18:30:26.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e4003d20 con 0x7f26dc005190 2023-11-28T18:30:26.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26eb7fe640 1 -- 172.21.15.7:0/1468598137 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4005260 con 0x7f26dc005190 2023-11-28T18:30:26.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f189c640 1 -- 172.21.15.7:0/1468598137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26dc005190 msgr2=0x7f26dc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:26.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f189c640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26dc005190 0x7f26dc00b990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f26e4001e70 tx=0x7f26e401c760 comp rx=0 tx=0).stop 2023-11-28T18:30:26.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f189c640 1 -- 172.21.15.7:0/1468598137 shutdown_connections 2023-11-28T18:30:26.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f189c640 1 --2- 172.21.15.7:0/1468598137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26dc005190 0x7f26dc00b990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f189c640 1 -- 172.21.15.7:0/1468598137 >> 172.21.15.7:0/1468598137 conn(0x7f26ec06c060 msgr2=0x7f26ec0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f189c640 1 -- 172.21.15.7:0/1468598137 shutdown_connections 2023-11-28T18:30:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.151+0000 7f26f189c640 1 -- 172.21.15.7:0/1468598137 wait complete. 2023-11-28T18:30:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 Processor -- start 2023-11-28T18:30:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 -- start start 2023-11-28T18:30:26.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc005190 0x7f26ec1f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:26.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26ec1512a0 0x7f26ec0c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26ec15b3c0 0x7f26ec14e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec0c37b0 con 0x7f26ec1512a0 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec154600 con 0x7f26dc005190 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec1f3610 con 0x7f26ec15b3c0 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26ec1512a0 0x7f26ec0c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26ec1512a0 0x7f26ec0c4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55920/0 (socket says 172.21.15.7:55920) 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 -- 172.21.15.7:0/4186043953 learned_addr learned my addr 172.21.15.7:0/4186043953 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f089a640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc005190 0x7f26ec1f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f109b640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26ec15b3c0 0x7f26ec14e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 -- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26ec15b3c0 msgr2=0x7f26ec14e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26ec15b3c0 0x7f26ec14e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 -- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc005190 msgr2=0x7f26ec1f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc005190 0x7f26ec1f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 -- 172.21.15.7:0/4186043953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec154830 con 0x7f26ec1512a0 2023-11-28T18:30:26.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f109b640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26ec15b3c0 0x7f26ec14e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:30:26.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26ebfff640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26ec1512a0 0x7f26ec0c4620 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f26d80072c0 tx=0x7f26d8007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:26.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26e97fa640 1 -- 172.21.15.7:0/4186043953 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d800f940 con 0x7f26ec1512a0 2023-11-28T18:30:26.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26e97fa640 1 -- 172.21.15.7:0/4186043953 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d800faa0 con 0x7f26ec1512a0 2023-11-28T18:30:26.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ec154af0 con 0x7f26ec1512a0 2023-11-28T18:30:26.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26e97fa640 1 -- 172.21.15.7:0/4186043953 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d800fe50 con 0x7f26ec1512a0 2023-11-28T18:30:26.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.155+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ec150aa0 con 0x7f26ec1512a0 2023-11-28T18:30:26.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.159+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ec150620 con 0x7f26ec1512a0 2023-11-28T18:30:26.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.159+0000 7f26e97fa640 1 -- 172.21.15.7:0/4186043953 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f26d801d460 con 0x7f26ec1512a0 2023-11-28T18:30:26.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.159+0000 7f26e97fa640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f26d0066900 0x7f26d0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:26.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.159+0000 7f26e97fa640 1 -- 172.21.15.7:0/4186043953 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f26d8084750 con 0x7f26ec1512a0 2023-11-28T18:30:26.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.159+0000 7f26f089a640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f26d0066900 0x7f26d0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:26.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.159+0000 7f26f089a640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f26d0066900 0x7f26d0068dc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009c40 tx=0x7f26e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:26.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.163+0000 7f26e97fa640 1 -- 172.21.15.7:0/4186043953 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f26ec150620 con 0x7f26ec1512a0 2023-11-28T18:30:26.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.391+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26ec150620 con 0x7f26ec1512a0 2023-11-28T18:30:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.391+0000 7f26e97fa640 1 -- 172.21.15.7:0/4186043953 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5763 (secure 0 0 0) 0x7f26ec150620 con 0x7f26ec1512a0 2023-11-28T18:30:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:26.397 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 40 2023-11-28T18:30:26.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f26d0066900 msgr2=0x7f26d0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:26.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f26d0066900 0x7f26d0068dc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009c40 tx=0x7f26e0008040 comp rx=0 tx=0).stop 2023-11-28T18:30:26.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26ec1512a0 msgr2=0x7f26ec0c4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:26.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26ec1512a0 0x7f26ec0c4620 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f26d80072c0 tx=0x7f26d8007780 comp rx=0 tx=0).stop 2023-11-28T18:30:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 shutdown_connections 2023-11-28T18:30:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f26d0066900 0x7f26d0068dc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f26ec15b3c0 0x7f26ec14e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26ec1512a0 0x7f26ec0c4620 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 --2- 172.21.15.7:0/4186043953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26dc005190 0x7f26ec1f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 >> 172.21.15.7:0/4186043953 conn(0x7f26ec06c060 msgr2=0x7f26ec0ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 shutdown_connections 2023-11-28T18:30:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:26.395+0000 7f26f189c640 1 -- 172.21.15.7:0/4186043953 wait complete. 2023-11-28T18:30:26.423 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=3, replacing gid=4284 2023-11-28T18:30:26.423 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2023-11-28T18:30:26.990 INFO:tasks.ceph.mds.e.smithi007.stderr:2023-11-28T18:30:26.987+0000 7f7e1dce5640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-11-28T18:30:27.423 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-11-28T18:30:27.424 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T18:30:27.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T18:30:27.428 INFO:tasks.ceph.mds.a:Started 2023-11-28T18:30:27.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:27.529 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2023-11-28T18:30:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.583+0000 7f88d5a59640 1 Processor -- start 2023-11-28T18:30:27.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.583+0000 7f88d5a59640 1 -- start start 2023-11-28T18:30:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88d5a59640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f88d00c9080 0x7f88d006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88d5a59640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88d00c8050 0x7f88d0151870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88d5a59640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00c5670 con 0x7f88d006e3c0 2023-11-28T18:30:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88d5a59640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d00c41c0 con 0x7f88d00c8050 2023-11-28T18:30:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88d5a59640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d00c5c10 con 0x7f88d00c9080 2023-11-28T18:30:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88cffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88d00c8050 0x7f88d0151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88cffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88d00c8050 0x7f88d0151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55948/0 (socket says 172.21.15.7:55948) 2023-11-28T18:30:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88d5258640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f88d006e3c0 0x7f88d00662e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51178/0 (socket says 172.21.15.7:51178) 2023-11-28T18:30:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88cffff640 1 -- 172.21.15.7:0/2777411400 learned_addr learned my addr 172.21.15.7:0/2777411400 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88d4a57640 1 --2- 172.21.15.7:0/2777411400 >> v2:172.21.15.193:3300/0 conn(0x7f88d00c9080 0x7f88d006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88cffff640 1 -- 172.21.15.7:0/2777411400 >> v2:172.21.15.193:3300/0 conn(0x7f88d00c9080 msgr2=0x7f88d006ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88cffff640 1 --2- 172.21.15.7:0/2777411400 >> v2:172.21.15.193:3300/0 conn(0x7f88d00c9080 0x7f88d006ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88cffff640 1 -- 172.21.15.7:0/2777411400 >> v1:172.21.15.7:6789/0 conn(0x7f88d006e3c0 legacy=0x7f88d00662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.587+0000 7f88cffff640 1 -- 172.21.15.7:0/2777411400 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d00c60a0 con 0x7f88d00c8050 2023-11-28T18:30:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88cffff640 1 --2- 172.21.15.7:0/2777411400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c8050 0x7f88d0151870 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f88d0073230 tx=0x7f88bc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa47df472013a58 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88cf7fe640 1 -- 172.21.15.7:0/2777411400 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0ab5f0 con 0x7f88d00c8050 2023-11-28T18:30:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88cf7fe640 1 -- 172.21.15.7:0/2777411400 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc0ab750 con 0x7f88d00c8050 2023-11-28T18:30:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88cf7fe640 1 -- 172.21.15.7:0/2777411400 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0abae0 con 0x7f88d00c8050 2023-11-28T18:30:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88d5a59640 1 -- 172.21.15.7:0/2777411400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c8050 msgr2=0x7f88d0151870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88d5a59640 1 --2- 172.21.15.7:0/2777411400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c8050 0x7f88d0151870 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f88d0073230 tx=0x7f88bc01c500 comp rx=0 tx=0).stop 2023-11-28T18:30:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88d5a59640 1 -- 172.21.15.7:0/2777411400 shutdown_connections 2023-11-28T18:30:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88d5a59640 1 --2- 172.21.15.7:0/2777411400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c8050 0x7f88d0151870 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88d5a59640 1 --2- 172.21.15.7:0/2777411400 >> v2:172.21.15.193:3300/0 conn(0x7f88d00c9080 0x7f88d006ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88d5a59640 1 -- 172.21.15.7:0/2777411400 >> 172.21.15.7:0/2777411400 conn(0x7f88d006a5f0 msgr2=0x7f88d006aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88d5a59640 1 -- 172.21.15.7:0/2777411400 shutdown_connections 2023-11-28T18:30:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.591+0000 7f88d5a59640 1 -- 172.21.15.7:0/2777411400 wait complete. 2023-11-28T18:30:27.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 Processor -- start 2023-11-28T18:30:27.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 -- start start 2023-11-28T18:30:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88d006e3c0 0x7f88d0154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88d00c8050 0x7f88d0072360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c9080 0x7f88d01426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d0071970 con 0x7f88d00c9080 2023-11-28T18:30:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d006aa70 con 0x7f88d00c8050 2023-11-28T18:30:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d006ad50 con 0x7f88d006e3c0 2023-11-28T18:30:27.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c9080 0x7f88d01426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d4a57640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88d006e3c0 0x7f88d0154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c9080 0x7f88d01426e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55960/0 (socket says 172.21.15.7:55960) 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 -- 172.21.15.7:0/545348462 learned_addr learned my addr 172.21.15.7:0/545348462 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88cffff640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88d00c8050 0x7f88d0072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 -- 172.21.15.7:0/545348462 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88d006e3c0 msgr2=0x7f88d0154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88d006e3c0 0x7f88d0154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 -- 172.21.15.7:0/545348462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88d00c8050 msgr2=0x7f88d0072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88d00c8050 0x7f88d0072360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 -- 172.21.15.7:0/545348462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc0094d0 con 0x7f88d00c9080 2023-11-28T18:30:27.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5258640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c9080 0x7f88d01426e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f88c800c700 tx=0x7f88c800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:27.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88cd7fa640 1 -- 172.21.15.7:0/545348462 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c800ce10 con 0x7f88d00c9080 2023-11-28T18:30:27.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88cd7fa640 1 -- 172.21.15.7:0/545348462 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c800a640 con 0x7f88d00c9080 2023-11-28T18:30:27.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88cd7fa640 1 -- 172.21.15.7:0/545348462 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c800a9f0 con 0x7f88d00c9080 2023-11-28T18:30:27.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.595+0000 7f88d5a59640 1 -- 172.21.15.7:0/545348462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d0071b60 con 0x7f88d00c9080 2023-11-28T18:30:27.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.599+0000 7f88d5a59640 1 -- 172.21.15.7:0/545348462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d01f66d0 con 0x7f88d00c9080 2023-11-28T18:30:27.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.599+0000 7f88baffd640 1 -- 172.21.15.7:0/545348462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d006af10 con 0x7f88d00c9080 2023-11-28T18:30:27.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.599+0000 7f88cd7fa640 1 -- 172.21.15.7:0/545348462 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f88c800ab50 con 0x7f88d00c9080 2023-11-28T18:30:27.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.599+0000 7f88cd7fa640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88b40668d0 0x7f88b4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:27.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.599+0000 7f88cd7fa640 1 -- 172.21.15.7:0/545348462 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f88c8083c80 con 0x7f88d00c9080 2023-11-28T18:30:27.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.603+0000 7f88d4a57640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88b40668d0 0x7f88b4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:27.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.603+0000 7f88d4a57640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88b40668d0 0x7f88b4068d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88c40079c0 tx=0x7f88c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:27.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.607+0000 7f88cd7fa640 1 -- 172.21.15.7:0/545348462 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f88c8050e80 con 0x7f88d00c9080 2023-11-28T18:30:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.879+0000 7f88baffd640 1 -- 172.21.15.7:0/545348462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88d0072080 con 0x7f88d00c9080 2023-11-28T18:30:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.879+0000 7f88cd7fa640 1 -- 172.21.15.7:0/545348462 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5763 (secure 0 0 0) 0x7f88d0072080 con 0x7f88d00c9080 2023-11-28T18:30:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:27.884 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:27.884 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 40 2023-11-28T18:30:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 -- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88b40668d0 msgr2=0x7f88b4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88b40668d0 0x7f88b4068d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88c40079c0 tx=0x7f88c4008040 comp rx=0 tx=0).stop 2023-11-28T18:30:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 -- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c9080 msgr2=0x7f88d01426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c9080 0x7f88d01426e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f88c800c700 tx=0x7f88c800cbc0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 -- 172.21.15.7:0/545348462 shutdown_connections 2023-11-28T18:30:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88b40668d0 0x7f88b4068d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88d00c9080 0x7f88d01426e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88d00c8050 0x7f88d0072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 --2- 172.21.15.7:0/545348462 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88d006e3c0 0x7f88d0154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 -- 172.21.15.7:0/545348462 >> 172.21.15.7:0/545348462 conn(0x7f88d006a5f0 msgr2=0x7f88d006f420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 -- 172.21.15.7:0/545348462 shutdown_connections 2023-11-28T18:30:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:27.883+0000 7f88baffd640 1 -- 172.21.15.7:0/545348462 wait complete. 2023-11-28T18:30:27.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-11-28T18:30:29.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:30.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5b59e640 1 Processor -- start 2023-11-28T18:30:30.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5b59e640 1 -- start start 2023-11-28T18:30:30.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5b59e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:30.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5b59e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fef5c0cafe0 0x7fef5c0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:30.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5b59e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c15a620 con 0x7fef5c15ce10 2023-11-28T18:30:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5b59e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5c15a900 con 0x7fef5c0cafe0 2023-11-28T18:30:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5b59e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c15abe0 con 0x7fef5c15ca30 2023-11-28T18:30:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5ad9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fef5c0cafe0 0x7fef5c0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5a59c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5a59c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:47502/0 (socket says 172.21.15.7:47502) 2023-11-28T18:30:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5ad9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fef5c0cafe0 0x7fef5c0cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:47910/0 (socket says 172.21.15.7:47910) 2023-11-28T18:30:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5a59c640 1 -- 172.21.15.7:0/2844027912 learned_addr learned my addr 172.21.15.7:0/2844027912 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef59d9b640 1 --1- 172.21.15.7:0/2844027912 >> v1:172.21.15.193:6789/0 conn(0x7fef5c15ce10 0x7fef5c0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:36420/0 (socket says 172.21.15.7:36420) 2023-11-28T18:30:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5959a640 1 -- 172.21.15.7:0/2844027912 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759577482 0 0) 0x7fef5c15a620 con 0x7fef5c15ce10 2023-11-28T18:30:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5959a640 1 -- 172.21.15.7:0/2844027912 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef44003530 con 0x7fef5c15ce10 2023-11-28T18:30:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5a59c640 1 -- 172.21.15.7:0/2844027912 >> v2:172.21.15.193:3300/0 conn(0x7fef5c0cafe0 msgr2=0x7fef5c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5a59c640 1 --2- 172.21.15.7:0/2844027912 >> v2:172.21.15.193:3300/0 conn(0x7fef5c0cafe0 0x7fef5c0cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5a59c640 1 -- 172.21.15.7:0/2844027912 >> v1:172.21.15.193:6789/0 conn(0x7fef5c15ce10 legacy=0x7fef5c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5a59c640 1 -- 172.21.15.7:0/2844027912 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c0cbb10 con 0x7fef5c15ca30 2023-11-28T18:30:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.059+0000 7fef5ad9d640 1 --2- 172.21.15.7:0/2844027912 >> v2:172.21.15.193:3300/0 conn(0x7fef5c0cafe0 0x7fef5c0cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.063+0000 7fef5a59c640 1 --2- 172.21.15.7:0/2844027912 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c159c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009870 tx=0x7fef4c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=852986fb3e15c9b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5959a640 1 -- 172.21.15.7:0/2844027912 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0a8a30 con 0x7fef5c15ca30 2023-11-28T18:30:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5959a640 1 -- 172.21.15.7:0/2844027912 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c0a8b90 con 0x7fef5c15ca30 2023-11-28T18:30:30.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5959a640 1 -- 172.21.15.7:0/2844027912 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0b0440 con 0x7fef5c15ca30 2023-11-28T18:30:30.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 -- 172.21.15.7:0/2844027912 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 msgr2=0x7fef5c159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:30.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 --2- 172.21.15.7:0/2844027912 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c159c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009870 tx=0x7fef4c01ed30 comp rx=0 tx=0).stop 2023-11-28T18:30:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 -- 172.21.15.7:0/2844027912 shutdown_connections 2023-11-28T18:30:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 --2- 172.21.15.7:0/2844027912 >> v2:172.21.15.193:3300/0 conn(0x7fef5c0cafe0 0x7fef5c0cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 --2- 172.21.15.7:0/2844027912 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c159c90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 -- 172.21.15.7:0/2844027912 >> 172.21.15.7:0/2844027912 conn(0x7fef5c06c060 msgr2=0x7fef5c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 -- 172.21.15.7:0/2844027912 shutdown_connections 2023-11-28T18:30:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 -- 172.21.15.7:0/2844027912 wait complete. 2023-11-28T18:30:30.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 Processor -- start 2023-11-28T18:30:30.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.067+0000 7fef5b59e640 1 -- start start 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef5c0cafe0 0x7fef5c0c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef5c15ce10 0x7fef5c14e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c159050 con 0x7fef5c0cafe0 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0c2f20 con 0x7fef5c15ce10 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1552a0 con 0x7fef5c15ca30 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef5c0cafe0 0x7fef5c0c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef5c0cafe0 0x7fef5c0c4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55966/0 (socket says 172.21.15.7:55966) 2023-11-28T18:30:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 -- 172.21.15.7:0/954229582 learned_addr learned my addr 172.21.15.7:0/954229582 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef59d9b640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5ad9d640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef5c15ce10 0x7fef5c14e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:30.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:30:30.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 -- 172.21.15.7:0/954229582 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 msgr2=0x7fef5c154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 -- 172.21.15.7:0/954229582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef5c15ce10 msgr2=0x7fef5c14e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef5c15ce10 0x7fef5c14e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 -- 172.21.15.7:0/954229582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef54009a70 con 0x7fef5c0cafe0 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5a59c640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef5c0cafe0 0x7fef5c0c4ad0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0b3040 tx=0x7fef4c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef4affd640 1 -- 172.21.15.7:0/954229582 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0b0d20 con 0x7fef5c0cafe0 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef4affd640 1 -- 172.21.15.7:0/954229582 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c0bb460 con 0x7fef5c0cafe0 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4c0094d0 con 0x7fef5c0cafe0 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef4affd640 1 -- 172.21.15.7:0/954229582 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0b0ea0 con 0x7fef5c0cafe0 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c1f02a0 con 0x7fef5c0cafe0 2023-11-28T18:30:30.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c04e890 con 0x7fef5c0cafe0 2023-11-28T18:30:30.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:30:30.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.071+0000 7fef4affd640 1 -- 172.21.15.7:0/954229582 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fef4c0b83f0 con 0x7fef5c0cafe0 2023-11-28T18:30:30.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.075+0000 7fef4affd640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef3c0668d0 0x7fef3c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:30.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.075+0000 7fef4affd640 1 -- 172.21.15.7:0/954229582 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fef4c121f80 con 0x7fef5c0cafe0 2023-11-28T18:30:30.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.075+0000 7fef59d9b640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef3c0668d0 0x7fef3c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:30.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.075+0000 7fef59d9b640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef3c0668d0 0x7fef3c068d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0c5c00 tx=0x7fef50007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:30.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.079+0000 7fef4affd640 1 -- 172.21.15.7:0/954229582 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fef4c0eefc0 con 0x7fef5c0cafe0 2023-11-28T18:30:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef5c1541d0 con 0x7fef5c0cafe0 2023-11-28T18:30:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef4affd640 1 -- 172.21.15.7:0/954229582 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7fef4c0b5020 con 0x7fef5c0cafe0 2023-11-28T18:30:30.291 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:30.291 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef3c0668d0 msgr2=0x7fef3c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef3c0668d0 0x7fef3c068d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0c5c00 tx=0x7fef50007d40 comp rx=0 tx=0).stop 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef5c0cafe0 msgr2=0x7fef5c0c4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef5c0cafe0 0x7fef5c0c4ad0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0b3040 tx=0x7fef4c01bad0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 shutdown_connections 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef3c0668d0 0x7fef3c068d90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef5c15ce10 0x7fef5c14e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef5c15ca30 0x7fef5c154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 --2- 172.21.15.7:0/954229582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef5c0cafe0 0x7fef5c0c4ad0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 >> 172.21.15.7:0/954229582 conn(0x7fef5c06c060 msgr2=0x7fef5c15bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:30.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 shutdown_connections 2023-11-28T18:30:30.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:30.287+0000 7fef5b59e640 1 -- 172.21.15.7:0/954229582 wait complete. 2023-11-28T18:30:30.313 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-11-28T18:30:30.313 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before thrashing 2023-11-28T18:30:31.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62695b5640 1 Processor -- start 2023-11-28T18:30:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62695b5640 1 -- start start 2023-11-28T18:30:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62695b5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f626415ce10 0x7f62640ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62695b5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62640cafe0 0x7f62640ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62695b5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626415a3a0 con 0x7f626415ca30 2023-11-28T18:30:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62695b5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626415a620 con 0x7f626415ce10 2023-11-28T18:30:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62695b5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626415a900 con 0x7f62640cafe0 2023-11-28T18:30:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f6263fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f626415ca30 0x7f6264159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51182/0 (socket says 172.21.15.7:51182) 2023-11-28T18:30:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f6263fff640 1 -- 172.21.15.7:0/1103429124 learned_addr learned my addr 172.21.15.7:0/1103429124 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f6268db4640 1 --2- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62640cafe0 0x7f62640ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f625bfff640 1 --2- 172.21.15.7:0/1103429124 >> v2:172.21.15.193:3300/0 conn(0x7f626415ce10 0x7f62640ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62637fe640 1 -- 172.21.15.7:0/1103429124 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200458696 0 0) 0x7f626415a3a0 con 0x7f626415ca30 2023-11-28T18:30:31.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f62637fe640 1 -- 172.21.15.7:0/1103429124 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6250003530 con 0x7f626415ca30 2023-11-28T18:30:31.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f625bfff640 1 -- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62640cafe0 msgr2=0x7f62640ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:31.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f625bfff640 1 --2- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62640cafe0 0x7f62640ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f625bfff640 1 -- 172.21.15.7:0/1103429124 >> v1:172.21.15.7:6789/0 conn(0x7f626415ca30 legacy=0x7f6264159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f625bfff640 1 -- 172.21.15.7:0/1103429124 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626415abe0 con 0x7f626415ce10 2023-11-28T18:30:31.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.471+0000 7f6268db4640 1 --2- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62640cafe0 0x7f62640ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.475+0000 7f625bfff640 1 --2- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 0x7f62640ca1d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f624c0097c0 tx=0x7f624c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d4654c673b77b27 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.475+0000 7f62637fe640 1 -- 172.21.15.7:0/1103429124 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c0a8970 con 0x7f626415ce10 2023-11-28T18:30:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.475+0000 7f62637fe640 1 -- 172.21.15.7:0/1103429124 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c0a8ad0 con 0x7f626415ce10 2023-11-28T18:30:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.475+0000 7f62637fe640 1 -- 172.21.15.7:0/1103429124 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c0a8e60 con 0x7f626415ce10 2023-11-28T18:30:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 msgr2=0x7f62640ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 --2- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 0x7f62640ca1d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f624c0097c0 tx=0x7f624c01ec70 comp rx=0 tx=0).stop 2023-11-28T18:30:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- 172.21.15.7:0/1103429124 shutdown_connections 2023-11-28T18:30:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 --2- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62640cafe0 0x7f62640ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 --2- 172.21.15.7:0/1103429124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 0x7f62640ca1d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- 172.21.15.7:0/1103429124 >> 172.21.15.7:0/1103429124 conn(0x7f626406c060 msgr2=0x7f62640662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- 172.21.15.7:0/1103429124 shutdown_connections 2023-11-28T18:30:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- 172.21.15.7:0/1103429124 wait complete. 2023-11-28T18:30:31.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 Processor -- start 2023-11-28T18:30:31.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- start start 2023-11-28T18:30:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62640cafe0 0x7f6264154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f626415ca30 0x7f62640c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 0x7f62640c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641f3160 con 0x7f62640cafe0 2023-11-28T18:30:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641f3690 con 0x7f626415ce10 2023-11-28T18:30:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f62695b5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62641f33b0 con 0x7f626415ca30 2023-11-28T18:30:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6263fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62640cafe0 0x7f6264154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6263fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62640cafe0 0x7f6264154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55968/0 (socket says 172.21.15.7:55968) 2023-11-28T18:30:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6263fff640 1 -- 172.21.15.7:0/484840270 learned_addr learned my addr 172.21.15.7:0/484840270 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f625bfff640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f626415ca30 0x7f62640c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6268db4640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 0x7f62640c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6263fff640 1 -- 172.21.15.7:0/484840270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f626415ca30 msgr2=0x7f62640c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6263fff640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f626415ca30 0x7f62640c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6263fff640 1 -- 172.21.15.7:0/484840270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 msgr2=0x7f62640c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6263fff640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 0x7f62640c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.479+0000 7f6263fff640 1 -- 172.21.15.7:0/484840270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c009a70 con 0x7f62640cafe0 2023-11-28T18:30:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f6263fff640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62640cafe0 0x7f6264154860 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f625400c4d0 tx=0x7f625400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62617fa640 1 -- 172.21.15.7:0/484840270 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254005cf0 con 0x7f62640cafe0 2023-11-28T18:30:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62617fa640 1 -- 172.21.15.7:0/484840270 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6254005e50 con 0x7f62640cafe0 2023-11-28T18:30:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62617fa640 1 -- 172.21.15.7:0/484840270 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62540072d0 con 0x7f62640cafe0 2023-11-28T18:30:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624c009420 con 0x7f62640cafe0 2023-11-28T18:30:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6264158690 con 0x7f62640cafe0 2023-11-28T18:30:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6264158ce0 con 0x7f62640cafe0 2023-11-28T18:30:31.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62617fa640 1 -- 172.21.15.7:0/484840270 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6254007430 con 0x7f62640cafe0 2023-11-28T18:30:31.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62617fa640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440668d0 0x7f6244068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f62617fa640 1 -- 172.21.15.7:0/484840270 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f6254083b90 con 0x7f62640cafe0 2023-11-28T18:30:31.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.483+0000 7f625bfff640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440668d0 0x7f6244068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.487+0000 7f625bfff640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440668d0 0x7f6244068d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f624c0b2000 tx=0x7f624c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:31.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.491+0000 7f62617fa640 1 -- 172.21.15.7:0/484840270 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6254050c60 con 0x7f62640cafe0 2023-11-28T18:30:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.719+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62640c4640 con 0x7f62640cafe0 2023-11-28T18:30:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.719+0000 7f62617fa640 1 -- 172.21.15.7:0/484840270 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f6254017090 con 0x7f62640cafe0 2023-11-28T18:30:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:31.725 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440668d0 msgr2=0x7f6244068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440668d0 0x7f6244068d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f624c0b2000 tx=0x7f624c0aa070 comp rx=0 tx=0).stop 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62640cafe0 msgr2=0x7f6264154860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62640cafe0 0x7f6264154860 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f625400c4d0 tx=0x7f625400c8d0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 shutdown_connections 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440668d0 0x7f6244068d90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f626415ce10 0x7f62640c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f626415ca30 0x7f62640c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 --2- 172.21.15.7:0/484840270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62640cafe0 0x7f6264154860 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 >> 172.21.15.7:0/484840270 conn(0x7f626406c060 msgr2=0x7f62640cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 shutdown_connections 2023-11-28T18:30:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.723+0000 7f62695b5640 1 -- 172.21.15.7:0/484840270 wait complete. 2023-11-28T18:30:31.748 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=2) 2023-11-28T18:30:31.748 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T18:30:31.748 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:30:31.767 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T18:30:31.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:31.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f6a05a17640 1 Processor -- start 2023-11-28T18:30:31.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f6a05a17640 1 -- start start 2023-11-28T18:30:31.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f6a05a17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6a00154300 0x7f6a00159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f6a05a17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0015d490 con 0x7f6a0015e090 2023-11-28T18:30:31.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f6a05a17640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0015a480 con 0x7f6a0015dc50 2023-11-28T18:30:31.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f6a05a17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0015a760 con 0x7f6a00154300 2023-11-28T18:30:31.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69ff7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6a0015e090 0x7f6a000ca410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51194/0 (socket says 172.21.15.7:51194) 2023-11-28T18:30:31.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69ff7fe640 1 -- 172.21.15.7:0/3902540737 learned_addr learned my addr 172.21.15.7:0/3902540737 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:31.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69feffd640 1 --2- 172.21.15.7:0/3902540737 >> v2:172.21.15.193:3300/0 conn(0x7f6a00154300 0x7f6a00159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69fdffb640 1 -- 172.21.15.7:0/3902540737 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723597471 0 0) 0x7f6a0015d490 con 0x7f6a0015e090 2023-11-28T18:30:31.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69fdffb640 1 -- 172.21.15.7:0/3902540737 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e8003530 con 0x7f6a0015e090 2023-11-28T18:30:31.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69fdffb640 1 -- 172.21.15.7:0/3902540737 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194733660 0 0) 0x7f69e8003530 con 0x7f6a0015e090 2023-11-28T18:30:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69feffd640 1 -- 172.21.15.7:0/3902540737 >> v1:172.21.15.193:6789/0 conn(0x7f6a0015dc50 legacy=0x7f6a000cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69feffd640 1 -- 172.21.15.7:0/3902540737 >> v1:172.21.15.7:6789/0 conn(0x7f6a0015e090 legacy=0x7f6a000ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.935+0000 7f69feffd640 1 -- 172.21.15.7:0/3902540737 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0015aa40 con 0x7f6a00154300 2023-11-28T18:30:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.939+0000 7f69feffd640 1 --2- 172.21.15.7:0/3902540737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 0x7f6a00159e40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f69f00097c0 tx=0x7f69f001cdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1ca65ae76bfbe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.939+0000 7f69fdffb640 1 -- 172.21.15.7:0/3902540737 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f8001a00 con 0x7f6a00154300 2023-11-28T18:30:31.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.939+0000 7f69fdffb640 1 -- 172.21.15.7:0/3902540737 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f00ac440 con 0x7f6a00154300 2023-11-28T18:30:31.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.939+0000 7f69fdffb640 1 -- 172.21.15.7:0/3902540737 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f00ac7b0 con 0x7f6a00154300 2023-11-28T18:30:31.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.939+0000 7f6a05a17640 1 -- 172.21.15.7:0/3902540737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 msgr2=0x7f6a00159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.939+0000 7f6a05a17640 1 --2- 172.21.15.7:0/3902540737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 0x7f6a00159e40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f69f00097c0 tx=0x7f69f001cdd0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 -- 172.21.15.7:0/3902540737 shutdown_connections 2023-11-28T18:30:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 --2- 172.21.15.7:0/3902540737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 0x7f6a00159e40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 -- 172.21.15.7:0/3902540737 >> 172.21.15.7:0/3902540737 conn(0x7f6a000662e0 msgr2=0x7f6a000666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 -- 172.21.15.7:0/3902540737 shutdown_connections 2023-11-28T18:30:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 -- 172.21.15.7:0/3902540737 wait complete. 2023-11-28T18:30:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 Processor -- start 2023-11-28T18:30:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 -- start start 2023-11-28T18:30:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 0x7f6a000b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a0015dc50 0x7f6a00157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a0015e090 0x7f6a001581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00149470 con 0x7f6a0015e090 2023-11-28T18:30:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00148580 con 0x7f6a00154300 2023-11-28T18:30:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f6a05a17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00148860 con 0x7f6a0015dc50 2023-11-28T18:30:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a0015e090 0x7f6a001581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a0015e090 0x7f6a001581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55974/0 (socket says 172.21.15.7:55974) 2023-11-28T18:30:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.943+0000 7f69ff7fe640 1 -- 172.21.15.7:0/1965559263 learned_addr learned my addr 172.21.15.7:0/1965559263 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69feffd640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 0x7f6a000b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69fe7fc640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a0015dc50 0x7f6a00157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69ff7fe640 1 -- 172.21.15.7:0/1965559263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a0015dc50 msgr2=0x7f6a00157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69ff7fe640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a0015dc50 0x7f6a00157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69ff7fe640 1 -- 172.21.15.7:0/1965559263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 msgr2=0x7f6a000b4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69ff7fe640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 0x7f6a000b4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69ff7fe640 1 -- 172.21.15.7:0/1965559263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0009420 con 0x7f6a0015e090 2023-11-28T18:30:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69ff7fe640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a0015e090 0x7f6a001581c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f69f800cc00 tx=0x7f69f800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69e7fff640 1 -- 172.21.15.7:0/1965559263 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f800ce10 con 0x7f6a0015e090 2023-11-28T18:30:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69e7fff640 1 -- 172.21.15.7:0/1965559263 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f8012400 con 0x7f6a0015e090 2023-11-28T18:30:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69e7fff640 1 -- 172.21.15.7:0/1965559263 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80127b0 con 0x7f6a0015e090 2023-11-28T18:30:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a00148b40 con 0x7f6a0015e090 2023-11-28T18:30:31.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a00149100 con 0x7f6a0015e090 2023-11-28T18:30:31.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a001492c0 con 0x7f6a0015e090 2023-11-28T18:30:31.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.947+0000 7f69e7fff640 1 -- 172.21.15.7:0/1965559263 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f69f8012910 con 0x7f6a0015e090 2023-11-28T18:30:31.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.951+0000 7f69e7fff640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69ec066900 0x7f69ec068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:31.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.951+0000 7f69e7fff640 1 -- 172.21.15.7:0/1965559263 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f69f8084b30 con 0x7f6a0015e090 2023-11-28T18:30:31.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.951+0000 7f69feffd640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69ec066900 0x7f69ec068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:31.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.951+0000 7f69feffd640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69ec066900 0x7f69ec068dc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f69f00ae000 tx=0x7f69f0004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:31.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:31.955+0000 7f69e7fff640 1 -- 172.21.15.7:0/1965559263 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f69f8051c90 con 0x7f6a0015e090 2023-11-28T18:30:32.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.151+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a0004e890 con 0x7f6a0015e090 2023-11-28T18:30:32.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.151+0000 7f69e7fff640 1 -- 172.21.15.7:0/1965559263 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f69f80552b0 con 0x7f6a0015e090 2023-11-28T18:30:32.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:32.157 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:32.157 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69ec066900 msgr2=0x7f69ec068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69ec066900 0x7f69ec068dc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f69f00ae000 tx=0x7f69f0004740 comp rx=0 tx=0).stop 2023-11-28T18:30:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a0015e090 msgr2=0x7f6a001581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a0015e090 0x7f6a001581c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f69f800cc00 tx=0x7f69f800a430 comp rx=0 tx=0).stop 2023-11-28T18:30:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 shutdown_connections 2023-11-28T18:30:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69ec066900 0x7f69ec068dc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a0015e090 0x7f6a001581c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6a0015dc50 0x7f6a00157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 --2- 172.21.15.7:0/1965559263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00154300 0x7f6a000b4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 >> 172.21.15.7:0/1965559263 conn(0x7f6a000662e0 msgr2=0x7f6a000ce840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 shutdown_connections 2023-11-28T18:30:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:32.155+0000 7f6a05a17640 1 -- 172.21.15.7:0/1965559263 wait complete. 2023-11-28T18:30:32.179 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:30:34.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:34.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f8910fd0640 1 Processor -- start 2023-11-28T18:30:34.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f8910fd0640 1 -- start start 2023-11-28T18:30:34.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f8910fd0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c071210 0x7f890c153b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:34.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f8910fd0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f890c0715f0 0x7f890c154060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:34.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f8910fd0640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f890c0c7fc0 0x7f890c15d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:34.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f8910fd0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c15a900 con 0x7f890c0715f0 2023-11-28T18:30:34.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f8910fd0640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c073060 con 0x7f890c0c7fc0 2023-11-28T18:30:34.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f8910fd0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c0732d0 con 0x7f890c071210 2023-11-28T18:30:34.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f890c0715f0 0x7f890c154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:34.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f890c0715f0 0x7f890c154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45746/0 (socket says 172.21.15.7:45746) 2023-11-28T18:30:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890affd640 1 -- 172.21.15.7:0/3382927937 learned_addr learned my addr 172.21.15.7:0/3382927937 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890b7fe640 1 --2- 172.21.15.7:0/3382927937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c071210 0x7f890c153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890bfff640 1 --2- 172.21.15.7:0/3382927937 >> v2:172.21.15.193:3300/0 conn(0x7f890c0c7fc0 0x7f890c15d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890affd640 1 -- 172.21.15.7:0/3382927937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c071210 msgr2=0x7f890c153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890affd640 1 --2- 172.21.15.7:0/3382927937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c071210 0x7f890c153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890affd640 1 -- 172.21.15.7:0/3382927937 >> v2:172.21.15.193:3300/0 conn(0x7f890c0c7fc0 msgr2=0x7f890c15d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890affd640 1 --2- 172.21.15.7:0/3382927937 >> v2:172.21.15.193:3300/0 conn(0x7f890c0c7fc0 0x7f890c15d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.335+0000 7f890affd640 1 -- 172.21.15.7:0/3382927937 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c159fd0 con 0x7f890c0715f0 2023-11-28T18:30:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f890affd640 1 --2- 172.21.15.7:0/3382927937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c0715f0 0x7f890c154060 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f890c1545a0 tx=0x7f890001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1136fd8e9da58a00 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f890a7fc640 1 -- 172.21.15.7:0/3382927937 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000a89b0 con 0x7f890c0715f0 2023-11-28T18:30:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f890a7fc640 1 -- 172.21.15.7:0/3382927937 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89000a8b10 con 0x7f890c0715f0 2023-11-28T18:30:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f890a7fc640 1 -- 172.21.15.7:0/3382927937 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000a8ea0 con 0x7f890c0715f0 2023-11-28T18:30:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 -- 172.21.15.7:0/3382927937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c0715f0 msgr2=0x7f890c154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 --2- 172.21.15.7:0/3382927937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c0715f0 0x7f890c154060 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f890c1545a0 tx=0x7f890001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 -- 172.21.15.7:0/3382927937 shutdown_connections 2023-11-28T18:30:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 --2- 172.21.15.7:0/3382927937 >> v2:172.21.15.193:3300/0 conn(0x7f890c0c7fc0 0x7f890c15d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 --2- 172.21.15.7:0/3382927937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c0715f0 0x7f890c154060 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 --2- 172.21.15.7:0/3382927937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c071210 0x7f890c153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 -- 172.21.15.7:0/3382927937 >> 172.21.15.7:0/3382927937 conn(0x7f890c06a5f0 msgr2=0x7f890c06aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 -- 172.21.15.7:0/3382927937 shutdown_connections 2023-11-28T18:30:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.339+0000 7f8910fd0640 1 -- 172.21.15.7:0/3382927937 wait complete. 2023-11-28T18:30:34.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f8910fd0640 1 Processor -- start 2023-11-28T18:30:34.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f8910fd0640 1 -- start start 2023-11-28T18:30:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f8910fd0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c071210 0x7f890c1f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f8910fd0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c0715f0 0x7f890c13e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f8910fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f890c0c7fc0 0x7f890c13ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f8910fd0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c150eb0 con 0x7f890c071210 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f8910fd0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c1f0620 con 0x7f890c0c7fc0 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f8910fd0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c1557e0 con 0x7f890c0715f0 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c071210 0x7f890c1f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c071210 0x7f890c1f00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45756/0 (socket says 172.21.15.7:45756) 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 -- 172.21.15.7:0/1620537310 learned_addr learned my addr 172.21.15.7:0/1620537310 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890affd640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c0715f0 0x7f890c13e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890bfff640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f890c0c7fc0 0x7f890c13ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 -- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c0715f0 msgr2=0x7f890c13e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c0715f0 0x7f890c13e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 -- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f890c0c7fc0 msgr2=0x7f890c13ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f890c0c7fc0 0x7f890c13ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 -- 172.21.15.7:0/1620537310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89000094d0 con 0x7f890c071210 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890bfff640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f890c0c7fc0 0x7f890c13ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.343+0000 7f890b7fe640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c071210 0x7f890c1f00e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00e8e0 tx=0x7f88fc00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f88f3fff640 1 -- 172.21.15.7:0/1620537310 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc00c9a0 con 0x7f890c071210 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f88f3fff640 1 -- 172.21.15.7:0/1620537310 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc00cb00 con 0x7f890c071210 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f88f3fff640 1 -- 172.21.15.7:0/1620537310 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc00ce90 con 0x7f890c071210 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c150350 con 0x7f890c071210 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c1508f0 con 0x7f890c071210 2023-11-28T18:30:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890c155640 con 0x7f890c071210 2023-11-28T18:30:34.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f88f3fff640 1 -- 172.21.15.7:0/1620537310 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f88fc013470 con 0x7f890c071210 2023-11-28T18:30:34.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f88f3fff640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88f80668d0 0x7f88f8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:34.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f88f3fff640 1 -- 172.21.15.7:0/1620537310 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f88fc084810 con 0x7f890c071210 2023-11-28T18:30:34.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.347+0000 7f890affd640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88f80668d0 0x7f88f8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:34.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.351+0000 7f890affd640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88f80668d0 0x7f88f8068d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f89000b2000 tx=0x7f8900008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.355+0000 7f88f3fff640 1 -- 172.21.15.7:0/1620537310 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f890c155640 con 0x7f890c071210 2023-11-28T18:30:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.563+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f890c155640 con 0x7f890c071210 2023-11-28T18:30:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f88f3fff640 1 -- 172.21.15.7:0/1620537310 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f890c155640 con 0x7f890c071210 2023-11-28T18:30:34.570 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:34.570 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88f80668d0 msgr2=0x7f88f8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88f80668d0 0x7f88f8068d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f89000b2000 tx=0x7f8900008040 comp rx=0 tx=0).stop 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c071210 msgr2=0x7f890c1f00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c071210 0x7f890c1f00e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00e8e0 tx=0x7f88fc00eda0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 shutdown_connections 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f88f80668d0 0x7f88f8068d90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f890c0c7fc0 0x7f890c13ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f890c0715f0 0x7f890c13e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 --2- 172.21.15.7:0/1620537310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c071210 0x7f890c1f00e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 >> 172.21.15.7:0/1620537310 conn(0x7f890c06a5f0 msgr2=0x7f890c1582f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 shutdown_connections 2023-11-28T18:30:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:34.567+0000 7f8910fd0640 1 -- 172.21.15.7:0/1620537310 wait complete. 2023-11-28T18:30:34.591 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:30:36.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:36.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f355d0640 1 Processor -- start 2023-11-28T18:30:36.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f355d0640 1 -- start start 2023-11-28T18:30:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f355d0640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6f3015ca30 0x7f6f30159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f355d0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6f3015ce10 0x7f6f300ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f355d0640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3015a620 con 0x7f6f300cafe0 2023-11-28T18:30:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f355d0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f3015a900 con 0x7f6f3015ce10 2023-11-28T18:30:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f355d0640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f3015abe0 con 0x7f6f3015ca30 2023-11-28T18:30:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f27fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6f3015ce10 0x7f6f300ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f27fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6f3015ce10 0x7f6f300ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45760/0 (socket says 172.21.15.7:45760) 2023-11-28T18:30:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f27fff640 1 -- 172.21.15.7:0/1296703403 learned_addr learned my addr 172.21.15.7:0/1296703403 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f2ffff640 1 --2- 172.21.15.7:0/1296703403 >> v2:172.21.15.193:3300/0 conn(0x7f6f3015ca30 0x7f6f30159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f27fff640 1 -- 172.21.15.7:0/1296703403 >> v2:172.21.15.193:3300/0 conn(0x7f6f3015ca30 msgr2=0x7f6f30159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f27fff640 1 --2- 172.21.15.7:0/1296703403 >> v2:172.21.15.193:3300/0 conn(0x7f6f3015ca30 0x7f6f30159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f27fff640 1 -- 172.21.15.7:0/1296703403 >> v1:172.21.15.193:6789/0 conn(0x7f6f300cafe0 legacy=0x7f6f300cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.751+0000 7f6f27fff640 1 -- 172.21.15.7:0/1296703403 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f300cbb10 con 0x7f6f3015ce10 2023-11-28T18:30:36.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f27fff640 1 --2- 172.21.15.7:0/1296703403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f3015ce10 0x7f6f300ca1d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6f3006c660 tx=0x7f6f1801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bfcf46928352c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:36.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f2f7fe640 1 -- 172.21.15.7:0/1296703403 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f180ac640 con 0x7f6f3015ce10 2023-11-28T18:30:36.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f2f7fe640 1 -- 172.21.15.7:0/1296703403 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f180ac7a0 con 0x7f6f3015ce10 2023-11-28T18:30:36.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f2f7fe640 1 -- 172.21.15.7:0/1296703403 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f180acb30 con 0x7f6f3015ce10 2023-11-28T18:30:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f355d0640 1 -- 172.21.15.7:0/1296703403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f3015ce10 msgr2=0x7f6f300ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1296703403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f3015ce10 0x7f6f300ca1d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6f3006c660 tx=0x7f6f1801c4d0 comp rx=0 tx=0).stop 2023-11-28T18:30:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f355d0640 1 -- 172.21.15.7:0/1296703403 shutdown_connections 2023-11-28T18:30:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1296703403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f3015ce10 0x7f6f300ca1d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1296703403 >> v2:172.21.15.193:3300/0 conn(0x7f6f3015ca30 0x7f6f30159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f355d0640 1 -- 172.21.15.7:0/1296703403 >> 172.21.15.7:0/1296703403 conn(0x7f6f3006c060 msgr2=0x7f6f300662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f355d0640 1 -- 172.21.15.7:0/1296703403 shutdown_connections 2023-11-28T18:30:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.755+0000 7f6f355d0640 1 -- 172.21.15.7:0/1296703403 wait complete. 2023-11-28T18:30:36.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f355d0640 1 Processor -- start 2023-11-28T18:30:36.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f355d0640 1 -- start start 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f355d0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f300cafe0 0x7f6f300c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f355d0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f3015ca30 0x7f6f30149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f355d0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f3015ce10 0x7f6f301f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f355d0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f301f6d10 con 0x7f6f300cafe0 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f355d0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f30154800 con 0x7f6f3015ce10 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f355d0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f300c8cb0 con 0x7f6f3015ca30 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f2ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f300cafe0 0x7f6f300c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f2ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f300cafe0 0x7f6f300c8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45770/0 (socket says 172.21.15.7:45770) 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f2ffff640 1 -- 172.21.15.7:0/1749690472 learned_addr learned my addr 172.21.15.7:0/1749690472 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:36.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f34dcf640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f3015ce10 0x7f6f301f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:36.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f27fff640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f3015ca30 0x7f6f30149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:36.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f2ffff640 1 -- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f3015ca30 msgr2=0x7f6f30149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:36.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f2ffff640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f3015ca30 0x7f6f30149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:36.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f2ffff640 1 -- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f3015ce10 msgr2=0x7f6f301f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:36.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f2ffff640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f3015ce10 0x7f6f301f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:36.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f2ffff640 1 -- 172.21.15.7:0/1749690472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f180094d0 con 0x7f6f300cafe0 2023-11-28T18:30:36.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.759+0000 7f6f34dcf640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f3015ce10 0x7f6f301f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:30:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f2ffff640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f300cafe0 0x7f6f300c8770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000e870 tx=0x7f6f2000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f2d7fa640 1 -- 172.21.15.7:0/1749690472 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20009510 con 0x7f6f300cafe0 2023-11-28T18:30:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f2d7fa640 1 -- 172.21.15.7:0/1749690472 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2000eec0 con 0x7f6f300cafe0 2023-11-28T18:30:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f2d7fa640 1 -- 172.21.15.7:0/1749690472 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2000cb40 con 0x7f6f300cafe0 2023-11-28T18:30:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f30153800 con 0x7f6f300cafe0 2023-11-28T18:30:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f300c7490 con 0x7f6f300cafe0 2023-11-28T18:30:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f300c72f0 con 0x7f6f300cafe0 2023-11-28T18:30:36.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f2d7fa640 1 -- 172.21.15.7:0/1749690472 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6f2000cca0 con 0x7f6f300cafe0 2023-11-28T18:30:36.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.763+0000 7f6f2d7fa640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f100668d0 0x7f6f10068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:36.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.767+0000 7f6f2d7fa640 1 -- 172.21.15.7:0/1749690472 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f6f20084750 con 0x7f6f300cafe0 2023-11-28T18:30:36.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.767+0000 7f6f27fff640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f100668d0 0x7f6f10068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.767+0000 7f6f27fff640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f100668d0 0x7f6f10068d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6f180ae000 tx=0x7f6f180a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:36.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.771+0000 7f6f2d7fa640 1 -- 172.21.15.7:0/1749690472 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6f300c72f0 con 0x7f6f300cafe0 2023-11-28T18:30:37.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.999+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f300c72f0 con 0x7f6f300cafe0 2023-11-28T18:30:37.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:36.999+0000 7f6f2d7fa640 1 -- 172.21.15.7:0/1749690472 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f6f300c72f0 con 0x7f6f300cafe0 2023-11-28T18:30:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:37.005 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:37.005 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:37.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f100668d0 msgr2=0x7f6f10068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:37.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f100668d0 0x7f6f10068d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6f180ae000 tx=0x7f6f180a7000 comp rx=0 tx=0).stop 2023-11-28T18:30:37.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f300cafe0 msgr2=0x7f6f300c8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:37.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f300cafe0 0x7f6f300c8770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000e870 tx=0x7f6f2000ec70 comp rx=0 tx=0).stop 2023-11-28T18:30:37.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 shutdown_connections 2023-11-28T18:30:37.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6f100668d0 0x7f6f10068d90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:37.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f3015ce10 0x7f6f301f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:37.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6f3015ca30 0x7f6f30149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:37.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 --2- 172.21.15.7:0/1749690472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f300cafe0 0x7f6f300c8770 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:37.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 >> 172.21.15.7:0/1749690472 conn(0x7f6f3006c060 msgr2=0x7f6f301578f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:37.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 shutdown_connections 2023-11-28T18:30:37.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:37.003+0000 7f6f355d0640 1 -- 172.21.15.7:0/1749690472 wait complete. 2023-11-28T18:30:37.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:30:39.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:39.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.195+0000 7f346f0dd640 1 Processor -- start 2023-11-28T18:30:39.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.195+0000 7f346f0dd640 1 -- start start 2023-11-28T18:30:39.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346f0dd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f34681542e0 0x7f346815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:39.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346f0dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f34680728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:39.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346f0dd640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34680ca920 con 0x7f34680cbb70 2023-11-28T18:30:39.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346f0dd640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3468071d50 con 0x7f34681542e0 2023-11-28T18:30:39.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346f0dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3468072030 con 0x7f3468072450 2023-11-28T18:30:39.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f3467fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f34680cbb70 0x7f34680cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35576/0 (socket says 172.21.15.7:35576) 2023-11-28T18:30:39.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346d67a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f34680728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:39.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f3467fff640 1 -- 172.21.15.7:0/3352665885 learned_addr learned my addr 172.21.15.7:0/3352665885 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:39.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346ce79640 1 --2- 172.21.15.7:0/3352665885 >> v2:172.21.15.193:3300/0 conn(0x7f34681542e0 0x7f346815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:39.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f34677fe640 1 -- 172.21.15.7:0/3352665885 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826691198 0 0) 0x7f34680ca920 con 0x7f34680cbb70 2023-11-28T18:30:39.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f34677fe640 1 -- 172.21.15.7:0/3352665885 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3458003530 con 0x7f34680cbb70 2023-11-28T18:30:39.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346ce79640 1 -- 172.21.15.7:0/3352665885 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 msgr2=0x7f34680728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:39.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346ce79640 1 --2- 172.21.15.7:0/3352665885 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f34680728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346ce79640 1 -- 172.21.15.7:0/3352665885 >> v1:172.21.15.193:6789/0 conn(0x7f34680cbb70 legacy=0x7f34680cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:39.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.199+0000 7f346ce79640 1 -- 172.21.15.7:0/3352665885 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346815e550 con 0x7f34681542e0 2023-11-28T18:30:39.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.203+0000 7f346d67a640 1 --2- 172.21.15.7:0/3352665885 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f34680728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:39.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.203+0000 7f346ce79640 1 --2- 172.21.15.7:0/3352665885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 0x7f346815e010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f345c0097f0 tx=0x7f345c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85097824cd39bc0e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:39.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f34677fe640 1 -- 172.21.15.7:0/3352665885 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c0ad640 con 0x7f34681542e0 2023-11-28T18:30:39.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f34677fe640 1 -- 172.21.15.7:0/3352665885 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c0ad7a0 con 0x7f34681542e0 2023-11-28T18:30:39.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f34677fe640 1 -- 172.21.15.7:0/3352665885 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c0adb30 con 0x7f34681542e0 2023-11-28T18:30:39.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 -- 172.21.15.7:0/3352665885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 msgr2=0x7f346815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:39.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 --2- 172.21.15.7:0/3352665885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 0x7f346815e010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f345c0097f0 tx=0x7f345c01c4a0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 -- 172.21.15.7:0/3352665885 shutdown_connections 2023-11-28T18:30:39.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 --2- 172.21.15.7:0/3352665885 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f34680728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 --2- 172.21.15.7:0/3352665885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 0x7f346815e010 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 -- 172.21.15.7:0/3352665885 >> 172.21.15.7:0/3352665885 conn(0x7f34680717e0 msgr2=0x7f34680662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:39.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 -- 172.21.15.7:0/3352665885 shutdown_connections 2023-11-28T18:30:39.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 -- 172.21.15.7:0/3352665885 wait complete. 2023-11-28T18:30:39.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 Processor -- start 2023-11-28T18:30:39.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.207+0000 7f346f0dd640 1 -- start start 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f346815be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34680cbb70 0x7f346814c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 0x7f346815c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346815ba30 con 0x7f34680cbb70 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34680d0580 con 0x7f34681542e0 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34680d0140 con 0x7f3468072450 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34680cbb70 0x7f346814c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34680cbb70 0x7f346814c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45778/0 (socket says 172.21.15.7:45778) 2023-11-28T18:30:39.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 -- 172.21.15.7:0/4157662946 learned_addr learned my addr 172.21.15.7:0/4157662946 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:39.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346ce79640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f346815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:39.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346d67a640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 0x7f346815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:39.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 -- 172.21.15.7:0/4157662946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 msgr2=0x7f346815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:39.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f346815be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 -- 172.21.15.7:0/4157662946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 msgr2=0x7f346815c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:39.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 0x7f346815c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 -- 172.21.15.7:0/4157662946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460009a70 con 0x7f34680cbb70 2023-11-28T18:30:39.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f3467fff640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34680cbb70 0x7f346814c8d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f345400c4d0 tx=0x7f345400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:39.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f34657fa640 1 -- 172.21.15.7:0/4157662946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454005cf0 con 0x7f34680cbb70 2023-11-28T18:30:39.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f34657fa640 1 -- 172.21.15.7:0/4157662946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3454005e50 con 0x7f34680cbb70 2023-11-28T18:30:39.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345c009420 con 0x7f34680cbb70 2023-11-28T18:30:39.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f34657fa640 1 -- 172.21.15.7:0/4157662946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34540072d0 con 0x7f34680cbb70 2023-11-28T18:30:39.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34680c9c10 con 0x7f34680cbb70 2023-11-28T18:30:39.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346806c670 con 0x7f34680cbb70 2023-11-28T18:30:39.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f34657fa640 1 -- 172.21.15.7:0/4157662946 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3454007430 con 0x7f34680cbb70 2023-11-28T18:30:39.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f34657fa640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f344c0668d0 0x7f344c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:39.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f34657fa640 1 -- 172.21.15.7:0/4157662946 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f3454083a60 con 0x7f34680cbb70 2023-11-28T18:30:39.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346ce79640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f344c0668d0 0x7f344c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:39.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.215+0000 7f346ce79640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f344c0668d0 0x7f344c068d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f345c0af000 tx=0x7f345c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:39.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.219+0000 7f34657fa640 1 -- 172.21.15.7:0/4157662946 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3454050c60 con 0x7f34680cbb70 2023-11-28T18:30:39.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.439+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f346815bb70 con 0x7f34680cbb70 2023-11-28T18:30:39.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.439+0000 7f34657fa640 1 -- 172.21.15.7:0/4157662946 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f346815bb70 con 0x7f34680cbb70 2023-11-28T18:30:39.444 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:39.445 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:39.445 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:39.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f344c0668d0 msgr2=0x7f344c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:39.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f344c0668d0 0x7f344c068d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f345c0af000 tx=0x7f345c008040 comp rx=0 tx=0).stop 2023-11-28T18:30:39.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34680cbb70 msgr2=0x7f346814c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:39.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34680cbb70 0x7f346814c8d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f345400c4d0 tx=0x7f345400c8d0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 shutdown_connections 2023-11-28T18:30:39.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f344c0668d0 0x7f344c068d90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34681542e0 0x7f346815c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34680cbb70 0x7f346814c8d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 --2- 172.21.15.7:0/4157662946 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3468072450 0x7f346815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:39.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 >> 172.21.15.7:0/4157662946 conn(0x7f34680717e0 msgr2=0x7f34681583b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:39.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 shutdown_connections 2023-11-28T18:30:39.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:39.443+0000 7f346f0dd640 1 -- 172.21.15.7:0/4157662946 wait complete. 2023-11-28T18:30:39.469 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:30:41.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ef417640 1 Processor -- start 2023-11-28T18:30:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ef417640 1 -- start start 2023-11-28T18:30:41.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ef417640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f25e815ce10 0x7f25e80ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:41.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ef417640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e815a3a0 con 0x7f25e80cafe0 2023-11-28T18:30:41.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ef417640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e815a620 con 0x7f25e815ca30 2023-11-28T18:30:41.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ef417640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e815a900 con 0x7f25e815ce10 2023-11-28T18:30:41.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25edc14640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f25e815ce10 0x7f25e80ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:41.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25eec16640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f25e80cafe0 0x7f25e80cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36588/0 (socket says 172.21.15.7:36588) 2023-11-28T18:30:41.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25eec16640 1 -- 172.21.15.7:0/751908463 learned_addr learned my addr 172.21.15.7:0/751908463 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:41.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ed413640 1 -- 172.21.15.7:0/751908463 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259737885 0 0) 0x7f25e815a3a0 con 0x7f25e80cafe0 2023-11-28T18:30:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ed413640 1 -- 172.21.15.7:0/751908463 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d4003530 con 0x7f25e80cafe0 2023-11-28T18:30:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25edc14640 1 -- 172.21.15.7:0/751908463 >> v1:172.21.15.193:6789/0 conn(0x7f25e815ca30 legacy=0x7f25e8159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25ed413640 1 -- 172.21.15.7:0/751908463 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116018292 0 0) 0x7f25e815a620 con 0x7f25e815ca30 2023-11-28T18:30:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25edc14640 1 -- 172.21.15.7:0/751908463 >> v1:172.21.15.7:6789/0 conn(0x7f25e80cafe0 legacy=0x7f25e80cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.619+0000 7f25edc14640 1 -- 172.21.15.7:0/751908463 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e815abe0 con 0x7f25e815ce10 2023-11-28T18:30:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.623+0000 7f25edc14640 1 --2- 172.21.15.7:0/751908463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e815ce10 0x7f25e80ca1d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097c0 tx=0x7f25d801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c16f8c34ff00fd3a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.623+0000 7f25ed413640 1 -- 172.21.15.7:0/751908463 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d80ad660 con 0x7f25e815ce10 2023-11-28T18:30:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.623+0000 7f25ed413640 1 -- 172.21.15.7:0/751908463 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d80ad7c0 con 0x7f25e815ce10 2023-11-28T18:30:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.623+0000 7f25ed413640 1 -- 172.21.15.7:0/751908463 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d80adb50 con 0x7f25e815ce10 2023-11-28T18:30:41.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.623+0000 7f25ef417640 1 -- 172.21.15.7:0/751908463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e815ce10 msgr2=0x7f25e80ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:41.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.623+0000 7f25ef417640 1 --2- 172.21.15.7:0/751908463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e815ce10 0x7f25e80ca1d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097c0 tx=0x7f25d801c450 comp rx=0 tx=0).stop 2023-11-28T18:30:41.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 -- 172.21.15.7:0/751908463 shutdown_connections 2023-11-28T18:30:41.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 --2- 172.21.15.7:0/751908463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e815ce10 0x7f25e80ca1d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:41.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 -- 172.21.15.7:0/751908463 >> 172.21.15.7:0/751908463 conn(0x7f25e806c060 msgr2=0x7f25e80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:41.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 -- 172.21.15.7:0/751908463 shutdown_connections 2023-11-28T18:30:41.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 -- 172.21.15.7:0/751908463 wait complete. 2023-11-28T18:30:41.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 Processor -- start 2023-11-28T18:30:41.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 -- start start 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80cafe0 0x7f25e8151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e815ca30 0x7f25e80724e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f25e815ce10 0x7f25e814ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e80c46e0 con 0x7f25e815ca30 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e8150600 con 0x7f25e80cafe0 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ef417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e81508e0 con 0x7f25e815ce10 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25edc14640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e815ca30 0x7f25e80724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25edc14640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e815ca30 0x7f25e80724e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45788/0 (socket says 172.21.15.7:45788) 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25edc14640 1 -- 172.21.15.7:0/1386929393 learned_addr learned my addr 172.21.15.7:0/1386929393 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ee415640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80cafe0 0x7f25e8151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:41.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25eec16640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f25e815ce10 0x7f25e814ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:41.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25edc14640 1 -- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f25e815ce10 msgr2=0x7f25e814ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25edc14640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f25e815ce10 0x7f25e814ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25edc14640 1 -- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80cafe0 msgr2=0x7f25e8151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25edc14640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80cafe0 0x7f25e8151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25edc14640 1 -- 172.21.15.7:0/1386929393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d8009420 con 0x7f25e815ca30 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.627+0000 7f25ee415640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80cafe0 0x7f25e8151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25edc14640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e815ca30 0x7f25e80724e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f25d80b0040 tx=0x7f25d801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25deffd640 1 -- 172.21.15.7:0/1386929393 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8004020 con 0x7f25e815ca30 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25deffd640 1 -- 172.21.15.7:0/1386929393 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d8004180 con 0x7f25e815ca30 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25deffd640 1 -- 172.21.15.7:0/1386929393 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d80044f0 con 0x7f25e815ca30 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e8150bc0 con 0x7f25e815ca30 2023-11-28T18:30:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e80c49c0 con 0x7f25e815ca30 2023-11-28T18:30:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e804e890 con 0x7f25e815ca30 2023-11-28T18:30:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25deffd640 1 -- 172.21.15.7:0/1386929393 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f25d8003620 con 0x7f25e815ca30 2023-11-28T18:30:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25deffd640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f25cc0668d0 0x7f25cc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.631+0000 7f25deffd640 1 -- 172.21.15.7:0/1386929393 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f25d811e9c0 con 0x7f25e815ca30 2023-11-28T18:30:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.635+0000 7f25ee415640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f25cc0668d0 0x7f25cc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:41.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.635+0000 7f25ee415640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f25cc0668d0 0x7f25cc068d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f25e4007fd0 tx=0x7f25e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:41.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.639+0000 7f25deffd640 1 -- 172.21.15.7:0/1386929393 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f25d80ebbc0 con 0x7f25e815ca30 2023-11-28T18:30:41.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.847+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25e8165400 con 0x7f25e815ca30 2023-11-28T18:30:41.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.847+0000 7f25deffd640 1 -- 172.21.15.7:0/1386929393 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f25d80ef1e0 con 0x7f25e815ca30 2023-11-28T18:30:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:41.852 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:41.852 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.847+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f25cc0668d0 msgr2=0x7f25cc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.847+0000 7f25ef417640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f25cc0668d0 0x7f25cc068d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f25e4007fd0 tx=0x7f25e4009040 comp rx=0 tx=0).stop 2023-11-28T18:30:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.847+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e815ca30 msgr2=0x7f25e80724e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.847+0000 7f25ef417640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e815ca30 0x7f25e80724e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f25d80b0040 tx=0x7f25d801ac90 comp rx=0 tx=0).stop 2023-11-28T18:30:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.847+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 shutdown_connections 2023-11-28T18:30:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.851+0000 7f25ef417640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f25cc0668d0 0x7f25cc068d90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.851+0000 7f25ef417640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f25e815ce10 0x7f25e814ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.851+0000 7f25ef417640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e815ca30 0x7f25e80724e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.851+0000 7f25ef417640 1 --2- 172.21.15.7:0/1386929393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80cafe0 0x7f25e8151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.851+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 >> 172.21.15.7:0/1386929393 conn(0x7f25e806c060 msgr2=0x7f25e80c89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.851+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 shutdown_connections 2023-11-28T18:30:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:41.851+0000 7f25ef417640 1 -- 172.21.15.7:0/1386929393 wait complete. 2023-11-28T18:30:41.877 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:30:43.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:44.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa759e640 1 Processor -- start 2023-11-28T18:30:44.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa759e640 1 -- start start 2023-11-28T18:30:44.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa759e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2aa815ce10 0x7f2aa80ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:44.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa759e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa815a3a0 con 0x7f2aa80cafe0 2023-11-28T18:30:44.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa759e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa815a620 con 0x7f2aa815ca30 2023-11-28T18:30:44.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa759e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa815a900 con 0x7f2aa815ce10 2023-11-28T18:30:44.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa6d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2aa80cafe0 0x7f2aa80cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35936/0 (socket says 172.21.15.7:35936) 2023-11-28T18:30:44.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa6d9d640 1 -- 172.21.15.7:0/600249393 learned_addr learned my addr 172.21.15.7:0/600249393 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:44.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa5d9b640 1 --2- 172.21.15.7:0/600249393 >> v2:172.21.15.193:3300/0 conn(0x7f2aa815ce10 0x7f2aa80ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:44.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa559a640 1 -- 172.21.15.7:0/600249393 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988374149 0 0) 0x7f2aa815a3a0 con 0x7f2aa80cafe0 2023-11-28T18:30:44.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa559a640 1 -- 172.21.15.7:0/600249393 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a90003530 con 0x7f2aa80cafe0 2023-11-28T18:30:44.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.027+0000 7f2aa5d9b640 1 -- 172.21.15.7:0/600249393 >> v1:172.21.15.193:6789/0 conn(0x7f2aa815ca30 legacy=0x7f2aa8159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:44.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa559a640 1 -- 172.21.15.7:0/600249393 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990772767 0 0) 0x7f2aa815a620 con 0x7f2aa815ca30 2023-11-28T18:30:44.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa5d9b640 1 -- 172.21.15.7:0/600249393 >> v1:172.21.15.7:6789/0 conn(0x7f2aa80cafe0 legacy=0x7f2aa80cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:44.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa5d9b640 1 -- 172.21.15.7:0/600249393 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa815abe0 con 0x7f2aa815ce10 2023-11-28T18:30:44.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa5d9b640 1 --2- 172.21.15.7:0/600249393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ce10 0x7f2aa80ca1d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0097c0 tx=0x7f2a9c01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7763ca47935cd1e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:44.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa559a640 1 -- 172.21.15.7:0/600249393 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c0ad660 con 0x7f2aa815ce10 2023-11-28T18:30:44.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa559a640 1 -- 172.21.15.7:0/600249393 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c0ad7c0 con 0x7f2aa815ce10 2023-11-28T18:30:44.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa559a640 1 -- 172.21.15.7:0/600249393 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c0adb50 con 0x7f2aa815ce10 2023-11-28T18:30:44.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa759e640 1 -- 172.21.15.7:0/600249393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ce10 msgr2=0x7f2aa80ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:44.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.031+0000 7f2aa759e640 1 --2- 172.21.15.7:0/600249393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ce10 0x7f2aa80ca1d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0097c0 tx=0x7f2a9c01c450 comp rx=0 tx=0).stop 2023-11-28T18:30:44.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 -- 172.21.15.7:0/600249393 shutdown_connections 2023-11-28T18:30:44.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 --2- 172.21.15.7:0/600249393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ce10 0x7f2aa80ca1d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:44.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 -- 172.21.15.7:0/600249393 >> 172.21.15.7:0/600249393 conn(0x7f2aa806c060 msgr2=0x7f2aa80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:44.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 -- 172.21.15.7:0/600249393 shutdown_connections 2023-11-28T18:30:44.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 -- 172.21.15.7:0/600249393 wait complete. 2023-11-28T18:30:44.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 Processor -- start 2023-11-28T18:30:44.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 -- start start 2023-11-28T18:30:44.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2aa80cafe0 0x7f2aa80c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:44.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ca30 0x7f2aa80c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:44.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2aa815ce10 0x7f2aa80c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:44.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80c5dd0 con 0x7f2aa815ce10 2023-11-28T18:30:44.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa81f5fd0 con 0x7f2aa815ca30 2023-11-28T18:30:44.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa759e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa8155c90 con 0x7f2aa80cafe0 2023-11-28T18:30:44.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2aa815ce10 0x7f2aa80c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:44.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2aa815ce10 0x7f2aa80c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54538/0 (socket says 172.21.15.7:54538) 2023-11-28T18:30:44.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa6d9d640 1 -- 172.21.15.7:0/2661876425 learned_addr learned my addr 172.21.15.7:0/2661876425 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:44.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa659c640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2aa80cafe0 0x7f2aa80c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:44.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.035+0000 7f2aa5d9b640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ca30 0x7f2aa80c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:44.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa6d9d640 1 -- 172.21.15.7:0/2661876425 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2aa80cafe0 msgr2=0x7f2aa80c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:44.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa6d9d640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2aa80cafe0 0x7f2aa80c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:44.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa6d9d640 1 -- 172.21.15.7:0/2661876425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ca30 msgr2=0x7f2aa80c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:44.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa6d9d640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ca30 0x7f2aa80c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:44.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa6d9d640 1 -- 172.21.15.7:0/2661876425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c009420 con 0x7f2aa815ce10 2023-11-28T18:30:44.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa6d9d640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2aa815ce10 0x7f2aa80c34a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000caa0 tx=0x7f2aa000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:44.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2a96ffd640 1 -- 172.21.15.7:0/2661876425 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa000ac00 con 0x7f2aa815ce10 2023-11-28T18:30:44.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2a96ffd640 1 -- 172.21.15.7:0/2661876425 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa000ad60 con 0x7f2aa815ce10 2023-11-28T18:30:44.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2a96ffd640 1 -- 172.21.15.7:0/2661876425 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0012620 con 0x7f2aa815ce10 2023-11-28T18:30:44.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa81f6220 con 0x7f2aa815ce10 2023-11-28T18:30:44.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa81f67e0 con 0x7f2aa815ce10 2023-11-28T18:30:44.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa804e890 con 0x7f2aa815ce10 2023-11-28T18:30:44.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.039+0000 7f2a96ffd640 1 -- 172.21.15.7:0/2661876425 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2aa0007070 con 0x7f2aa815ce10 2023-11-28T18:30:44.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.043+0000 7f2a96ffd640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a880668d0 0x7f2a88068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:44.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.043+0000 7f2a96ffd640 1 -- 172.21.15.7:0/2661876425 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f2aa0084500 con 0x7f2aa815ce10 2023-11-28T18:30:44.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.043+0000 7f2aa659c640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a880668d0 0x7f2a88068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:44.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.043+0000 7f2aa659c640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a880668d0 0x7f2a88068d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2a98000f70 tx=0x7f2a98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:44.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.047+0000 7f2a96ffd640 1 -- 172.21.15.7:0/2661876425 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2aa0051670 con 0x7f2aa815ce10 2023-11-28T18:30:44.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa80c2bb0 con 0x7f2aa815ce10 2023-11-28T18:30:44.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2a96ffd640 1 -- 172.21.15.7:0/2661876425 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f2aa0054c90 con 0x7f2aa815ce10 2023-11-28T18:30:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:44.295 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:44.295 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:44.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a880668d0 msgr2=0x7f2a88068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:44.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a880668d0 0x7f2a88068d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2a98000f70 tx=0x7f2a98009040 comp rx=0 tx=0).stop 2023-11-28T18:30:44.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2aa815ce10 msgr2=0x7f2aa80c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:44.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2aa815ce10 0x7f2aa80c34a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000caa0 tx=0x7f2aa000cf60 comp rx=0 tx=0).stop 2023-11-28T18:30:44.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 shutdown_connections 2023-11-28T18:30:44.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a880668d0 0x7f2a88068d90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:44.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2aa815ce10 0x7f2aa80c34a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:44.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2aa815ca30 0x7f2aa80c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:44.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 --2- 172.21.15.7:0/2661876425 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2aa80cafe0 0x7f2aa80c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:44.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 >> 172.21.15.7:0/2661876425 conn(0x7f2aa806c060 msgr2=0x7f2aa80cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:44.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 shutdown_connections 2023-11-28T18:30:44.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:44.291+0000 7f2aa759e640 1 -- 172.21.15.7:0/2661876425 wait complete. 2023-11-28T18:30:44.318 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:30:46.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1dec8dd640 1 Processor -- start 2023-11-28T18:30:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1dec8dd640 1 -- start start 2023-11-28T18:30:46.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1dec8dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de80cbb70 0x7f1de80cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:46.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1dec8dd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1de80cc810 0x7f1de8072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:46.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1dec8dd640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de80ca920 con 0x7f1de81542e0 2023-11-28T18:30:46.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1dec8dd640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de8071d50 con 0x7f1de80cc810 2023-11-28T18:30:46.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1dec8dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8072030 con 0x7f1de80cbb70 2023-11-28T18:30:46.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1de5d9b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de80cbb70 0x7f1de80cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:46.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1de6d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1de80cc810 0x7f1de8072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:46.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1de5d9b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de80cbb70 0x7f1de80cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:58900/0 (socket says 172.21.15.7:58900) 2023-11-28T18:30:46.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1de5d9b640 1 -- 172.21.15.7:0/834513094 learned_addr learned my addr 172.21.15.7:0/834513094 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:46.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1de5d9b640 1 -- 172.21.15.7:0/834513094 >> v2:172.21.15.193:3300/0 conn(0x7f1de80cc810 msgr2=0x7f1de8072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:46.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1de5d9b640 1 --2- 172.21.15.7:0/834513094 >> v2:172.21.15.193:3300/0 conn(0x7f1de80cc810 0x7f1de8072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1de5d9b640 1 -- 172.21.15.7:0/834513094 >> v1:172.21.15.193:6789/0 conn(0x7f1de81542e0 legacy=0x7f1de815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:46.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.467+0000 7f1de5d9b640 1 -- 172.21.15.7:0/834513094 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de80c4cd0 con 0x7f1de80cbb70 2023-11-28T18:30:46.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.471+0000 7f1de5d9b640 1 --2- 172.21.15.7:0/834513094 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de80cbb70 0x7f1de80cc2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009870 tx=0x7f1ddc01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcd2f2e347481343 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:46.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.471+0000 7f1de559a640 1 -- 172.21.15.7:0/834513094 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0ab650 con 0x7f1de80cbb70 2023-11-28T18:30:46.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.471+0000 7f1de559a640 1 -- 172.21.15.7:0/834513094 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc0ab7b0 con 0x7f1de80cbb70 2023-11-28T18:30:46.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.471+0000 7f1de559a640 1 -- 172.21.15.7:0/834513094 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0abb40 con 0x7f1de80cbb70 2023-11-28T18:30:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- 172.21.15.7:0/834513094 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de80cbb70 msgr2=0x7f1de80cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/834513094 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de80cbb70 0x7f1de80cc2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009870 tx=0x7f1ddc01c540 comp rx=0 tx=0).stop 2023-11-28T18:30:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- 172.21.15.7:0/834513094 shutdown_connections 2023-11-28T18:30:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/834513094 >> v2:172.21.15.193:3300/0 conn(0x7f1de80cc810 0x7f1de8072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/834513094 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de80cbb70 0x7f1de80cc2d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- 172.21.15.7:0/834513094 >> 172.21.15.7:0/834513094 conn(0x7f1de80717e0 msgr2=0x7f1de80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- 172.21.15.7:0/834513094 shutdown_connections 2023-11-28T18:30:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- 172.21.15.7:0/834513094 wait complete. 2023-11-28T18:30:46.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 Processor -- start 2023-11-28T18:30:46.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- start start 2023-11-28T18:30:46.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1de80cbb70 0x7f1de814b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:46.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de80cc810 0x7f1de80cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:46.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de81542e0 0x7f1de8141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:46.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de81480d0 con 0x7f1de80cc810 2023-11-28T18:30:46.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de814c160 con 0x7f1de80cbb70 2023-11-28T18:30:46.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1dec8dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80c97d0 con 0x7f1de81542e0 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1de5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de80cc810 0x7f1de80cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1de659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1de80cbb70 0x7f1de814b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1de5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de80cc810 0x7f1de80cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54548/0 (socket says 172.21.15.7:54548) 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.475+0000 7f1de5d9b640 1 -- 172.21.15.7:0/1889263255 learned_addr learned my addr 172.21.15.7:0/1889263255 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1de6d9d640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de81542e0 0x7f1de8141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1de5d9b640 1 -- 172.21.15.7:0/1889263255 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de81542e0 msgr2=0x7f1de8141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1de5d9b640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de81542e0 0x7f1de8141c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1de5d9b640 1 -- 172.21.15.7:0/1889263255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1de80cbb70 msgr2=0x7f1de814b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1de5d9b640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1de80cbb70 0x7f1de814b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1de5d9b640 1 -- 172.21.15.7:0/1889263255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc0094d0 con 0x7f1de80cc810 2023-11-28T18:30:46.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1de5d9b640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de80cc810 0x7f1de80cfb10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc01abf0 tx=0x7f1ddc01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:46.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1dd6ffd640 1 -- 172.21.15.7:0/1889263255 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0b4800 con 0x7f1de80cc810 2023-11-28T18:30:46.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1dd6ffd640 1 -- 172.21.15.7:0/1889263255 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc0b4960 con 0x7f1de80cc810 2023-11-28T18:30:46.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1dd6ffd640 1 -- 172.21.15.7:0/1889263255 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0b4d10 con 0x7f1de80cc810 2023-11-28T18:30:46.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de80c7f90 con 0x7f1de80cc810 2023-11-28T18:30:46.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de814be80 con 0x7f1de80cc810 2023-11-28T18:30:46.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de806c670 con 0x7f1de80cc810 2023-11-28T18:30:46.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.479+0000 7f1dd6ffd640 1 -- 172.21.15.7:0/1889263255 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1ddc0b3380 con 0x7f1de80cc810 2023-11-28T18:30:46.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.483+0000 7f1dd6ffd640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1dc80668d0 0x7f1dc8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:46.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.483+0000 7f1dd6ffd640 1 -- 172.21.15.7:0/1889263255 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f1ddc11ec00 con 0x7f1de80cc810 2023-11-28T18:30:46.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.483+0000 7f1de659c640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1dc80668d0 0x7f1dc8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:46.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.483+0000 7f1de659c640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1dc80668d0 0x7f1dc8068d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007fd0 tx=0x7f1dd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:46.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.487+0000 7f1dd6ffd640 1 -- 172.21.15.7:0/1889263255 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1ddc125080 con 0x7f1de80cc810 2023-11-28T18:30:46.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.715+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1de814bce0 con 0x7f1de80cc810 2023-11-28T18:30:46.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.715+0000 7f1dd6ffd640 1 -- 172.21.15.7:0/1889263255 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6550 (secure 0 0 0) 0x7f1ddc0f0e60 con 0x7f1de80cc810 2023-11-28T18:30:46.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:46.720 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:23.561602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4285,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6837/3911527535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3911527535},{"type":"v1","addr":"172.21.15.193:6837","nonce":3911527535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:46.720 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 41 2023-11-28T18:30:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1dc80668d0 msgr2=0x7f1dc8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1dc80668d0 0x7f1dc8068d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007fd0 tx=0x7f1dd8009040 comp rx=0 tx=0).stop 2023-11-28T18:30:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de80cc810 msgr2=0x7f1de80cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de80cc810 0x7f1de80cfb10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc01abf0 tx=0x7f1ddc01ad80 comp rx=0 tx=0).stop 2023-11-28T18:30:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 shutdown_connections 2023-11-28T18:30:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1dc80668d0 0x7f1dc8068d90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1de81542e0 0x7f1de8141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de80cc810 0x7f1de80cfb10 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 --2- 172.21.15.7:0/1889263255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1de80cbb70 0x7f1de814b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 >> 172.21.15.7:0/1889263255 conn(0x7f1de80717e0 msgr2=0x7f1de80b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 shutdown_connections 2023-11-28T18:30:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:46.719+0000 7f1dec8dd640 1 -- 172.21.15.7:0/1889263255 wait complete. 2023-11-28T18:30:46.746 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:30:48.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea3548640 1 Processor -- start 2023-11-28T18:30:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea3548640 1 -- start start 2023-11-28T18:30:48.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea3548640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ce10 0x7f6e9c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:48.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea3548640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e9c0cafe0 0x7f6e9c0ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:48.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea3548640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c15a3a0 con 0x7f6e9c15ca30 2023-11-28T18:30:48.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea3548640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c15a620 con 0x7f6e9c0cafe0 2023-11-28T18:30:48.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea3548640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c15a900 con 0x7f6e9c15ce10 2023-11-28T18:30:48.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea2d47640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e9c0cafe0 0x7f6e9c0ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:48.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea2d47640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e9c0cafe0 0x7f6e9c0ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54560/0 (socket says 172.21.15.7:54560) 2023-11-28T18:30:48.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea2d47640 1 -- 172.21.15.7:0/2309770022 learned_addr learned my addr 172.21.15.7:0/2309770022 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:48.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea1d45640 1 --2- 172.21.15.7:0/2309770022 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ce10 0x7f6e9c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:48.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea2d47640 1 -- 172.21.15.7:0/2309770022 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ce10 msgr2=0x7f6e9c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:48.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea2d47640 1 --2- 172.21.15.7:0/2309770022 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ce10 0x7f6e9c0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:48.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea2d47640 1 -- 172.21.15.7:0/2309770022 >> v1:172.21.15.193:6789/0 conn(0x7f6e9c15ca30 legacy=0x7f6e9c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:48.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea2d47640 1 -- 172.21.15.7:0/2309770022 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c15abe0 con 0x7f6e9c0cafe0 2023-11-28T18:30:48.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.899+0000 7f6ea1d45640 1 --2- 172.21.15.7:0/2309770022 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ce10 0x7f6e9c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:30:48.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.903+0000 7f6ea2d47640 1 --2- 172.21.15.7:0/2309770022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c0cafe0 0x7f6e9c0ca710 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009e50 tx=0x7f6e9401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455b70f4573cd045 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:48.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.903+0000 7f6ea1544640 1 -- 172.21.15.7:0/2309770022 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e940af400 con 0x7f6e9c0cafe0 2023-11-28T18:30:48.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.903+0000 7f6ea1544640 1 -- 172.21.15.7:0/2309770022 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e940ab050 con 0x7f6e9c0cafe0 2023-11-28T18:30:48.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.903+0000 7f6ea1544640 1 -- 172.21.15.7:0/2309770022 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e940af780 con 0x7f6e9c0cafe0 2023-11-28T18:30:48.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.903+0000 7f6ea3548640 1 -- 172.21.15.7:0/2309770022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c0cafe0 msgr2=0x7f6e9c0ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:48.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.903+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2309770022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c0cafe0 0x7f6e9c0ca710 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009e50 tx=0x7f6e9401f1f0 comp rx=0 tx=0).stop 2023-11-28T18:30:48.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 -- 172.21.15.7:0/2309770022 shutdown_connections 2023-11-28T18:30:48.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2309770022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c0cafe0 0x7f6e9c0ca710 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:48.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2309770022 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ce10 0x7f6e9c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:48.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 -- 172.21.15.7:0/2309770022 >> 172.21.15.7:0/2309770022 conn(0x7f6e9c06c060 msgr2=0x7f6e9c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:48.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 -- 172.21.15.7:0/2309770022 shutdown_connections 2023-11-28T18:30:48.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 -- 172.21.15.7:0/2309770022 wait complete. 2023-11-28T18:30:48.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 Processor -- start 2023-11-28T18:30:48.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 -- start start 2023-11-28T18:30:48.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e9c0cafe0 0x7f6e9c157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:48.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ca30 0x7f6e9c0c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:48.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c15ce10 0x7f6e9c0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:48.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c155f20 con 0x7f6e9c15ce10 2023-11-28T18:30:48.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c156d50 con 0x7f6e9c0cafe0 2023-11-28T18:30:48.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea3548640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c155190 con 0x7f6e9c15ca30 2023-11-28T18:30:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2d47640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c15ce10 0x7f6e9c0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2d47640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c15ce10 0x7f6e9c0c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54574/0 (socket says 172.21.15.7:54574) 2023-11-28T18:30:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2d47640 1 -- 172.21.15.7:0/2916455542 learned_addr learned my addr 172.21.15.7:0/2916455542 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2546640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e9c0cafe0 0x7f6e9c157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea1d45640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ca30 0x7f6e9c0c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:48.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2d47640 1 -- 172.21.15.7:0/2916455542 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ca30 msgr2=0x7f6e9c0c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:48.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2d47640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ca30 0x7f6e9c0c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:48.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2d47640 1 -- 172.21.15.7:0/2916455542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e9c0cafe0 msgr2=0x7f6e9c157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:48.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2d47640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e9c0cafe0 0x7f6e9c157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:48.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.907+0000 7f6ea2d47640 1 -- 172.21.15.7:0/2916455542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e94009b30 con 0x7f6e9c15ce10 2023-11-28T18:30:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6ea2d47640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c15ce10 0x7f6e9c0c7cf0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6e9401ce20 tx=0x7f6e9401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6e92ffd640 1 -- 172.21.15.7:0/2916455542 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e940c0070 con 0x7f6e9c15ce10 2023-11-28T18:30:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6e92ffd640 1 -- 172.21.15.7:0/2916455542 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e94009510 con 0x7f6e9c15ce10 2023-11-28T18:30:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c155470 con 0x7f6e9c15ce10 2023-11-28T18:30:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6e92ffd640 1 -- 172.21.15.7:0/2916455542 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e940afe20 con 0x7f6e9c15ce10 2023-11-28T18:30:48.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c0c74f0 con 0x7f6e9c15ce10 2023-11-28T18:30:48.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c04e890 con 0x7f6e9c15ce10 2023-11-28T18:30:48.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6e92ffd640 1 -- 172.21.15.7:0/2916455542 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6e940b6040 con 0x7f6e9c15ce10 2023-11-28T18:30:48.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6e92ffd640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e80066900 0x7f6e80068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:48.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6e92ffd640 1 -- 172.21.15.7:0/2916455542 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f6e94121d70 con 0x7f6e9c15ce10 2023-11-28T18:30:48.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.911+0000 7f6ea2546640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e80066900 0x7f6e80068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.915+0000 7f6ea2546640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e80066900 0x7f6e80068dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e98007fd0 tx=0x7f6e98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:48.919+0000 7f6e92ffd640 1 -- 172.21.15.7:0/2916455542 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6e940ef070 con 0x7f6e9c15ce10 2023-11-28T18:30:49.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.175+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e9c0cad10 con 0x7f6e9c15ce10 2023-11-28T18:30:49.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.175+0000 7f6e92ffd640 1 -- 172.21.15.7:0/2916455542 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f6e940f2690 con 0x7f6e9c15ce10 2023-11-28T18:30:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:49.177 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:47.207708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:49.177 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 42 2023-11-28T18:30:49.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e80066900 msgr2=0x7f6e80068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:49.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e80066900 0x7f6e80068dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e98007fd0 tx=0x7f6e98009040 comp rx=0 tx=0).stop 2023-11-28T18:30:49.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c15ce10 msgr2=0x7f6e9c0c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:49.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c15ce10 0x7f6e9c0c7cf0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6e9401ce20 tx=0x7f6e9401bf90 comp rx=0 tx=0).stop 2023-11-28T18:30:49.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 shutdown_connections 2023-11-28T18:30:49.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6e80066900 0x7f6e80068dc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e9c15ce10 0x7f6e9c0c7cf0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6e9c15ca30 0x7f6e9c0c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 --2- 172.21.15.7:0/2916455542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e9c0cafe0 0x7f6e9c157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 >> 172.21.15.7:0/2916455542 conn(0x7f6e9c06c060 msgr2=0x7f6e9c0cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 shutdown_connections 2023-11-28T18:30:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.179+0000 7f6ea3548640 1 -- 172.21.15.7:0/2916455542 wait complete. 2023-11-28T18:30:49.202 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-11-28T18:30:49.202 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:30:49.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:49.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31c15df640 1 Processor -- start 2023-11-28T18:30:49.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31c15df640 1 -- start start 2023-11-28T18:30:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31c15df640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f31bc153f00 0x7f31bc0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31c15df640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc0cb020 0x7f31bc0b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31c15df640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31bc0cb5a0 0x7f31bc0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31c15df640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31bc0ca900 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31c15df640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31bc15d420 con 0x7f31bc153f00 2023-11-28T18:30:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31c15df640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc15d700 con 0x7f31bc0cb020 2023-11-28T18:30:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31bb7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31bc0cb5a0 0x7f31bc0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31bb7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31bc0cb5a0 0x7f31bc0b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54578/0 (socket says 172.21.15.7:54578) 2023-11-28T18:30:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31bb7fe640 1 -- 172.21.15.7:0/1053960496 learned_addr learned my addr 172.21.15.7:0/1053960496 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31baffd640 1 --2- 172.21.15.7:0/1053960496 >> v2:172.21.15.193:3300/0 conn(0x7f31bc153f00 0x7f31bc0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31ba7fc640 1 --2- 172.21.15.7:0/1053960496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc0cb020 0x7f31bc0b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31bb7fe640 1 -- 172.21.15.7:0/1053960496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc0cb020 msgr2=0x7f31bc0b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31bb7fe640 1 --2- 172.21.15.7:0/1053960496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc0cb020 0x7f31bc0b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31bb7fe640 1 -- 172.21.15.7:0/1053960496 >> v2:172.21.15.193:3300/0 conn(0x7f31bc153f00 msgr2=0x7f31bc0ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31bb7fe640 1 --2- 172.21.15.7:0/1053960496 >> v2:172.21.15.193:3300/0 conn(0x7f31bc153f00 0x7f31bc0ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.359+0000 7f31bb7fe640 1 -- 172.21.15.7:0/1053960496 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc15d9e0 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31bb7fe640 1 --2- 172.21.15.7:0/1053960496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc0b2500 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f31bc066f20 tx=0x7f31b401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f051d32956f40b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31b9ffb640 1 -- 172.21.15.7:0/1053960496 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b40af400 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31b9ffb640 1 -- 172.21.15.7:0/1053960496 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b40ab050 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31b9ffb640 1 -- 172.21.15.7:0/1053960496 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b40af780 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 -- 172.21.15.7:0/1053960496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 msgr2=0x7f31bc0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 --2- 172.21.15.7:0/1053960496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc0b2500 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f31bc066f20 tx=0x7f31b401f1c0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 -- 172.21.15.7:0/1053960496 shutdown_connections 2023-11-28T18:30:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 --2- 172.21.15.7:0/1053960496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc0b2500 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 --2- 172.21.15.7:0/1053960496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc0cb020 0x7f31bc0b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 --2- 172.21.15.7:0/1053960496 >> v2:172.21.15.193:3300/0 conn(0x7f31bc153f00 0x7f31bc0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 -- 172.21.15.7:0/1053960496 >> 172.21.15.7:0/1053960496 conn(0x7f31bc06c060 msgr2=0x7f31bc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 -- 172.21.15.7:0/1053960496 shutdown_connections 2023-11-28T18:30:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.363+0000 7f31c15df640 1 -- 172.21.15.7:0/1053960496 wait complete. 2023-11-28T18:30:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 Processor -- start 2023-11-28T18:30:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 -- start start 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31bc0cb020 0x7f31bc0c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc153f00 0x7f31bc1efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc1f3d90 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc155cc0 con 0x7f31bc0cb020 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc1f31a0 con 0x7f31bc153f00 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31bb7fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc153f00 0x7f31bc1efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc1582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54594/0 (socket says 172.21.15.7:54594) 2023-11-28T18:30:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 -- 172.21.15.7:0/723140011 learned_addr learned my addr 172.21.15.7:0/723140011 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31baffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31bc0cb020 0x7f31bc0c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 -- 172.21.15.7:0/723140011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc153f00 msgr2=0x7f31bc1efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc153f00 0x7f31bc1efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 -- 172.21.15.7:0/723140011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31bc0cb020 msgr2=0x7f31bc0c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31bc0cb020 0x7f31bc0c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 -- 172.21.15.7:0/723140011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b4009b30 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31baffd640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31bc0cb020 0x7f31bc0c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31ba7fc640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc1582d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f31b000d800 tx=0x7f31b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31a37fe640 1 -- 172.21.15.7:0/723140011 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b00096a0 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31a37fe640 1 -- 172.21.15.7:0/723140011 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b0009800 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31a37fe640 1 -- 172.21.15.7:0/723140011 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0009b70 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.367+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc158c80 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.371+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc1f36d0 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.371+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31bc04e890 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.371+0000 7f31a37fe640 1 -- 172.21.15.7:0/723140011 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f31b0009cd0 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.371+0000 7f31a37fe640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31a8066920 0x7f31a8068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.371+0000 7f31a37fe640 1 -- 172.21.15.7:0/723140011 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f31b0084d40 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.371+0000 7f31baffd640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31a8066920 0x7f31a8068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.375+0000 7f31baffd640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31a8066920 0x7f31a8068de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f31ac007db0 tx=0x7f31ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:49.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.379+0000 7f31a37fe640 1 -- 172.21.15.7:0/723140011 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f31b0051db0 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.595+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31bc0cae00 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.595+0000 7f31a37fe640 1 -- 172.21.15.7:0/723140011 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f31b00553d0 con 0x7f31bc0cb5a0 2023-11-28T18:30:49.597 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:49.597 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:47.207708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 42 2023-11-28T18:30:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31a8066920 msgr2=0x7f31a8068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31a8066920 0x7f31a8068de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f31ac007db0 tx=0x7f31ac008040 comp rx=0 tx=0).stop 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 msgr2=0x7f31bc1582d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc1582d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f31b000d800 tx=0x7f31b000dcc0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 shutdown_connections 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31a8066920 0x7f31a8068de0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31bc153f00 0x7f31bc1efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31bc0cb5a0 0x7f31bc1582d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 --2- 172.21.15.7:0/723140011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31bc0cb020 0x7f31bc0c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 >> 172.21.15.7:0/723140011 conn(0x7f31bc06c060 msgr2=0x7f31bc0ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 shutdown_connections 2023-11-28T18:30:49.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:49.599+0000 7f31c15df640 1 -- 172.21.15.7:0/723140011 wait complete. 2023-11-28T18:30:49.624 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4288": { "addr": "172.21.15.7:6838/2690556675", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 2690556675, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 2690556675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:30:47.207708+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 4288, "mds_2": 5307, "mds_3": 5086, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:30:51.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:51.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.771+0000 7fd1b8780640 1 Processor -- start 2023-11-28T18:30:51.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.771+0000 7fd1b8780640 1 -- start start 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b8780640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 0x7fd1b0074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b8780640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd1b006c920 0x7fd1b006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b8780640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b00c8490 con 0x7fd1b0072950 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b8780640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b006d2e0 con 0x7fd1b006c920 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b8780640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00c81f0 con 0x7fd1b0073070 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b6d1d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd1b006c920 0x7fd1b006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b6d1d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd1b006c920 0x7fd1b006cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54596/0 (socket says 172.21.15.7:54596) 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b6d1d640 1 -- 172.21.15.7:0/1055432793 learned_addr learned my addr 172.21.15.7:0/1055432793 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b5d1b640 1 --2- 172.21.15.7:0/1055432793 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 0x7fd1b0074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:51.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b6d1d640 1 -- 172.21.15.7:0/1055432793 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 msgr2=0x7fd1b0074030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:51.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b6d1d640 1 --2- 172.21.15.7:0/1055432793 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 0x7fd1b0074030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:51.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b6d1d640 1 -- 172.21.15.7:0/1055432793 >> v1:172.21.15.193:6789/0 conn(0x7fd1b0072950 legacy=0x7fd1b00c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:51.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.775+0000 7fd1b6d1d640 1 -- 172.21.15.7:0/1055432793 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b015ab00 con 0x7fd1b006c920 2023-11-28T18:30:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b6d1d640 1 --2- 172.21.15.7:0/1055432793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b006c920 0x7fd1b006cda0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009e50 tx=0x7fd1a801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa64ceaaf6e91168 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b551a640 1 -- 172.21.15.7:0/1055432793 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80a5590 con 0x7fd1b006c920 2023-11-28T18:30:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b551a640 1 -- 172.21.15.7:0/1055432793 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a80a56f0 con 0x7fd1b006c920 2023-11-28T18:30:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b551a640 1 -- 172.21.15.7:0/1055432793 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80a5a80 con 0x7fd1b006c920 2023-11-28T18:30:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b8780640 1 -- 172.21.15.7:0/1055432793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b006c920 msgr2=0x7fd1b006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b8780640 1 --2- 172.21.15.7:0/1055432793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b006c920 0x7fd1b006cda0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009e50 tx=0x7fd1a801c8f0 comp rx=0 tx=0).stop 2023-11-28T18:30:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b8780640 1 -- 172.21.15.7:0/1055432793 shutdown_connections 2023-11-28T18:30:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b8780640 1 --2- 172.21.15.7:0/1055432793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b006c920 0x7fd1b006cda0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b8780640 1 --2- 172.21.15.7:0/1055432793 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 0x7fd1b0074030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b8780640 1 -- 172.21.15.7:0/1055432793 >> 172.21.15.7:0/1055432793 conn(0x7fd1b00cbf80 msgr2=0x7fd1b00cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b8780640 1 -- 172.21.15.7:0/1055432793 shutdown_connections 2023-11-28T18:30:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.779+0000 7fd1b8780640 1 -- 172.21.15.7:0/1055432793 wait complete. 2023-11-28T18:30:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 Processor -- start 2023-11-28T18:30:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 -- start start 2023-11-28T18:30:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1b006c920 0x7fd1b0152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b0072950 0x7fd1b0151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 0x7fd1b01522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00c64a0 con 0x7fd1b0072950 2023-11-28T18:30:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b01f3410 con 0x7fd1b006c920 2023-11-28T18:30:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b01f2eb0 con 0x7fd1b0073070 2023-11-28T18:30:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b0072950 0x7fd1b0151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b0072950 0x7fd1b0151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54608/0 (socket says 172.21.15.7:54608) 2023-11-28T18:30:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 -- 172.21.15.7:0/2304566668 learned_addr learned my addr 172.21.15.7:0/2304566668 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b651c640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1b006c920 0x7fd1b0152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b6d1d640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 0x7fd1b01522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 -- 172.21.15.7:0/2304566668 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 msgr2=0x7fd1b01522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 0x7fd1b01522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 -- 172.21.15.7:0/2304566668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1b006c920 msgr2=0x7fd1b0152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1b006c920 0x7fd1b0152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 -- 172.21.15.7:0/2304566668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8009b30 con 0x7fd1b0072950 2023-11-28T18:30:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b5d1b640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b0072950 0x7fd1b0151d90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e930 tx=0x7fd1a000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1a6ffd640 1 -- 172.21.15.7:0/2304566668 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0009690 con 0x7fd1b0072950 2023-11-28T18:30:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1a6ffd640 1 -- 172.21.15.7:0/2304566668 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a000ca00 con 0x7fd1b0072950 2023-11-28T18:30:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b01517d0 con 0x7fd1b0072950 2023-11-28T18:30:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1a6ffd640 1 -- 172.21.15.7:0/2304566668 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a000cd90 con 0x7fd1b0072950 2023-11-28T18:30:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.783+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b01f3130 con 0x7fd1b0072950 2023-11-28T18:30:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.787+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b004e890 con 0x7fd1b0072950 2023-11-28T18:30:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.787+0000 7fd1a6ffd640 1 -- 172.21.15.7:0/2304566668 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd1a0013470 con 0x7fd1b0072950 2023-11-28T18:30:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.787+0000 7fd1a6ffd640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd194066900 0x7fd194068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.787+0000 7fd1a6ffd640 1 -- 172.21.15.7:0/2304566668 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fd1a0084b30 con 0x7fd1b0072950 2023-11-28T18:30:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.787+0000 7fd1b651c640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd194066900 0x7fd194068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.787+0000 7fd1b651c640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd194066900 0x7fd194068dc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac007fd0 tx=0x7fd1ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:51.791+0000 7fd1a6ffd640 1 -- 172.21.15.7:0/2304566668 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd1a0051d00 con 0x7fd1b0072950 2023-11-28T18:30:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.043+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1b00c7e50 con 0x7fd1b0072950 2023-11-28T18:30:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.043+0000 7fd1a6ffd640 1 -- 172.21.15.7:0/2304566668 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5757 (secure 0 0 0) 0x7fd1a0017090 con 0x7fd1b0072950 2023-11-28T18:30:52.045 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:52.045 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:47.207708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 42 2023-11-28T18:30:52.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd194066900 msgr2=0x7fd194068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:52.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd194066900 0x7fd194068dc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac007fd0 tx=0x7fd1ac009040 comp rx=0 tx=0).stop 2023-11-28T18:30:52.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b0072950 msgr2=0x7fd1b0151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:52.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b0072950 0x7fd1b0151d90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e930 tx=0x7fd1a000edf0 comp rx=0 tx=0).stop 2023-11-28T18:30:52.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 shutdown_connections 2023-11-28T18:30:52.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd194066900 0x7fd194068dc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:52.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1b0073070 0x7fd1b01522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:52.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1b0072950 0x7fd1b0151d90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:52.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 --2- 172.21.15.7:0/2304566668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1b006c920 0x7fd1b0152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:52.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 >> 172.21.15.7:0/2304566668 conn(0x7fd1b00cbf80 msgr2=0x7fd1b014d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:52.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 shutdown_connections 2023-11-28T18:30:52.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:52.047+0000 7fd1b8780640 1 -- 172.21.15.7:0/2304566668 wait complete. 2023-11-28T18:30:52.068 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:54.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:54.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8b61c4640 1 Processor -- start 2023-11-28T18:30:54.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8b61c4640 1 -- start start 2023-11-28T18:30:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8b61c4640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe8b01542e0 0x7fe8b015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8b61c4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe8b0072450 0x7fe8b0072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8b61c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00ca920 con 0x7fe8b00cbb70 2023-11-28T18:30:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8b61c4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b0071d50 con 0x7fe8b0072450 2023-11-28T18:30:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8b61c4640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b0072030 con 0x7fe8b01542e0 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8affff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe8b0072450 0x7fe8b0072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8aeffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe8b00cbb70 0x7fe8b00cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37094/0 (socket says 172.21.15.7:37094) 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8aeffd640 1 -- 172.21.15.7:0/1241446317 learned_addr learned my addr 172.21.15.7:0/1241446317 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8af7fe640 1 --2- 172.21.15.7:0/1241446317 >> v2:172.21.15.193:3300/0 conn(0x7fe8b01542e0 0x7fe8b015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8ae7fc640 1 -- 172.21.15.7:0/1241446317 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287621059 0 0) 0x7fe8b00ca920 con 0x7fe8b00cbb70 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8ae7fc640 1 -- 172.21.15.7:0/1241446317 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe898003530 con 0x7fe8b00cbb70 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8affff640 1 -- 172.21.15.7:0/1241446317 >> v2:172.21.15.193:3300/0 conn(0x7fe8b01542e0 msgr2=0x7fe8b015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8affff640 1 --2- 172.21.15.7:0/1241446317 >> v2:172.21.15.193:3300/0 conn(0x7fe8b01542e0 0x7fe8b015e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8affff640 1 -- 172.21.15.7:0/1241446317 >> v1:172.21.15.7:6789/0 conn(0x7fe8b00cbb70 legacy=0x7fe8b00cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.231+0000 7fe8affff640 1 -- 172.21.15.7:0/1241446317 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b015e550 con 0x7fe8b0072450 2023-11-28T18:30:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8affff640 1 --2- 172.21.15.7:0/1241446317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b0072450 0x7fe8b0072870 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0071d20 tx=0x7fe8a801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60327db024eb0fcb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8ae7fc640 1 -- 172.21.15.7:0/1241446317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a80a58e0 con 0x7fe8b0072450 2023-11-28T18:30:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8ae7fc640 1 -- 172.21.15.7:0/1241446317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a80a5a40 con 0x7fe8b0072450 2023-11-28T18:30:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8ae7fc640 1 -- 172.21.15.7:0/1241446317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a80a5dd0 con 0x7fe8b0072450 2023-11-28T18:30:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8b61c4640 1 -- 172.21.15.7:0/1241446317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b0072450 msgr2=0x7fe8b0072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/1241446317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b0072450 0x7fe8b0072870 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0071d20 tx=0x7fe8a801c910 comp rx=0 tx=0).stop 2023-11-28T18:30:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8b61c4640 1 -- 172.21.15.7:0/1241446317 shutdown_connections 2023-11-28T18:30:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/1241446317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b0072450 0x7fe8b0072870 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/1241446317 >> v2:172.21.15.193:3300/0 conn(0x7fe8b01542e0 0x7fe8b015e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.235+0000 7fe8b61c4640 1 -- 172.21.15.7:0/1241446317 >> 172.21.15.7:0/1241446317 conn(0x7fe8b00717e0 msgr2=0x7fe8b00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 -- 172.21.15.7:0/1241446317 shutdown_connections 2023-11-28T18:30:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 -- 172.21.15.7:0/1241446317 wait complete. 2023-11-28T18:30:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 Processor -- start 2023-11-28T18:30:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 -- start start 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8b0072450 0x7fe8b0159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b00cbb70 0x7fe8b015a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b01542e0 0x7fe8b00c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00c8110 con 0x7fe8b00cbb70 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00c9b30 con 0x7fe8b01542e0 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8b61c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00c7dc0 con 0x7fe8b0072450 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8aeffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b00cbb70 0x7fe8b015a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8aeffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b00cbb70 0x7fe8b015a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51548/0 (socket says 172.21.15.7:51548) 2023-11-28T18:30:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8aeffd640 1 -- 172.21.15.7:0/807302919 learned_addr learned my addr 172.21.15.7:0/807302919 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8af7fe640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8b0072450 0x7fe8b0159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8affff640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b01542e0 0x7fe8b00c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8aeffd640 1 -- 172.21.15.7:0/807302919 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8b0072450 msgr2=0x7fe8b0159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8aeffd640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8b0072450 0x7fe8b0159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8aeffd640 1 -- 172.21.15.7:0/807302919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b01542e0 msgr2=0x7fe8b00c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8aeffd640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b01542e0 0x7fe8b00c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.239+0000 7fe8aeffd640 1 -- 172.21.15.7:0/807302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a8009a70 con 0x7fe8b00cbb70 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe8aeffd640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b00cbb70 0x7fe8b015a800 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe8a400c480 tx=0x7fe8a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe897fff640 1 -- 172.21.15.7:0/807302919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4005cf0 con 0x7fe8b00cbb70 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe897fff640 1 -- 172.21.15.7:0/807302919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a4005e50 con 0x7fe8b00cbb70 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe897fff640 1 -- 172.21.15.7:0/807302919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4007290 con 0x7fe8b00cbb70 2023-11-28T18:30:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b0158860 con 0x7fe8b00cbb70 2023-11-28T18:30:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b0158e20 con 0x7fe8b00cbb70 2023-11-28T18:30:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b006c670 con 0x7fe8b00cbb70 2023-11-28T18:30:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe897fff640 1 -- 172.21.15.7:0/807302919 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe8a40073f0 con 0x7fe8b00cbb70 2023-11-28T18:30:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe897fff640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe89c0668d0 0x7fe89c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe897fff640 1 -- 172.21.15.7:0/807302919 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe8a4083ac0 con 0x7fe8b00cbb70 2023-11-28T18:30:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.243+0000 7fe8af7fe640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe89c0668d0 0x7fe89c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.247+0000 7fe8af7fe640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe89c0668d0 0x7fe89c068d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009ac0 tx=0x7fe8a0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:54.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.251+0000 7fe897fff640 1 -- 172.21.15.7:0/807302919 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe8a4050c70 con 0x7fe8b00cbb70 2023-11-28T18:30:54.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.479+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8b0159040 con 0x7fe8b00cbb70 2023-11-28T18:30:54.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.479+0000 7fe897fff640 1 -- 172.21.15.7:0/807302919 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5758 (secure 0 0 0) 0x7fe8a4083e60 con 0x7fe8b00cbb70 2023-11-28T18:30:54.479 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:54.479 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:52.234279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:resolve","state_seq":8,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:54.480 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 43 2023-11-28T18:30:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe89c0668d0 msgr2=0x7fe89c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe89c0668d0 0x7fe89c068d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009ac0 tx=0x7fe8a0009190 comp rx=0 tx=0).stop 2023-11-28T18:30:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b00cbb70 msgr2=0x7fe8b015a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b00cbb70 0x7fe8b015a800 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe8a400c480 tx=0x7fe8a400c940 comp rx=0 tx=0).stop 2023-11-28T18:30:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 shutdown_connections 2023-11-28T18:30:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe89c0668d0 0x7fe89c068d90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b01542e0 0x7fe8b00c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b00cbb70 0x7fe8b015a800 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 --2- 172.21.15.7:0/807302919 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8b0072450 0x7fe8b0159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 >> 172.21.15.7:0/807302919 conn(0x7fe8b00717e0 msgr2=0x7fe8b015b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 shutdown_connections 2023-11-28T18:30:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:54.483+0000 7fe8b61c4640 1 -- 172.21.15.7:0/807302919 wait complete. 2023-11-28T18:30:54.506 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:56.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:56.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.687+0000 7fde01e9a640 1 Processor -- start 2023-11-28T18:30:56.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.687+0000 7fde01e9a640 1 -- start start 2023-11-28T18:30:56.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fde01e9a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddfc151cd0 0x7fddfc15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:56.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fde01e9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc157b90 0x7fddfc0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:56.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fde01e9a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddfc157fe0 con 0x7fddfc06e560 2023-11-28T18:30:56.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fde01e9a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddfc070730 con 0x7fddfc151cd0 2023-11-28T18:30:56.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fde01e9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc071e60 con 0x7fddfc157b90 2023-11-28T18:30:56.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfb7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddfc151cd0 0x7fddfc15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:56.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfb7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddfc151cd0 0x7fddfc15bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51560/0 (socket says 172.21.15.7:51560) 2023-11-28T18:30:56.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfb7fe640 1 -- 172.21.15.7:0/4218418787 learned_addr learned my addr 172.21.15.7:0/4218418787 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:56.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfaffd640 1 --1- 172.21.15.7:0/4218418787 >> v1:172.21.15.7:6789/0 conn(0x7fddfc06e560 0x7fddfc0c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37098/0 (socket says 172.21.15.7:37098) 2023-11-28T18:30:56.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfbfff640 1 --2- 172.21.15.7:0/4218418787 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc157b90 0x7fddfc0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:56.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfb7fe640 1 -- 172.21.15.7:0/4218418787 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc157b90 msgr2=0x7fddfc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:56.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfb7fe640 1 --2- 172.21.15.7:0/4218418787 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc157b90 0x7fddfc0701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfb7fe640 1 -- 172.21.15.7:0/4218418787 >> v1:172.21.15.7:6789/0 conn(0x7fddfc06e560 legacy=0x7fddfc0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:56.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfb7fe640 1 -- 172.21.15.7:0/4218418787 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc070930 con 0x7fddfc151cd0 2023-11-28T18:30:56.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.691+0000 7fddfa7fc640 1 -- 172.21.15.7:0/4218418787 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433347056 0 0) 0x7fddfc157fe0 con 0x7fddfc06e560 2023-11-28T18:30:56.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fddfb7fe640 1 --2- 172.21.15.7:0/4218418787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc151cd0 0x7fddfc15bd00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fddec009870 tx=0x7fddec01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5052d8499ca17abc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:56.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fddfa7fc640 1 -- 172.21.15.7:0/4218418787 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec0ac600 con 0x7fddfc151cd0 2023-11-28T18:30:56.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fddfa7fc640 1 -- 172.21.15.7:0/4218418787 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddec0a8050 con 0x7fddfc151cd0 2023-11-28T18:30:56.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fddfa7fc640 1 -- 172.21.15.7:0/4218418787 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec0ac980 con 0x7fddfc151cd0 2023-11-28T18:30:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fde01e9a640 1 -- 172.21.15.7:0/4218418787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc151cd0 msgr2=0x7fddfc15bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fde01e9a640 1 --2- 172.21.15.7:0/4218418787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc151cd0 0x7fddfc15bd00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fddec009870 tx=0x7fddec01c4b0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fde01e9a640 1 -- 172.21.15.7:0/4218418787 shutdown_connections 2023-11-28T18:30:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fde01e9a640 1 --2- 172.21.15.7:0/4218418787 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc157b90 0x7fddfc0701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fde01e9a640 1 --2- 172.21.15.7:0/4218418787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc151cd0 0x7fddfc15bd00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fde01e9a640 1 -- 172.21.15.7:0/4218418787 >> 172.21.15.7:0/4218418787 conn(0x7fddfc0662e0 msgr2=0x7fddfc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fde01e9a640 1 -- 172.21.15.7:0/4218418787 shutdown_connections 2023-11-28T18:30:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.695+0000 7fde01e9a640 1 -- 172.21.15.7:0/4218418787 wait complete. 2023-11-28T18:30:56.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 Processor -- start 2023-11-28T18:30:56.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 -- start start 2023-11-28T18:30:56.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc06e560 0x7fddfc15a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:56.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddfc151cd0 0x7fddfc1505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:56.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc157b90 0x7fddfc14c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:56.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc0c72d0 con 0x7fddfc157b90 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc150380 con 0x7fddfc151cd0 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc0c7010 con 0x7fddfc06e560 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc157b90 0x7fddfc14c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc157b90 0x7fddfc14c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51574/0 (socket says 172.21.15.7:51574) 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 -- 172.21.15.7:0/2991568645 learned_addr learned my addr 172.21.15.7:0/2991568645 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfb7fe640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc06e560 0x7fddfc15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfaffd640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddfc151cd0 0x7fddfc1505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 -- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc06e560 msgr2=0x7fddfc15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc06e560 0x7fddfc15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 -- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddfc151cd0 msgr2=0x7fddfc1505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:56.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddfc151cd0 0x7fddfc1505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 -- 172.21.15.7:0/2991568645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec0094d0 con 0x7fddfc157b90 2023-11-28T18:30:56.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfaffd640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddfc151cd0 0x7fddfc1505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:30:56.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fddfbfff640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc157b90 0x7fddfc14c270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fddf4007e60 tx=0x7fddf400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:56.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fdde3fff640 1 -- 172.21.15.7:0/2991568645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4014420 con 0x7fddfc157b90 2023-11-28T18:30:56.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fdde3fff640 1 -- 172.21.15.7:0/2991568645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf40145a0 con 0x7fddfc157b90 2023-11-28T18:30:56.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddfc0c7780 con 0x7fddfc157b90 2023-11-28T18:30:56.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fdde3fff640 1 -- 172.21.15.7:0/2991568645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4014950 con 0x7fddfc157b90 2023-11-28T18:30:56.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.699+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddfc14fe60 con 0x7fddfc157b90 2023-11-28T18:30:56.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.703+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddfc0c7ca0 con 0x7fddfc157b90 2023-11-28T18:30:56.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.703+0000 7fdde3fff640 1 -- 172.21.15.7:0/2991568645 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fddf4014ab0 con 0x7fddfc157b90 2023-11-28T18:30:56.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.703+0000 7fdde3fff640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdde80668d0 0x7fdde8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:56.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.703+0000 7fdde3fff640 1 -- 172.21.15.7:0/2991568645 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fddf4084ea0 con 0x7fddfc157b90 2023-11-28T18:30:56.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.703+0000 7fddfb7fe640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdde80668d0 0x7fdde8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:56.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.703+0000 7fddfb7fe640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdde80668d0 0x7fdde8068d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fddec0ae000 tx=0x7fddec0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:56.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.707+0000 7fdde3fff640 1 -- 172.21.15.7:0/2991568645 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fddf4051f00 con 0x7fddfc157b90 2023-11-28T18:30:56.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.942+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddfc14fca0 con 0x7fddfc157b90 2023-11-28T18:30:56.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.942+0000 7fdde3fff640 1 -- 172.21.15.7:0/2991568645 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5758 (secure 0 0 0) 0x7fddf4055520 con 0x7fddfc157b90 2023-11-28T18:30:56.944 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:56.945 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:52.234279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:resolve","state_seq":8,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:56.945 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 43 2023-11-28T18:30:56.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdde80668d0 msgr2=0x7fdde8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:56.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdde80668d0 0x7fdde8068d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fddec0ae000 tx=0x7fddec0a7000 comp rx=0 tx=0).stop 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc157b90 msgr2=0x7fddfc14c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc157b90 0x7fddfc14c270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fddf4007e60 tx=0x7fddf400e4b0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 shutdown_connections 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdde80668d0 0x7fdde8068d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddfc157b90 0x7fddfc14c270 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddfc151cd0 0x7fddfc1505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 --2- 172.21.15.7:0/2991568645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fddfc06e560 0x7fddfc15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 >> 172.21.15.7:0/2991568645 conn(0x7fddfc0662e0 msgr2=0x7fddfc1610a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 shutdown_connections 2023-11-28T18:30:56.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:56.946+0000 7fde01e9a640 1 -- 172.21.15.7:0/2991568645 wait complete. 2023-11-28T18:30:56.970 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:30:58.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:30:59.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.142+0000 7f61ad162640 1 Processor -- start 2023-11-28T18:30:59.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.142+0000 7f61ad162640 1 -- start start 2023-11-28T18:30:59.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.142+0000 7f61ad162640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f61a815ca30 0x7f61a8159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.142+0000 7f61ad162640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61a80cafe0 0x7f61a80cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ad162640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a815a620 con 0x7f61a815ce10 2023-11-28T18:30:59.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ad162640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a815a900 con 0x7f61a815ca30 2023-11-28T18:30:59.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ad162640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a815abe0 con 0x7f61a80cafe0 2023-11-28T18:30:59.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a77fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f61a815ce10 0x7f61a80ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37102/0 (socket says 172.21.15.7:37102) 2023-11-28T18:30:59.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a77fe640 1 -- 172.21.15.7:0/3523879226 learned_addr learned my addr 172.21.15.7:0/3523879226 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a7fff640 1 --2- 172.21.15.7:0/3523879226 >> v2:172.21.15.193:3300/0 conn(0x7f61a815ca30 0x7f61a8159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ac961640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61a80cafe0 0x7f61a80cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099179066 0 0) 0x7f61a815a620 con 0x7f61a815ce10 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6190003530 con 0x7f61a815ce10 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544283223 0 0) 0x7f6190003530 con 0x7f61a815ce10 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61a80cafe0 msgr2=0x7f61a80cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61a80cafe0 0x7f61a80cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 >> v2:172.21.15.193:3300/0 conn(0x7f61a815ca30 msgr2=0x7f61a8159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 --2- 172.21.15.7:0/3523879226 >> v2:172.21.15.193:3300/0 conn(0x7f61a815ca30 0x7f61a8159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a80cbb10 con 0x7f61a815ce10 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f619c001680 con 0x7f61a815ce10 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 >> v1:172.21.15.7:6789/0 conn(0x7f61a815ce10 legacy=0x7f61a80ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6190003980 0x7f6190003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61900047d0 0x7f6190004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6190005190 0x7f619000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a815a900 con 0x7f6190005190 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a815abe0 con 0x7f6190003980 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619000bed0 con 0x7f61900047d0 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a7fff640 1 --2- 172.21.15.7:0/3523879226 >> v2:172.21.15.193:3300/0 conn(0x7f61a815ca30 0x7f61a8159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ac961640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61a80cafe0 0x7f61a80cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a77fe640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61900047d0 0x7f6190004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ac961640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6190005190 0x7f619000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61a7fff640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6190003980 0x7f6190003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ac961640 1 -- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61900047d0 msgr2=0x7f6190004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ac961640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61900047d0 0x7f6190004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ac961640 1 -- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6190003980 msgr2=0x7f6190003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:30:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ac961640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6190003980 0x7f6190003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.146+0000 7f61ac961640 1 -- 172.21.15.7:0/3523879226 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6190003530 con 0x7f6190005190 2023-11-28T18:30:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61ac961640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6190005190 0x7f619000b990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f61a80cb3c0 tx=0x7f61a001cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08dfbd990edcb40 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a00a5d50 con 0x7f6190005190 2023-11-28T18:30:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a000f040 con 0x7f6190005190 2023-11-28T18:30:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61a6ffd640 1 -- 172.21.15.7:0/3523879226 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0003bc0 con 0x7f6190005190 2023-11-28T18:30:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61ad162640 1 -- 172.21.15.7:0/3523879226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6190005190 msgr2=0x7f619000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61ad162640 1 --2- 172.21.15.7:0/3523879226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6190005190 0x7f619000b990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f61a80cb3c0 tx=0x7f61a001cd00 comp rx=0 tx=0).stop 2023-11-28T18:30:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61ac961640 1 -- 172.21.15.7:0/3523879226 reap_dead start 2023-11-28T18:30:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61ad162640 1 -- 172.21.15.7:0/3523879226 shutdown_connections 2023-11-28T18:30:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61ad162640 1 -- 172.21.15.7:0/3523879226 >> 172.21.15.7:0/3523879226 conn(0x7f61a806c060 msgr2=0x7f61a80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61ad162640 1 -- 172.21.15.7:0/3523879226 shutdown_connections 2023-11-28T18:30:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.150+0000 7f61ad162640 1 -- 172.21.15.7:0/3523879226 wait complete. 2023-11-28T18:30:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 Processor -- start 2023-11-28T18:30:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 -- start start 2023-11-28T18:30:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6190005190 0x7f61a814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a8154600 0x7f61a814e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61a8150a70 0x7f61a8150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8156130 con 0x7f61a8154600 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8151430 con 0x7f61a8150a70 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80c37d0 con 0x7f6190005190 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a8154600 0x7f61a814e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a8154600 0x7f61a814e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51592/0 (socket says 172.21.15.7:51592) 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 -- 172.21.15.7:0/3761128186 learned_addr learned my addr 172.21.15.7:0/3761128186 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a7fff640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6190005190 0x7f61a814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ac961640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61a8150a70 0x7f61a8150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 -- 172.21.15.7:0/3761128186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6190005190 msgr2=0x7f61a814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6190005190 0x7f61a814e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 -- 172.21.15.7:0/3761128186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61a8150a70 msgr2=0x7f61a8150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61a8150a70 0x7f61a8150ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 -- 172.21.15.7:0/3761128186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8153570 con 0x7f61a8154600 2023-11-28T18:30:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a77fe640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a8154600 0x7f61a814e590 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f619c007310 tx=0x7f619c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a4ff9640 1 -- 172.21.15.7:0/3761128186 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c007c20 con 0x7f61a8154600 2023-11-28T18:30:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a4ff9640 1 -- 172.21.15.7:0/3761128186 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c00f9c0 con 0x7f61a8154600 2023-11-28T18:30:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a8153850 con 0x7f61a8154600 2023-11-28T18:30:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61a4ff9640 1 -- 172.21.15.7:0/3761128186 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c00fd70 con 0x7f61a8154600 2023-11-28T18:30:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.154+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a8155ae0 con 0x7f61a8154600 2023-11-28T18:30:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.158+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a804e890 con 0x7f61a8154600 2023-11-28T18:30:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.158+0000 7f61a4ff9640 1 -- 172.21.15.7:0/3761128186 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f619c01d460 con 0x7f61a8154600 2023-11-28T18:30:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.158+0000 7f61a4ff9640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6194066900 0x7f6194068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:30:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.158+0000 7f61a4ff9640 1 -- 172.21.15.7:0/3761128186 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f619c0847c0 con 0x7f61a8154600 2023-11-28T18:30:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.158+0000 7f61a7fff640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6194066900 0x7f6194068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:30:59.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.158+0000 7f61a7fff640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6194066900 0x7f6194068dc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6198009de0 tx=0x7f6198009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:30:59.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.162+0000 7f61a4ff9640 1 -- 172.21.15.7:0/3761128186 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f619c051840 con 0x7f61a8154600 2023-11-28T18:30:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61a8066380 con 0x7f61a8154600 2023-11-28T18:30:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61a4ff9640 1 -- 172.21.15.7:0/3761128186 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5761 (secure 0 0 0) 0x7f619c054e60 con 0x7f61a8154600 2023-11-28T18:30:59.375 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:30:59.375 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:58.217648+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:30:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 44 2023-11-28T18:30:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6194066900 msgr2=0x7f6194068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6194066900 0x7f6194068dc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6198009de0 tx=0x7f6198009190 comp rx=0 tx=0).stop 2023-11-28T18:30:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a8154600 msgr2=0x7f61a814e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:30:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a8154600 0x7f61a814e590 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f619c007310 tx=0x7f619c0077d0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 shutdown_connections 2023-11-28T18:30:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6194066900 0x7f6194068dc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61a8150a70 0x7f61a8150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a8154600 0x7f61a814e590 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 --2- 172.21.15.7:0/3761128186 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6190005190 0x7f61a814e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:30:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 >> 172.21.15.7:0/3761128186 conn(0x7f61a806c060 msgr2=0x7f61a81599e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:30:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 shutdown_connections 2023-11-28T18:30:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:30:59.374+0000 7f61ad162640 1 -- 172.21.15.7:0/3761128186 wait complete. 2023-11-28T18:30:59.402 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:31:00.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:31:00.154 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:31:01.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:31:01.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.566+0000 7f6f02948640 1 Processor -- start 2023-11-28T18:31:01.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.566+0000 7f6f02948640 1 -- start start 2023-11-28T18:31:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f02948640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc0cafe0 0x7f6efc0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f02948640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc15a3a0 con 0x7f6efc15ce10 2023-11-28T18:31:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f02948640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc15a620 con 0x7f6efc15ca30 2023-11-28T18:31:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f02948640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc15a900 con 0x7f6efc0cafe0 2023-11-28T18:31:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f01145640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6efc15ce10 0x7f6efc0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37112/0 (socket says 172.21.15.7:37112) 2023-11-28T18:31:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f01145640 1 -- 172.21.15.7:0/2189449937 learned_addr learned my addr 172.21.15.7:0/2189449937 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:31:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f02147640 1 --2- 172.21.15.7:0/2189449937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc0cafe0 0x7f6efc0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f00944640 1 -- 172.21.15.7:0/2189449937 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077506285 0 0) 0x7f6efc15a3a0 con 0x7f6efc15ce10 2023-11-28T18:31:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f00944640 1 -- 172.21.15.7:0/2189449937 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee8003530 con 0x7f6efc15ce10 2023-11-28T18:31:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f00944640 1 -- 172.21.15.7:0/2189449937 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061128342 0 0) 0x7f6efc15a620 con 0x7f6efc15ca30 2023-11-28T18:31:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f00944640 1 -- 172.21.15.7:0/2189449937 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6efc15a3a0 con 0x7f6efc15ca30 2023-11-28T18:31:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f02147640 1 -- 172.21.15.7:0/2189449937 >> v1:172.21.15.193:6789/0 conn(0x7f6efc15ca30 legacy=0x7f6efc159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f02147640 1 -- 172.21.15.7:0/2189449937 >> v1:172.21.15.7:6789/0 conn(0x7f6efc15ce10 legacy=0x7f6efc0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.570+0000 7f6f02147640 1 -- 172.21.15.7:0/2189449937 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc15abe0 con 0x7f6efc0cafe0 2023-11-28T18:31:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02147640 1 --2- 172.21.15.7:0/2189449937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc0cafe0 0x7f6efc0cc5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009dc0 tx=0x7f6ef401f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=876519b45f62e074 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f00944640 1 -- 172.21.15.7:0/2189449937 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef40af600 con 0x7f6efc0cafe0 2023-11-28T18:31:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f00944640 1 -- 172.21.15.7:0/2189449937 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef40ab050 con 0x7f6efc0cafe0 2023-11-28T18:31:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f00944640 1 -- 172.21.15.7:0/2189449937 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef40af960 con 0x7f6efc0cafe0 2023-11-28T18:31:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 -- 172.21.15.7:0/2189449937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc0cafe0 msgr2=0x7f6efc0cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 --2- 172.21.15.7:0/2189449937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc0cafe0 0x7f6efc0cc5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009dc0 tx=0x7f6ef401f170 comp rx=0 tx=0).stop 2023-11-28T18:31:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 -- 172.21.15.7:0/2189449937 shutdown_connections 2023-11-28T18:31:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 --2- 172.21.15.7:0/2189449937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc0cafe0 0x7f6efc0cc5f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 -- 172.21.15.7:0/2189449937 >> 172.21.15.7:0/2189449937 conn(0x7f6efc06c060 msgr2=0x7f6efc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 -- 172.21.15.7:0/2189449937 shutdown_connections 2023-11-28T18:31:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 -- 172.21.15.7:0/2189449937 wait complete. 2023-11-28T18:31:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 Processor -- start 2023-11-28T18:31:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.574+0000 7f6f02948640 1 -- start start 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc0cafe0 0x7f6efc1e22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc15ca30 0x7f6efc1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6efc15ce10 0x7f6efc1596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc156680 con 0x7f6efc15ce10 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc156100 con 0x7f6efc0cafe0 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc1563a0 con 0x7f6efc15ca30 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02147640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6efc15ce10 0x7f6efc1596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01145640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc15ca30 0x7f6efc1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01946640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc0cafe0 0x7f6efc1e22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:01.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02147640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6efc15ce10 0x7f6efc1596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51594/0 (socket says 172.21.15.7:51594) 2023-11-28T18:31:01.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02147640 1 -- 172.21.15.7:0/3838719247 learned_addr learned my addr 172.21.15.7:0/3838719247 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:31:01.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01946640 1 -- 172.21.15.7:0/3838719247 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc15ca30 msgr2=0x7f6efc1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:01.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01946640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc15ca30 0x7f6efc1513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:01.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01946640 1 -- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6efc15ce10 msgr2=0x7f6efc1596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:01.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01946640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6efc15ce10 0x7f6efc1596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:01.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01946640 1 -- 172.21.15.7:0/3838719247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef4009a70 con 0x7f6efc0cafe0 2023-11-28T18:31:01.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01145640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc15ca30 0x7f6efc1513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:31:01.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02147640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6efc15ce10 0x7f6efc1596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:31:01.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f01946640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc0cafe0 0x7f6efc1e22c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c430 tx=0x7f6ef800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:01.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6ef27fc640 1 -- 172.21.15.7:0/3838719247 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80052d0 con 0x7f6efc0cafe0 2023-11-28T18:31:01.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6ef27fc640 1 -- 172.21.15.7:0/3838719247 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef8005430 con 0x7f6efc0cafe0 2023-11-28T18:31:01.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc1f0730 con 0x7f6efc0cafe0 2023-11-28T18:31:01.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6ef27fc640 1 -- 172.21.15.7:0/3838719247 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8003840 con 0x7f6efc0cafe0 2023-11-28T18:31:01.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc1e6a70 con 0x7f6efc0cafe0 2023-11-28T18:31:01.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.578+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6efc04e890 con 0x7f6efc0cafe0 2023-11-28T18:31:01.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.582+0000 7f6ef27fc640 1 -- 172.21.15.7:0/3838719247 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6ef80039a0 con 0x7f6efc0cafe0 2023-11-28T18:31:01.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.582+0000 7f6ef27fc640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6ee00668d0 0x7f6ee0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:01.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.582+0000 7f6ef27fc640 1 -- 172.21.15.7:0/3838719247 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f6ef8083c40 con 0x7f6efc0cafe0 2023-11-28T18:31:01.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.582+0000 7f6f01145640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6ee00668d0 0x7f6ee0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:01.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.582+0000 7f6f01145640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6ee00668d0 0x7f6ee0068d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0b4b60 tx=0x7f6eec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:01.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.586+0000 7f6ef27fc640 1 -- 172.21.15.7:0/3838719247 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6ef8050cc0 con 0x7f6efc0cafe0 2023-11-28T18:31:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.870+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6efc0cad10 con 0x7f6efc0cafe0 2023-11-28T18:31:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6ef27fc640 1 -- 172.21.15.7:0/3838719247 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5761 (secure 0 0 0) 0x7f6ef8017090 con 0x7f6efc0cafe0 2023-11-28T18:31:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:31:01.874 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:30:58.217648+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:31:01.874 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 44 2023-11-28T18:31:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6ee00668d0 msgr2=0x7f6ee0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6ee00668d0 0x7f6ee0068d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0b4b60 tx=0x7f6eec009040 comp rx=0 tx=0).stop 2023-11-28T18:31:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc0cafe0 msgr2=0x7f6efc1e22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc0cafe0 0x7f6efc1e22c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c430 tx=0x7f6ef800c8f0 comp rx=0 tx=0).stop 2023-11-28T18:31:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 shutdown_connections 2023-11-28T18:31:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6ee00668d0 0x7f6ee0068d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6efc15ce10 0x7f6efc1596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:01.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6efc15ca30 0x7f6efc1513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:01.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 --2- 172.21.15.7:0/3838719247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc0cafe0 0x7f6efc1e22c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:01.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 >> 172.21.15.7:0/3838719247 conn(0x7f6efc06c060 msgr2=0x7f6efc0ce0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:01.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 shutdown_connections 2023-11-28T18:31:01.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:01.874+0000 7f6f02948640 1 -- 172.21.15.7:0/3838719247 wait complete. 2023-11-28T18:31:01.900 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4288": { "addr": "172.21.15.7:6838/2690556675", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 2690556675, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 2690556675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:30:58.217648+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 4288, "mds_2": 5307, "mds_3": 5086, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:31:03.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:31:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.058+0000 7f77d2cfe640 1 Processor -- start 2023-11-28T18:31:04.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.058+0000 7f77d2cfe640 1 -- start start 2023-11-28T18:31:04.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77d2cfe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc1540e0 0x7f77cc0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77d2cfe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77cc15e380 con 0x7f77cc15a8a0 2023-11-28T18:31:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77d2cfe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77cc15d7e0 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77d2cfe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77cc0cab80 con 0x7f77cc1540e0 2023-11-28T18:31:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77cbfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f77cc15a8a0 0x7f77cc15dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59818/0 (socket says 172.21.15.7:59818) 2023-11-28T18:31:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77cbfff640 1 -- 172.21.15.7:0/1881560233 learned_addr learned my addr 172.21.15.7:0/1881560233 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:31:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77d0a9a640 1 --2- 172.21.15.7:0/1881560233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc1540e0 0x7f77cc0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77cb7fe640 1 -- 172.21.15.7:0/1881560233 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222865904 0 0) 0x7f77b8001120 con 0x7f77cc15a8a0 2023-11-28T18:31:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77cb7fe640 1 -- 172.21.15.7:0/1881560233 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77bc003530 con 0x7f77cc15a8a0 2023-11-28T18:31:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77cb7fe640 1 -- 172.21.15.7:0/1881560233 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402872958 0 0) 0x7f77cc15d7e0 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77cb7fe640 1 -- 172.21.15.7:0/1881560233 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b8001120 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77d0a9a640 1 -- 172.21.15.7:0/1881560233 >> v1:172.21.15.193:6789/0 conn(0x7f77cc0cf6c0 legacy=0x7f77cc164760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:04.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77d0a9a640 1 -- 172.21.15.7:0/1881560233 >> v1:172.21.15.7:6789/0 conn(0x7f77cc15a8a0 legacy=0x7f77cc15dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:04.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.062+0000 7f77d0a9a640 1 -- 172.21.15.7:0/1881560233 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77cc0cb850 con 0x7f77cc1540e0 2023-11-28T18:31:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d0a9a640 1 --2- 172.21.15.7:0/1881560233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc1540e0 0x7f77cc0ca4d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f77c00097f0 tx=0x7f77c001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dbf19b1c3649827 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77cb7fe640 1 -- 172.21.15.7:0/1881560233 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c00ab600 con 0x7f77cc1540e0 2023-11-28T18:31:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77cb7fe640 1 -- 172.21.15.7:0/1881560233 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c00a7050 con 0x7f77cc1540e0 2023-11-28T18:31:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77cb7fe640 1 -- 172.21.15.7:0/1881560233 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c00ab960 con 0x7f77cc1540e0 2023-11-28T18:31:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 -- 172.21.15.7:0/1881560233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc1540e0 msgr2=0x7f77cc0ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 --2- 172.21.15.7:0/1881560233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc1540e0 0x7f77cc0ca4d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f77c00097f0 tx=0x7f77c001c480 comp rx=0 tx=0).stop 2023-11-28T18:31:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 -- 172.21.15.7:0/1881560233 shutdown_connections 2023-11-28T18:31:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 --2- 172.21.15.7:0/1881560233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc1540e0 0x7f77cc0ca4d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 -- 172.21.15.7:0/1881560233 >> 172.21.15.7:0/1881560233 conn(0x7f77cc0662e0 msgr2=0x7f77cc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:04.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 -- 172.21.15.7:0/1881560233 shutdown_connections 2023-11-28T18:31:04.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 -- 172.21.15.7:0/1881560233 wait complete. 2023-11-28T18:31:04.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 Processor -- start 2023-11-28T18:31:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.066+0000 7f77d2cfe640 1 -- start start 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77cc0cf6c0 0x7f77cc156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77cc1540e0 0x7f77cc15cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc15a8a0 0x7f77cc15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77cc15d2f0 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77cc156b40 con 0x7f77cc1540e0 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77cc156840 con 0x7f77cc15a8a0 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77cc0cf6c0 0x7f77cc156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77cc0cf6c0 0x7f77cc156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45652/0 (socket says 172.21.15.7:45652) 2023-11-28T18:31:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 -- 172.21.15.7:0/563623418 learned_addr learned my addr 172.21.15.7:0/563623418 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d129b640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc15a8a0 0x7f77cc15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77cbfff640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77cc1540e0 0x7f77cc15cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 -- 172.21.15.7:0/563623418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc15a8a0 msgr2=0x7f77cc15afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc15a8a0 0x7f77cc15afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 -- 172.21.15.7:0/563623418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77cc1540e0 msgr2=0x7f77cc15cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77cc1540e0 0x7f77cc15cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 -- 172.21.15.7:0/563623418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c0009420 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d0a9a640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77cc0cf6c0 0x7f77cc156fd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f77c0000c10 tx=0x7f77c0019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77c97fa640 1 -- 172.21.15.7:0/563623418 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c00b4400 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77c97fa640 1 -- 172.21.15.7:0/563623418 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c00b4560 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77c97fa640 1 -- 172.21.15.7:0/563623418 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c00b48d0 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77cc159580 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77cc159b00 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.070+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77cc04e890 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.074+0000 7f77c97fa640 1 -- 172.21.15.7:0/563623418 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f77c00b2040 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.074+0000 7f77c97fa640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f77b00668d0 0x7f77b0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:04.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.074+0000 7f77c97fa640 1 -- 172.21.15.7:0/563623418 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f77c011e340 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.074+0000 7f77cbfff640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f77b00668d0 0x7f77b0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.074+0000 7f77cbfff640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f77b00668d0 0x7f77b0068d90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f77cc1472b0 tx=0x7f77b8007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.078+0000 7f77c97fa640 1 -- 172.21.15.7:0/563623418 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f77c00eb3c0 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.290+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77cc156980 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77c97fa640 1 -- 172.21.15.7:0/563623418 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5758 (secure 0 0 0) 0x7f77c00f3e60 con 0x7f77cc0cf6c0 2023-11-28T18:31:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:31:04.294 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:03.353358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:31:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 45 2023-11-28T18:31:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f77b00668d0 msgr2=0x7f77b0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f77b00668d0 0x7f77b0068d90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f77cc1472b0 tx=0x7f77b8007d40 comp rx=0 tx=0).stop 2023-11-28T18:31:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77cc0cf6c0 msgr2=0x7f77cc156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77cc0cf6c0 0x7f77cc156fd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f77c0000c10 tx=0x7f77c0019220 comp rx=0 tx=0).stop 2023-11-28T18:31:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 shutdown_connections 2023-11-28T18:31:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f77b00668d0 0x7f77b0068d90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77cc15a8a0 0x7f77cc15afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77cc1540e0 0x7f77cc15cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 --2- 172.21.15.7:0/563623418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77cc0cf6c0 0x7f77cc156fd0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 >> 172.21.15.7:0/563623418 conn(0x7f77cc0662e0 msgr2=0x7f77cc0726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 shutdown_connections 2023-11-28T18:31:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:04.294+0000 7f77d2cfe640 1 -- 172.21.15.7:0/563623418 wait complete. 2023-11-28T18:31:04.320 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:31:06.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:31:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.482+0000 7f8b6b632640 1 Processor -- start 2023-11-28T18:31:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.482+0000 7f8b6b632640 1 -- start start 2023-11-28T18:31:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b6b632640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b64072450 0x7f8b64072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b6b632640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640ca920 con 0x7f8b641542e0 2023-11-28T18:31:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b6b632640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640c4cd0 con 0x7f8b640cbb70 2023-11-28T18:31:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b6b632640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b64071d50 con 0x7f8b64072450 2023-11-28T18:31:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b69bcf640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b64072450 0x7f8b64072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b693ce640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8b641542e0 0x7f8b6415e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59828/0 (socket says 172.21.15.7:59828) 2023-11-28T18:31:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b693ce640 1 -- 172.21.15.7:0/3940226065 learned_addr learned my addr 172.21.15.7:0/3940226065 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:31:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b69bcf640 1 -- 172.21.15.7:0/3940226065 >> v1:172.21.15.193:6789/0 conn(0x7f8b640cbb70 legacy=0x7f8b640cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:06.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b69bcf640 1 -- 172.21.15.7:0/3940226065 >> v1:172.21.15.7:6789/0 conn(0x7f8b641542e0 legacy=0x7f8b6415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:06.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.486+0000 7f8b69bcf640 1 -- 172.21.15.7:0/3940226065 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b640cab70 con 0x7f8b64072450 2023-11-28T18:31:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b69bcf640 1 --2- 172.21.15.7:0/3940226065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b64072450 0x7f8b64072870 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009d90 tx=0x7f8b5c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb21733bb5a6d714 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b5bfff640 1 -- 172.21.15.7:0/3940226065 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c0a5920 con 0x7f8b64072450 2023-11-28T18:31:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b5bfff640 1 -- 172.21.15.7:0/3940226065 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c0a5a80 con 0x7f8b64072450 2023-11-28T18:31:06.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b5bfff640 1 -- 172.21.15.7:0/3940226065 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c0a5e10 con 0x7f8b64072450 2023-11-28T18:31:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b6b632640 1 -- 172.21.15.7:0/3940226065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b64072450 msgr2=0x7f8b64072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b6b632640 1 --2- 172.21.15.7:0/3940226065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b64072450 0x7f8b64072870 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009d90 tx=0x7f8b5c01c950 comp rx=0 tx=0).stop 2023-11-28T18:31:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b6b632640 1 -- 172.21.15.7:0/3940226065 shutdown_connections 2023-11-28T18:31:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b6b632640 1 --2- 172.21.15.7:0/3940226065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b64072450 0x7f8b64072870 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b6b632640 1 -- 172.21.15.7:0/3940226065 >> 172.21.15.7:0/3940226065 conn(0x7f8b640717e0 msgr2=0x7f8b640662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b6b632640 1 -- 172.21.15.7:0/3940226065 shutdown_connections 2023-11-28T18:31:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.490+0000 7f8b6b632640 1 -- 172.21.15.7:0/3940226065 wait complete. 2023-11-28T18:31:06.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 Processor -- start 2023-11-28T18:31:06.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 -- start start 2023-11-28T18:31:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b64072450 0x7f8b6415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b640cbb70 0x7f8b64158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b641542e0 0x7f8b641f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64072030 con 0x7f8b640cbb70 2023-11-28T18:31:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64156110 con 0x7f8b64072450 2023-11-28T18:31:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6415a8c0 con 0x7f8b641542e0 2023-11-28T18:31:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b640cbb70 0x7f8b64158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b640cbb70 0x7f8b64158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45660/0 (socket says 172.21.15.7:45660) 2023-11-28T18:31:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 -- 172.21.15.7:0/1738473307 learned_addr learned my addr 172.21.15.7:0/1738473307 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:31:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b693ce640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b64072450 0x7f8b6415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b69bcf640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b641542e0 0x7f8b641f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 -- 172.21.15.7:0/1738473307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b641542e0 msgr2=0x7f8b641f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b641542e0 0x7f8b641f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 -- 172.21.15.7:0/1738473307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b64072450 msgr2=0x7f8b6415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b64072450 0x7f8b6415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 -- 172.21.15.7:0/1738473307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c009a70 con 0x7f8b640cbb70 2023-11-28T18:31:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b68bcd640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b640cbb70 0x7f8b64158c20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c450 tx=0x7f8b5400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b59ffb640 1 -- 172.21.15.7:0/1738473307 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540052d0 con 0x7f8b640cbb70 2023-11-28T18:31:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b59ffb640 1 -- 172.21.15.7:0/1738473307 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b54005430 con 0x7f8b640cbb70 2023-11-28T18:31:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6415a080 con 0x7f8b640cbb70 2023-11-28T18:31:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b59ffb640 1 -- 172.21.15.7:0/1738473307 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54007260 con 0x7f8b640cbb70 2023-11-28T18:31:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.494+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b64159500 con 0x7f8b640cbb70 2023-11-28T18:31:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.498+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6406c670 con 0x7f8b640cbb70 2023-11-28T18:31:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.498+0000 7f8b59ffb640 1 -- 172.21.15.7:0/1738473307 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8b540073c0 con 0x7f8b640cbb70 2023-11-28T18:31:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.498+0000 7f8b59ffb640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8b480668d0 0x7f8b48068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.498+0000 7f8b59ffb640 1 -- 172.21.15.7:0/1738473307 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f8b54083e40 con 0x7f8b640cbb70 2023-11-28T18:31:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.498+0000 7f8b693ce640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8b480668d0 0x7f8b48068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.498+0000 7f8b693ce640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8b480668d0 0x7f8b48068d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8b60005890 tx=0x7f8b60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.502+0000 7f8b59ffb640 1 -- 172.21.15.7:0/1738473307 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8b54050ec0 con 0x7f8b640cbb70 2023-11-28T18:31:06.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.718+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b6404e890 con 0x7f8b640cbb70 2023-11-28T18:31:06.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b59ffb640 1 -- 172.21.15.7:0/1738473307 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5758 (secure 0 0 0) 0x7f8b540544e0 con 0x7f8b640cbb70 2023-11-28T18:31:06.721 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:31:06.722 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:03.353358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:31:06.722 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 45 2023-11-28T18:31:06.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8b480668d0 msgr2=0x7f8b48068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8b480668d0 0x7f8b48068d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8b60005890 tx=0x7f8b60009040 comp rx=0 tx=0).stop 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b640cbb70 msgr2=0x7f8b64158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b640cbb70 0x7f8b64158c20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c450 tx=0x7f8b5400c910 comp rx=0 tx=0).stop 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 shutdown_connections 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8b480668d0 0x7f8b48068d90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b641542e0 0x7f8b641f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b640cbb70 0x7f8b64158c20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 --2- 172.21.15.7:0/1738473307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b64072450 0x7f8b6415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 >> 172.21.15.7:0/1738473307 conn(0x7f8b640717e0 msgr2=0x7f8b640662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 shutdown_connections 2023-11-28T18:31:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:06.722+0000 7f8b6b632640 1 -- 172.21.15.7:0/1738473307 wait complete. 2023-11-28T18:31:06.746 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:31:08.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:31:08.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5e92a640 1 Processor -- start 2023-11-28T18:31:08.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5e92a640 1 -- start start 2023-11-28T18:31:08.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5e92a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5f581542e0 0x7f5f5815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:08.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5e92a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f580cbb70 0x7f5f580cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:08.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5e92a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f580cc810 0x7f5f58072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:08.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5e92a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f58071d50 con 0x7f5f580cc810 2023-11-28T18:31:08.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5e92a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f58072030 con 0x7f5f581542e0 2023-11-28T18:31:08.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5e92a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580ca920 con 0x7f5f580cbb70 2023-11-28T18:31:08.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5cec7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f580cc810 0x7f5f58072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5cec7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f580cc810 0x7f5f58072450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45664/0 (socket says 172.21.15.7:45664) 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5cec7640 1 -- 172.21.15.7:0/1236715878 learned_addr learned my addr 172.21.15.7:0/1236715878 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f577fe640 1 --2- 172.21.15.7:0/1236715878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f580cbb70 0x7f5f580cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f57fff640 1 --2- 172.21.15.7:0/1236715878 >> v2:172.21.15.193:3300/0 conn(0x7f5f581542e0 0x7f5f5815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5cec7640 1 -- 172.21.15.7:0/1236715878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f580cbb70 msgr2=0x7f5f580cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5cec7640 1 --2- 172.21.15.7:0/1236715878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f580cbb70 0x7f5f580cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5cec7640 1 -- 172.21.15.7:0/1236715878 >> v2:172.21.15.193:3300/0 conn(0x7f5f581542e0 msgr2=0x7f5f5815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5cec7640 1 --2- 172.21.15.7:0/1236715878 >> v2:172.21.15.193:3300/0 conn(0x7f5f581542e0 0x7f5f5815e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f5cec7640 1 -- 172.21.15.7:0/1236715878 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5815e550 con 0x7f5f580cc810 2023-11-28T18:31:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f577fe640 1 --2- 172.21.15.7:0/1236715878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f580cbb70 0x7f5f580cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:31:08.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.918+0000 7f5f57fff640 1 --2- 172.21.15.7:0/1236715878 >> v2:172.21.15.193:3300/0 conn(0x7f5f581542e0 0x7f5f5815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:31:08.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.922+0000 7f5f5cec7640 1 --2- 172.21.15.7:0/1236715878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cc810 0x7f5f58072450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5f58071d20 tx=0x7f5f5001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729ffdce8121c703 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:08.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.922+0000 7f5f56ffd640 1 -- 172.21.15.7:0/1236715878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f500a5760 con 0x7f5f580cc810 2023-11-28T18:31:08.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.922+0000 7f5f56ffd640 1 -- 172.21.15.7:0/1236715878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f500a58c0 con 0x7f5f580cc810 2023-11-28T18:31:08.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.922+0000 7f5f56ffd640 1 -- 172.21.15.7:0/1236715878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f500a5c50 con 0x7f5f580cc810 2023-11-28T18:31:08.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.922+0000 7f5f5e92a640 1 -- 172.21.15.7:0/1236715878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cc810 msgr2=0x7f5f58072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:08.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.922+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/1236715878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cc810 0x7f5f58072450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5f58071d20 tx=0x7f5f5001c9d0 comp rx=0 tx=0).stop 2023-11-28T18:31:08.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 -- 172.21.15.7:0/1236715878 shutdown_connections 2023-11-28T18:31:08.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/1236715878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cc810 0x7f5f58072450 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:08.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/1236715878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f580cbb70 0x7f5f580cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:08.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/1236715878 >> v2:172.21.15.193:3300/0 conn(0x7f5f581542e0 0x7f5f5815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:08.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 -- 172.21.15.7:0/1236715878 >> 172.21.15.7:0/1236715878 conn(0x7f5f580717e0 msgr2=0x7f5f580662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:08.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 -- 172.21.15.7:0/1236715878 shutdown_connections 2023-11-28T18:31:08.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 -- 172.21.15.7:0/1236715878 wait complete. 2023-11-28T18:31:08.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 Processor -- start 2023-11-28T18:31:08.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 -- start start 2023-11-28T18:31:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cbb70 0x7f5f5815d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f580cc810 0x7f5f5815cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f581542e0 0x7f5f580c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f581f29a0 con 0x7f5f580cbb70 2023-11-28T18:31:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f581f2c40 con 0x7f5f580cc810 2023-11-28T18:31:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5e92a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5815b700 con 0x7f5f581542e0 2023-11-28T18:31:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f57fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cbb70 0x7f5f5815d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f57fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cbb70 0x7f5f5815d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45678/0 (socket says 172.21.15.7:45678) 2023-11-28T18:31:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f57fff640 1 -- 172.21.15.7:0/2417687725 learned_addr learned my addr 172.21.15.7:0/2417687725 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:31:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f577fe640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f580cc810 0x7f5f5815cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f5cec7640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f581542e0 0x7f5f580c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f57fff640 1 -- 172.21.15.7:0/2417687725 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f581542e0 msgr2=0x7f5f580c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f57fff640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f581542e0 0x7f5f580c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f57fff640 1 -- 172.21.15.7:0/2417687725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f580cc810 msgr2=0x7f5f5815cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f57fff640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f580cc810 0x7f5f5815cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.926+0000 7f5f57fff640 1 -- 172.21.15.7:0/2417687725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f50009b30 con 0x7f5f580cbb70 2023-11-28T18:31:08.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f57fff640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cbb70 0x7f5f5815d4f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800e810 tx=0x7f5f4800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:08.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f54ff9640 1 -- 172.21.15.7:0/2417687725 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4800c8d0 con 0x7f5f580cbb70 2023-11-28T18:31:08.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f54ff9640 1 -- 172.21.15.7:0/2417687725 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4800ca30 con 0x7f5f580cbb70 2023-11-28T18:31:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f58151250 con 0x7f5f580cbb70 2023-11-28T18:31:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f54ff9640 1 -- 172.21.15.7:0/2417687725 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4800cdc0 con 0x7f5f580cbb70 2023-11-28T18:31:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f581514c0 con 0x7f5f580cbb70 2023-11-28T18:31:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5806c670 con 0x7f5f580cbb70 2023-11-28T18:31:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f54ff9640 1 -- 172.21.15.7:0/2417687725 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5f48013410 con 0x7f5f580cbb70 2023-11-28T18:31:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f54ff9640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5f440668d0 0x7f5f44068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.930+0000 7f5f54ff9640 1 -- 172.21.15.7:0/2417687725 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f5f48084470 con 0x7f5f580cbb70 2023-11-28T18:31:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.934+0000 7f5f577fe640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5f440668d0 0x7f5f44068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.934+0000 7f5f577fe640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5f440668d0 0x7f5f44068d90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5f58152d80 tx=0x7f5f4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:08.938+0000 7f5f54ff9640 1 -- 172.21.15.7:0/2417687725 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5f48051460 con 0x7f5f580cbb70 2023-11-28T18:31:09.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.178+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f581516e0 con 0x7f5f580cbb70 2023-11-28T18:31:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.178+0000 7f5f54ff9640 1 -- 172.21.15.7:0/2417687725 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5758 (secure 0 0 0) 0x7f5f48054a80 con 0x7f5f580cbb70 2023-11-28T18:31:09.180 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:31:09.180 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:31:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 46 2023-11-28T18:31:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5f440668d0 msgr2=0x7f5f44068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5f440668d0 0x7f5f44068d90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5f58152d80 tx=0x7f5f4c008040 comp rx=0 tx=0).stop 2023-11-28T18:31:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cbb70 msgr2=0x7f5f5815d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cbb70 0x7f5f5815d4f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800e810 tx=0x7f5f4800ecd0 comp rx=0 tx=0).stop 2023-11-28T18:31:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 shutdown_connections 2023-11-28T18:31:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5f440668d0 0x7f5f44068d90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5f581542e0 0x7f5f580c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f580cc810 0x7f5f5815cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 --2- 172.21.15.7:0/2417687725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f580cbb70 0x7f5f5815d4f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 >> 172.21.15.7:0/2417687725 conn(0x7f5f580717e0 msgr2=0x7f5f580b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 shutdown_connections 2023-11-28T18:31:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:09.182+0000 7f5f5e92a640 1 -- 172.21.15.7:0/2417687725 wait complete. 2023-11-28T18:31:09.204 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=2, replacing gid=4285 2023-11-28T18:31:09.205 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.d 2023-11-28T18:31:16.206 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-11-28T18:31:16.206 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T18:31:16.206 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T18:31:16.210 INFO:tasks.ceph.mds.d:Started 2023-11-28T18:31:16.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:31:16.298 INFO:tasks.ceph.mds.d.smithi193.stdout:starting mds.d at 2023-11-28T18:31:16.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.362+0000 7fef80b80640 1 Processor -- start 2023-11-28T18:31:16.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef80b80640 1 -- start start 2023-11-28T18:31:16.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef80b80640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fef7c1542e0 0x7fef7c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:16.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef80b80640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef7c0cbb70 0x7fef7c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:16.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef80b80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c0cc810 0x7fef7c072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:16.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef80b80640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c071d50 con 0x7fef7c0cbb70 2023-11-28T18:31:16.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef80b80640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c072030 con 0x7fef7c1542e0 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef80b80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c0ca920 con 0x7fef7c0cc810 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef79d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef7c0cbb70 0x7fef7c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef79d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef7c0cbb70 0x7fef7c0cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45542/0 (socket says 172.21.15.7:45542) 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef79d9b640 1 -- 172.21.15.7:0/3605291266 learned_addr learned my addr 172.21.15.7:0/3605291266 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef7a59c640 1 --2- 172.21.15.7:0/3605291266 >> v2:172.21.15.193:3300/0 conn(0x7fef7c1542e0 0x7fef7c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef7ad9d640 1 --2- 172.21.15.7:0/3605291266 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c0cc810 0x7fef7c072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef79d9b640 1 -- 172.21.15.7:0/3605291266 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c0cc810 msgr2=0x7fef7c072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef79d9b640 1 --2- 172.21.15.7:0/3605291266 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c0cc810 0x7fef7c072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef79d9b640 1 -- 172.21.15.7:0/3605291266 >> v2:172.21.15.193:3300/0 conn(0x7fef7c1542e0 msgr2=0x7fef7c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef79d9b640 1 --2- 172.21.15.7:0/3605291266 >> v2:172.21.15.193:3300/0 conn(0x7fef7c1542e0 0x7fef7c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.366+0000 7fef79d9b640 1 -- 172.21.15.7:0/3605291266 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c15e550 con 0x7fef7c0cbb70 2023-11-28T18:31:16.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef79d9b640 1 --2- 172.21.15.7:0/3605291266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cc2d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fef7c071d20 tx=0x7fef7001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88c1f5e4c91db6e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef7959a640 1 -- 172.21.15.7:0/3605291266 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700ac630 con 0x7fef7c0cbb70 2023-11-28T18:31:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef7959a640 1 -- 172.21.15.7:0/3605291266 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef700ac790 con 0x7fef7c0cbb70 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef7959a640 1 -- 172.21.15.7:0/3605291266 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700acb20 con 0x7fef7c0cbb70 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 -- 172.21.15.7:0/3605291266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cbb70 msgr2=0x7fef7c0cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 --2- 172.21.15.7:0/3605291266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cc2d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fef7c071d20 tx=0x7fef7001c500 comp rx=0 tx=0).stop 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 -- 172.21.15.7:0/3605291266 shutdown_connections 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 --2- 172.21.15.7:0/3605291266 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c0cc810 0x7fef7c072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 --2- 172.21.15.7:0/3605291266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cc2d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 --2- 172.21.15.7:0/3605291266 >> v2:172.21.15.193:3300/0 conn(0x7fef7c1542e0 0x7fef7c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 -- 172.21.15.7:0/3605291266 >> 172.21.15.7:0/3605291266 conn(0x7fef7c0717e0 msgr2=0x7fef7c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 -- 172.21.15.7:0/3605291266 shutdown_connections 2023-11-28T18:31:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.370+0000 7fef80b80640 1 -- 172.21.15.7:0/3605291266 wait complete. 2023-11-28T18:31:16.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 Processor -- start 2023-11-28T18:31:16.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 -- start start 2023-11-28T18:31:16.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:16.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cc810 0x7fef7c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c1542e0 0x7fef7c0cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c1f2bb0 con 0x7fef7c0cc810 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c1f3a10 con 0x7fef7c0cbb70 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c1f2e00 con 0x7fef7c1542e0 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef79d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cc810 0x7fef7c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef7a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef7a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:60146/0 (socket says 172.21.15.7:60146) 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef7a59c640 1 -- 172.21.15.7:0/1257577979 learned_addr learned my addr 172.21.15.7:0/1257577979 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef7ad9d640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c1542e0 0x7fef7c0cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef79d9b640 1 -- 172.21.15.7:0/1257577979 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c1542e0 msgr2=0x7fef7c0cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef79d9b640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c1542e0 0x7fef7c0cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef79d9b640 1 -- 172.21.15.7:0/1257577979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef7c0cbb70 msgr2=0x7fef7c0cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef79d9b640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef79d9b640 1 -- 172.21.15.7:0/1257577979 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef700094d0 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef7a59c640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef79d9b640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cc810 0x7fef7c0ce6a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fef700ac000 tx=0x7fef7c15b080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef6affd640 1 -- 172.21.15.7:0/1257577979 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700ba070 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef6affd640 1 -- 172.21.15.7:0/1257577979 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef700b5440 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef6affd640 1 -- 172.21.15.7:0/1257577979 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700b57d0 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c1f30e0 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.374+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c1f36a0 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.378+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c06c670 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.378+0000 7fef6affd640 1 -- 172.21.15.7:0/1257577979 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fef700b3070 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.378+0000 7fef6affd640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef5c0668d0 0x7fef5c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.378+0000 7fef6affd640 1 -- 172.21.15.7:0/1257577979 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fef7011d9a0 con 0x7fef7c0cc810 2023-11-28T18:31:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.378+0000 7fef7a59c640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef5c0668d0 0x7fef5c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:31:16.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.378+0000 7fef7a59c640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef5c0668d0 0x7fef5c068d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009f70 tx=0x7fef6c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:31:16.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.382+0000 7fef6affd640 1 -- 172.21.15.7:0/1257577979 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fef700eabe0 con 0x7fef7c0cc810 2023-11-28T18:31:16.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.602+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef7c04e890 con 0x7fef7c0cc810 2023-11-28T18:31:16.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.602+0000 7fef6affd640 1 -- 172.21.15.7:0/1257577979 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6551 (secure 0 0 0) 0x7fef700a8090 con 0x7fef7c0cc810 2023-11-28T18:31:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:31:16.603 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:31:16.603 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 47 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef5c0668d0 msgr2=0x7fef5c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef5c0668d0 0x7fef5c068d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009f70 tx=0x7fef6c009190 comp rx=0 tx=0).stop 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cc810 msgr2=0x7fef7c0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cc810 0x7fef7c0ce6a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fef700ac000 tx=0x7fef7c15b080 comp rx=0 tx=0).stop 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 shutdown_connections 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fef5c0668d0 0x7fef5c068d90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fef7c1542e0 0x7fef7c0cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef7c0cc810 0x7fef7c0ce6a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 --2- 172.21.15.7:0/1257577979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef7c0cbb70 0x7fef7c0cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 >> 172.21.15.7:0/1257577979 conn(0x7fef7c0717e0 msgr2=0x7fef7c0c5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:31:16.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 shutdown_connections 2023-11-28T18:31:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:31:16.606+0000 7fef80b80640 1 -- 172.21.15.7:0/1257577979 wait complete. 2023-11-28T18:31:16.628 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-11-28T18:31:16.628 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2023-11-28T18:31:30.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:31:30.358 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:32:00.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:32:00.412 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:32:30.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:32:30.477 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:32:37.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba2d7c4640 1 Processor -- start 2023-11-28T18:32:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba2d7c4640 1 -- start start 2023-11-28T18:32:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba2d7c4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba28072450 0x7fba28072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba2d7c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280ca920 con 0x7fba281542e0 2023-11-28T18:32:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba2d7c4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280c4cd0 con 0x7fba280cbb70 2023-11-28T18:32:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba2d7c4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba28071d50 con 0x7fba28072450 2023-11-28T18:32:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba26ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fba281542e0 0x7fba2815e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60276/0 (socket says 172.21.15.7:60276) 2023-11-28T18:32:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba277fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba28072450 0x7fba28072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.785+0000 7fba26ffd640 1 -- 172.21.15.7:0/1977942207 learned_addr learned my addr 172.21.15.7:0/1977942207 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:32:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.789+0000 7fba277fe640 1 -- 172.21.15.7:0/1977942207 >> v1:172.21.15.193:6789/0 conn(0x7fba280cbb70 legacy=0x7fba280cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.789+0000 7fba277fe640 1 -- 172.21.15.7:0/1977942207 >> v1:172.21.15.7:6789/0 conn(0x7fba281542e0 legacy=0x7fba2815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.789+0000 7fba277fe640 1 -- 172.21.15.7:0/1977942207 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba280cab70 con 0x7fba28072450 2023-11-28T18:32:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.789+0000 7fba277fe640 1 --2- 172.21.15.7:0/1977942207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba28072870 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fba20009d90 tx=0x7fba2001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f906c042706ffd6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.789+0000 7fba25ffb640 1 -- 172.21.15.7:0/1977942207 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200a5920 con 0x7fba28072450 2023-11-28T18:32:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.789+0000 7fba25ffb640 1 -- 172.21.15.7:0/1977942207 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba200a5a80 con 0x7fba28072450 2023-11-28T18:32:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.789+0000 7fba25ffb640 1 -- 172.21.15.7:0/1977942207 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200a5e10 con 0x7fba28072450 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- 172.21.15.7:0/1977942207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 msgr2=0x7fba28072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 --2- 172.21.15.7:0/1977942207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba28072870 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fba20009d90 tx=0x7fba2001c950 comp rx=0 tx=0).stop 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- 172.21.15.7:0/1977942207 shutdown_connections 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 --2- 172.21.15.7:0/1977942207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba28072870 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- 172.21.15.7:0/1977942207 >> 172.21.15.7:0/1977942207 conn(0x7fba280717e0 msgr2=0x7fba280662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- 172.21.15.7:0/1977942207 shutdown_connections 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- 172.21.15.7:0/1977942207 wait complete. 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 Processor -- start 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- start start 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba281464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba280cbb70 0x7fba28145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba281542e0 0x7fba28144870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba28072030 con 0x7fba28072450 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba281549c0 con 0x7fba280cbb70 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba2d7c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba281443c0 con 0x7fba281542e0 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba26ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba281464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba26ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba281464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56736/0 (socket says 172.21.15.7:56736) 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba26ffd640 1 -- 172.21.15.7:0/2773972421 learned_addr learned my addr 172.21.15.7:0/2773972421 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba267fc640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba280cbb70 0x7fba28145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.793+0000 7fba277fe640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba281542e0 0x7fba28144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba26ffd640 1 -- 172.21.15.7:0/2773972421 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba281542e0 msgr2=0x7fba28144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba26ffd640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba281542e0 0x7fba28144870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba26ffd640 1 -- 172.21.15.7:0/2773972421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba280cbb70 msgr2=0x7fba28145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba26ffd640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba280cbb70 0x7fba28145780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba26ffd640 1 -- 172.21.15.7:0/2773972421 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20009a70 con 0x7fba28072450 2023-11-28T18:32:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba26ffd640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba281464c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fba1800c400 tx=0x7fba1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba0f7fe640 1 -- 172.21.15.7:0/2773972421 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18005cf0 con 0x7fba28072450 2023-11-28T18:32:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba0f7fe640 1 -- 172.21.15.7:0/2773972421 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18005e50 con 0x7fba28072450 2023-11-28T18:32:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba28157f90 con 0x7fba28072450 2023-11-28T18:32:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba0f7fe640 1 -- 172.21.15.7:0/2773972421 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180072d0 con 0x7fba28072450 2023-11-28T18:32:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba281584f0 con 0x7fba28072450 2023-11-28T18:32:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2806c670 con 0x7fba28072450 2023-11-28T18:32:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.797+0000 7fba0f7fe640 1 -- 172.21.15.7:0/2773972421 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fba18007430 con 0x7fba28072450 2023-11-28T18:32:37.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.801+0000 7fba0f7fe640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba14066900 0x7fba14068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:37.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.801+0000 7fba0f7fe640 1 -- 172.21.15.7:0/2773972421 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fba18083af0 con 0x7fba28072450 2023-11-28T18:32:37.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.801+0000 7fba267fc640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba14066900 0x7fba14068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:37.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.801+0000 7fba267fc640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba14066900 0x7fba14068dc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fba281f7230 tx=0x7fba1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:37.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:37.805+0000 7fba0f7fe640 1 -- 172.21.15.7:0/2773972421 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fba18050b70 con 0x7fba28072450 2023-11-28T18:32:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba28144550 con 0x7fba28072450 2023-11-28T18:32:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba0f7fe640 1 -- 172.21.15.7:0/2773972421 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6551 (secure 0 0 0) 0x7fba18017090 con 0x7fba28072450 2023-11-28T18:32:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:38.039 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 47 2023-11-28T18:32:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba14066900 msgr2=0x7fba14068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba14066900 0x7fba14068dc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fba281f7230 tx=0x7fba1c009040 comp rx=0 tx=0).stop 2023-11-28T18:32:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 msgr2=0x7fba281464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba281464c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fba1800c400 tx=0x7fba1800c8c0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 shutdown_connections 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba14066900 0x7fba14068dc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba281542e0 0x7fba28144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba280cbb70 0x7fba28145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 --2- 172.21.15.7:0/2773972421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba28072450 0x7fba281464c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 >> 172.21.15.7:0/2773972421 conn(0x7fba280717e0 msgr2=0x7fba280c6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 shutdown_connections 2023-11-28T18:32:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.037+0000 7fba2d7c4640 1 -- 172.21.15.7:0/2773972421 wait complete. 2023-11-28T18:32:38.061 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-11-28T18:32:38.061 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T18:32:38.062 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:32:38.077 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T18:32:38.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:38.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.277+0000 7f9ede2bb640 1 Processor -- start 2023-11-28T18:32:38.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.277+0000 7f9ede2bb640 1 -- start start 2023-11-28T18:32:38.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.277+0000 7f9ede2bb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9ed8151cd0 0x7f9ed815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:38.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.277+0000 7f9ede2bb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9ed806e560 0x7f9ed8157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.277+0000 7f9ede2bb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed8071e60 con 0x7f9ed80701f0 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.277+0000 7f9ede2bb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed8070930 con 0x7f9ed806e560 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.277+0000 7f9ede2bb640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed8070bf0 con 0x7f9ed8151cd0 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9edc858640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9ed80701f0 0x7f9ed80c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60280/0 (socket says 172.21.15.7:60280) 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9edc858640 1 -- 172.21.15.7:0/1905931090 learned_addr learned my addr 172.21.15.7:0/1905931090 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9ed77fe640 1 --2- 172.21.15.7:0/1905931090 >> v2:172.21.15.7:3300/0 conn(0x7f9ed806e560 0x7f9ed8157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9ed7fff640 1 --2- 172.21.15.7:0/1905931090 >> v2:172.21.15.193:3300/0 conn(0x7f9ed8151cd0 0x7f9ed815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9ed77fe640 1 -- 172.21.15.7:0/1905931090 >> v2:172.21.15.193:3300/0 conn(0x7f9ed8151cd0 msgr2=0x7f9ed815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9ed77fe640 1 --2- 172.21.15.7:0/1905931090 >> v2:172.21.15.193:3300/0 conn(0x7f9ed8151cd0 0x7f9ed815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9ed77fe640 1 -- 172.21.15.7:0/1905931090 >> v1:172.21.15.7:6789/0 conn(0x7f9ed80701f0 legacy=0x7f9ed80c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:38.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9ed77fe640 1 -- 172.21.15.7:0/1905931090 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed8070ed0 con 0x7f9ed806e560 2023-11-28T18:32:38.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.281+0000 7f9ed6ffd640 1 -- 172.21.15.7:0/1905931090 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819980443 0 0) 0x7f9ed8071e60 con 0x7f9ed80701f0 2023-11-28T18:32:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ed77fe640 1 --2- 172.21.15.7:0/1905931090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed806e560 0x7f9ed8157b90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9ed806fee0 tx=0x7f9ecc01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d08fc5e9dea5d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ed6ffd640 1 -- 172.21.15.7:0/1905931090 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc0ab600 con 0x7f9ed806e560 2023-11-28T18:32:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ed6ffd640 1 -- 172.21.15.7:0/1905931090 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc0a7050 con 0x7f9ed806e560 2023-11-28T18:32:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ed6ffd640 1 -- 172.21.15.7:0/1905931090 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc0ab980 con 0x7f9ed806e560 2023-11-28T18:32:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1905931090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed806e560 msgr2=0x7f9ed8157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1905931090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed806e560 0x7f9ed8157b90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9ed806fee0 tx=0x7f9ecc01c3d0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1905931090 shutdown_connections 2023-11-28T18:32:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1905931090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed806e560 0x7f9ed8157b90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1905931090 >> v2:172.21.15.193:3300/0 conn(0x7f9ed8151cd0 0x7f9ed815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1905931090 >> 172.21.15.7:0/1905931090 conn(0x7f9ed80662e0 msgr2=0x7f9ed80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1905931090 shutdown_connections 2023-11-28T18:32:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1905931090 wait complete. 2023-11-28T18:32:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 Processor -- start 2023-11-28T18:32:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.285+0000 7f9ede2bb640 1 -- start start 2023-11-28T18:32:38.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ed806e560 0x7f9ed8158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:38.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9ed80701f0 0x7f9ed815a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:38.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed8151cd0 0x7f9ed80ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:38.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed815a9a0 con 0x7f9ed8151cd0 2023-11-28T18:32:38.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed8159f80 con 0x7f9ed806e560 2023-11-28T18:32:38.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed815acd0 con 0x7f9ed80701f0 2023-11-28T18:32:38.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed8151cd0 0x7f9ed80ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:38.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed8151cd0 0x7f9ed80ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56742/0 (socket says 172.21.15.7:56742) 2023-11-28T18:32:38.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 -- 172.21.15.7:0/1733768656 learned_addr learned my addr 172.21.15.7:0/1733768656 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:38.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ed77fe640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9ed80701f0 0x7f9ed815a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:38.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ed7fff640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ed806e560 0x7f9ed8158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:38.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 -- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9ed80701f0 msgr2=0x7f9ed815a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:38.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9ed80701f0 0x7f9ed815a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 -- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ed806e560 msgr2=0x7f9ed8158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ed806e560 0x7f9ed8158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 -- 172.21.15.7:0/1733768656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc009420 con 0x7f9ed8151cd0 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ed77fe640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9ed80701f0 0x7f9ed815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9edc858640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed8151cd0 0x7f9ed80ca010 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0001570 tx=0x7f9ed000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ed4ff9640 1 -- 172.21.15.7:0/1733768656 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed000a640 con 0x7f9ed8151cd0 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ed4ff9640 1 -- 172.21.15.7:0/1733768656 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed000a7c0 con 0x7f9ed8151cd0 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ed4ff9640 1 -- 172.21.15.7:0/1733768656 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed000ab70 con 0x7f9ed8151cd0 2023-11-28T18:32:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed815afb0 con 0x7f9ed8151cd0 2023-11-28T18:32:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed8159910 con 0x7f9ed8151cd0 2023-11-28T18:32:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.289+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed804e890 con 0x7f9ed8151cd0 2023-11-28T18:32:38.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.293+0000 7f9ed4ff9640 1 -- 172.21.15.7:0/1733768656 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9ed000acd0 con 0x7f9ed8151cd0 2023-11-28T18:32:38.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.293+0000 7f9ed4ff9640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9ec4066900 0x7f9ec4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:38.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.293+0000 7f9ed4ff9640 1 -- 172.21.15.7:0/1733768656 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f9ed00839b0 con 0x7f9ed8151cd0 2023-11-28T18:32:38.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.293+0000 7f9ed7fff640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9ec4066900 0x7f9ec4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:38.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.293+0000 7f9ed7fff640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9ec4066900 0x7f9ec4068dc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80079c0 tx=0x7f9ec8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.297+0000 7f9ed4ff9640 1 -- 172.21.15.7:0/1733768656 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9ed0050b60 con 0x7f9ed8151cd0 2023-11-28T18:32:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.569+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ed81555b0 con 0x7f9ed8151cd0 2023-11-28T18:32:38.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.569+0000 7f9ed4ff9640 1 -- 172.21.15.7:0/1733768656 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f9ed0018020 con 0x7f9ed8151cd0 2023-11-28T18:32:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:38.571 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:38.571 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 47 2023-11-28T18:32:38.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9ec4066900 msgr2=0x7f9ec4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9ec4066900 0x7f9ec4068dc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80079c0 tx=0x7f9ec8008040 comp rx=0 tx=0).stop 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed8151cd0 msgr2=0x7f9ed80ca010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed8151cd0 0x7f9ed80ca010 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0001570 tx=0x7f9ed000cca0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 shutdown_connections 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9ec4066900 0x7f9ec4068dc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed8151cd0 0x7f9ed80ca010 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9ed80701f0 0x7f9ed815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 --2- 172.21.15.7:0/1733768656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ed806e560 0x7f9ed8158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 >> 172.21.15.7:0/1733768656 conn(0x7f9ed80662e0 msgr2=0x7f9ed80685e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 shutdown_connections 2023-11-28T18:32:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:38.573+0000 7f9ede2bb640 1 -- 172.21.15.7:0/1733768656 wait complete. 2023-11-28T18:32:38.596 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:32:40.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:40.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f013049c640 1 Processor -- start 2023-11-28T18:32:40.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f013049c640 1 -- start start 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f013049c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0128150b30 0x7f01280c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f013049c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f01280c7980 0x7f01280c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f013049c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01280c6140 con 0x7f01281575c0 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f013049c640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01280c72b0 con 0x7f01280c7980 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f013049c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0128070c00 con 0x7f0128150b30 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012da37640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f01281575c0 0x7f01280c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60290/0 (socket says 172.21.15.7:60290) 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012da37640 1 -- 172.21.15.7:0/4077685402 learned_addr learned my addr 172.21.15.7:0/4077685402 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012e238640 1 --2- 172.21.15.7:0/4077685402 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0128150b30 0x7f01280c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012ea39640 1 --2- 172.21.15.7:0/4077685402 >> v2:172.21.15.193:3300/0 conn(0x7f01280c7980 0x7f01280c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012d236640 1 -- 172.21.15.7:0/4077685402 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152546219 0 0) 0x7f01280c6140 con 0x7f01281575c0 2023-11-28T18:32:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012d236640 1 -- 172.21.15.7:0/4077685402 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0114003530 con 0x7f01281575c0 2023-11-28T18:32:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012e238640 1 -- 172.21.15.7:0/4077685402 >> v2:172.21.15.193:3300/0 conn(0x7f01280c7980 msgr2=0x7f01280c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012e238640 1 --2- 172.21.15.7:0/4077685402 >> v2:172.21.15.193:3300/0 conn(0x7f01280c7980 0x7f01280c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012e238640 1 -- 172.21.15.7:0/4077685402 >> v1:172.21.15.7:6789/0 conn(0x7f01281575c0 legacy=0x7f01280c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012e238640 1 -- 172.21.15.7:0/4077685402 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01280c8350 con 0x7f0128150b30 2023-11-28T18:32:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.741+0000 7f012ea39640 1 --2- 172.21.15.7:0/4077685402 >> v2:172.21.15.193:3300/0 conn(0x7f01280c7980 0x7f01280c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:32:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f012e238640 1 --2- 172.21.15.7:0/4077685402 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0128150b30 0x7f01280c24f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f01240097c0 tx=0x7f012401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76cecb6829df07d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f012d236640 1 -- 172.21.15.7:0/4077685402 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240a8970 con 0x7f0128150b30 2023-11-28T18:32:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f012d236640 1 -- 172.21.15.7:0/4077685402 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01240a8ad0 con 0x7f0128150b30 2023-11-28T18:32:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f012d236640 1 -- 172.21.15.7:0/4077685402 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240a8e60 con 0x7f0128150b30 2023-11-28T18:32:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f013049c640 1 -- 172.21.15.7:0/4077685402 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0128150b30 msgr2=0x7f01280c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f013049c640 1 --2- 172.21.15.7:0/4077685402 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0128150b30 0x7f01280c24f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f01240097c0 tx=0x7f012401ec70 comp rx=0 tx=0).stop 2023-11-28T18:32:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f013049c640 1 -- 172.21.15.7:0/4077685402 shutdown_connections 2023-11-28T18:32:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f013049c640 1 --2- 172.21.15.7:0/4077685402 >> v2:172.21.15.193:3300/0 conn(0x7f01280c7980 0x7f01280c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f013049c640 1 --2- 172.21.15.7:0/4077685402 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0128150b30 0x7f01280c24f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.745+0000 7f013049c640 1 -- 172.21.15.7:0/4077685402 >> 172.21.15.7:0/4077685402 conn(0x7f01280662e0 msgr2=0x7f01280666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 -- 172.21.15.7:0/4077685402 shutdown_connections 2023-11-28T18:32:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 -- 172.21.15.7:0/4077685402 wait complete. 2023-11-28T18:32:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 Processor -- start 2023-11-28T18:32:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 -- start start 2023-11-28T18:32:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01280c7980 0x7f0128154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128150b30 0x7f01281ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01281575c0 0x7f0128071780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0128152290 con 0x7f01280c7980 2023-11-28T18:32:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01281f0160 con 0x7f0128150b30 2023-11-28T18:32:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f013049c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01281f0440 con 0x7f01281575c0 2023-11-28T18:32:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01280c7980 0x7f0128154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01280c7980 0x7f0128154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56756/0 (socket says 172.21.15.7:56756) 2023-11-28T18:32:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 -- 172.21.15.7:0/4221463873 learned_addr learned my addr 172.21.15.7:0/4221463873 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012da37640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128150b30 0x7f01281ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012ea39640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01281575c0 0x7f0128071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 -- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01281575c0 msgr2=0x7f0128071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01281575c0 0x7f0128071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 -- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128150b30 msgr2=0x7f01281ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128150b30 0x7f01281ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 -- 172.21.15.7:0/4221463873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0120009a70 con 0x7f01280c7980 2023-11-28T18:32:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012da37640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128150b30 0x7f01281ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:32:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012ea39640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01281575c0 0x7f0128071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:32:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.749+0000 7f012e238640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01280c7980 0x7f0128154100 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f01240b3040 tx=0x7f012401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f011effd640 1 -- 172.21.15.7:0/4221463873 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240b0d60 con 0x7f01280c7980 2023-11-28T18:32:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f011effd640 1 -- 172.21.15.7:0/4221463873 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01240b9430 con 0x7f01280c7980 2023-11-28T18:32:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f011effd640 1 -- 172.21.15.7:0/4221463873 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240b97e0 con 0x7f01280c7980 2023-11-28T18:32:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0124009420 con 0x7f01280c7980 2023-11-28T18:32:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01281527c0 con 0x7f01280c7980 2023-11-28T18:32:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012804e890 con 0x7f01280c7980 2023-11-28T18:32:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f011effd640 1 -- 172.21.15.7:0/4221463873 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f01240b7040 con 0x7f01280c7980 2023-11-28T18:32:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f011effd640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f010c0668d0 0x7f010c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f011effd640 1 -- 172.21.15.7:0/4221463873 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f0124122160 con 0x7f01280c7980 2023-11-28T18:32:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.753+0000 7f012da37640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f010c0668d0 0x7f010c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.757+0000 7f012da37640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f010c0668d0 0x7f010c068d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f01280c1930 tx=0x7f0118009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.757+0000 7f011effd640 1 -- 172.21.15.7:0/4221463873 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01240ef130 con 0x7f01280c7980 2023-11-28T18:32:40.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.949+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01280c2d80 con 0x7f01280c7980 2023-11-28T18:32:40.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.949+0000 7f011effd640 1 -- 172.21.15.7:0/4221463873 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f01240f2750 con 0x7f01280c7980 2023-11-28T18:32:40.953 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:40.953 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:40.953 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 47 2023-11-28T18:32:40.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f010c0668d0 msgr2=0x7f010c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f010c0668d0 0x7f010c068d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f01280c1930 tx=0x7f0118009040 comp rx=0 tx=0).stop 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01280c7980 msgr2=0x7f0128154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01280c7980 0x7f0128154100 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f01240b3040 tx=0x7f012401ba10 comp rx=0 tx=0).stop 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 shutdown_connections 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f010c0668d0 0x7f010c068d90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01281575c0 0x7f0128071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128150b30 0x7f01281ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 --2- 172.21.15.7:0/4221463873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01280c7980 0x7f0128154100 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 >> 172.21.15.7:0/4221463873 conn(0x7f01280662e0 msgr2=0x7f01281e5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 shutdown_connections 2023-11-28T18:32:40.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:40.953+0000 7f013049c640 1 -- 172.21.15.7:0/4221463873 wait complete. 2023-11-28T18:32:40.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:32:42.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.125+0000 7f7062ec9640 1 Processor -- start 2023-11-28T18:32:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7062ec9640 1 -- start start 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7062ec9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 0x7f705c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7062ec9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f705c06e560 0x7f705c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7062ec9640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f705c0701f0 0x7f705c0c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7062ec9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705c071e60 con 0x7f705c06e560 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7062ec9640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705c070930 con 0x7f705c0701f0 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7062ec9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705c070660 con 0x7f705c151cd0 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f705bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f705c06e560 0x7f705c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f705bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f705c06e560 0x7f705c157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56768/0 (socket says 172.21.15.7:56768) 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f705bfff640 1 -- 172.21.15.7:0/2960137017 learned_addr learned my addr 172.21.15.7:0/2960137017 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7060c65640 1 --2- 172.21.15.7:0/2960137017 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 0x7f705c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f7061466640 1 --2- 172.21.15.7:0/2960137017 >> v2:172.21.15.193:3300/0 conn(0x7f705c0701f0 0x7f705c0c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f705bfff640 1 -- 172.21.15.7:0/2960137017 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 msgr2=0x7f705c15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f705bfff640 1 --2- 172.21.15.7:0/2960137017 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 0x7f705c15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f705bfff640 1 -- 172.21.15.7:0/2960137017 >> v2:172.21.15.193:3300/0 conn(0x7f705c0701f0 msgr2=0x7f705c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f705bfff640 1 --2- 172.21.15.7:0/2960137017 >> v2:172.21.15.193:3300/0 conn(0x7f705c0701f0 0x7f705c0c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.129+0000 7f705bfff640 1 -- 172.21.15.7:0/2960137017 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c070bf0 con 0x7f705c06e560 2023-11-28T18:32:43.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f705bfff640 1 --2- 172.21.15.7:0/2960137017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c157b90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f705c0c95f0 tx=0x7f704801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225530e884a7c339 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:43.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f705b7fe640 1 -- 172.21.15.7:0/2960137017 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70480ab600 con 0x7f705c06e560 2023-11-28T18:32:43.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f705b7fe640 1 -- 172.21.15.7:0/2960137017 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70480a7050 con 0x7f705c06e560 2023-11-28T18:32:43.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f705b7fe640 1 -- 172.21.15.7:0/2960137017 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70480ab980 con 0x7f705c06e560 2023-11-28T18:32:43.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 -- 172.21.15.7:0/2960137017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 msgr2=0x7f705c157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:43.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2960137017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c157b90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f705c0c95f0 tx=0x7f704801c470 comp rx=0 tx=0).stop 2023-11-28T18:32:43.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 -- 172.21.15.7:0/2960137017 shutdown_connections 2023-11-28T18:32:43.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2960137017 >> v2:172.21.15.193:3300/0 conn(0x7f705c0701f0 0x7f705c0c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2960137017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c157b90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2960137017 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 0x7f705c15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 -- 172.21.15.7:0/2960137017 >> 172.21.15.7:0/2960137017 conn(0x7f705c0662e0 msgr2=0x7f705c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:43.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 -- 172.21.15.7:0/2960137017 shutdown_connections 2023-11-28T18:32:43.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 -- 172.21.15.7:0/2960137017 wait complete. 2023-11-28T18:32:43.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 Processor -- start 2023-11-28T18:32:43.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.133+0000 7f7062ec9640 1 -- start start 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c1598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f705c0701f0 0x7f705c15a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 0x7f705c152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c153d90 con 0x7f705c06e560 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c1553e0 con 0x7f705c0701f0 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705c0c80a0 con 0x7f705c151cd0 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c1598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c1598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56770/0 (socket says 172.21.15.7:56770) 2023-11-28T18:32:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 -- 172.21.15.7:0/2443084174 learned_addr learned my addr 172.21.15.7:0/2443084174 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:43.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7061466640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 0x7f705c152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:43.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f705bfff640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f705c0701f0 0x7f705c15a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 -- 172.21.15.7:0/2443084174 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 msgr2=0x7f705c152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 0x7f705c152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 -- 172.21.15.7:0/2443084174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f705c0701f0 msgr2=0x7f705c15a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f705c0701f0 0x7f705c15a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 -- 172.21.15.7:0/2443084174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70480094d0 con 0x7f705c06e560 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7060c65640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c1598d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f705000e870 tx=0x7f705000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f70597fa640 1 -- 172.21.15.7:0/2443084174 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7050009510 con 0x7f705c06e560 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f70597fa640 1 -- 172.21.15.7:0/2443084174 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705000eec0 con 0x7f705c06e560 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705c0c7ae0 con 0x7f705c06e560 2023-11-28T18:32:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f70597fa640 1 -- 172.21.15.7:0/2443084174 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705000cb40 con 0x7f705c06e560 2023-11-28T18:32:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705c158df0 con 0x7f705c06e560 2023-11-28T18:32:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.137+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705c04e890 con 0x7f705c06e560 2023-11-28T18:32:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.141+0000 7f70597fa640 1 -- 172.21.15.7:0/2443084174 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f705000cca0 con 0x7f705c06e560 2023-11-28T18:32:43.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.141+0000 7f70597fa640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70400668d0 0x7f7040068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:43.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.141+0000 7f70597fa640 1 -- 172.21.15.7:0/2443084174 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f70500846a0 con 0x7f705c06e560 2023-11-28T18:32:43.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.141+0000 7f705bfff640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70400668d0 0x7f7040068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:43.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.141+0000 7f705bfff640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70400668d0 0x7f7040068d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f70480ad000 tx=0x7f7048008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:43.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.145+0000 7f70597fa640 1 -- 172.21.15.7:0/2443084174 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f70500517b0 con 0x7f705c06e560 2023-11-28T18:32:43.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.333+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f705c1615d0 con 0x7f705c06e560 2023-11-28T18:32:43.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.333+0000 7f70597fa640 1 -- 172.21.15.7:0/2443084174 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f7050054dd0 con 0x7f705c06e560 2023-11-28T18:32:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:43.335 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:43.336 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 47 2023-11-28T18:32:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.333+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70400668d0 msgr2=0x7f7040068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.333+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70400668d0 0x7f7040068d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f70480ad000 tx=0x7f7048008040 comp rx=0 tx=0).stop 2023-11-28T18:32:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.333+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 msgr2=0x7f705c1598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.333+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c1598d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f705000e870 tx=0x7f705000ec70 comp rx=0 tx=0).stop 2023-11-28T18:32:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.337+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 shutdown_connections 2023-11-28T18:32:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.337+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f70400668d0 0x7f7040068d90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.337+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f705c151cd0 0x7f705c152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.337+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f705c0701f0 0x7f705c15a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.337+0000 7f7062ec9640 1 --2- 172.21.15.7:0/2443084174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705c06e560 0x7f705c1598d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:43.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.337+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 >> 172.21.15.7:0/2443084174 conn(0x7f705c0662e0 msgr2=0x7f705c1606e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:43.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.337+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 shutdown_connections 2023-11-28T18:32:43.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:43.337+0000 7f7062ec9640 1 -- 172.21.15.7:0/2443084174 wait complete. 2023-11-28T18:32:43.356 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:32:45.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.509+0000 7f293b2c9640 1 Processor -- start 2023-11-28T18:32:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f293b2c9640 1 -- start start 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f293b2c9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29340cbb70 0x7f29340cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f293b2c9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cc810 0x7f2934072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f293b2c9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29340ca920 con 0x7f29341542e0 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f293b2c9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2934071d50 con 0x7f29340cbb70 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f293b2c9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2934072030 con 0x7f29340cc810 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2938864640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29340cbb70 0x7f29340cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2938864640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29340cbb70 0x7f29340cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56946/0 (socket says 172.21.15.7:56946) 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2938864640 1 -- 172.21.15.7:0/3599292853 learned_addr learned my addr 172.21.15.7:0/3599292853 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2939866640 1 --2- 172.21.15.7:0/3599292853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cc810 0x7f2934072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2938864640 1 -- 172.21.15.7:0/3599292853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cc810 msgr2=0x7f2934072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:45.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2938864640 1 --2- 172.21.15.7:0/3599292853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cc810 0x7f2934072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2938864640 1 -- 172.21.15.7:0/3599292853 >> v1:172.21.15.193:6789/0 conn(0x7f29341542e0 legacy=0x7f293415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:45.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2938864640 1 -- 172.21.15.7:0/3599292853 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29340c4cd0 con 0x7f29340cbb70 2023-11-28T18:32:45.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.513+0000 7f2939866640 1 --2- 172.21.15.7:0/3599292853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cc810 0x7f2934072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:32:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f2938864640 1 --2- 172.21.15.7:0/3599292853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cbb70 0x7f29340cc2d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f29240098a0 tx=0x7f292401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474d38ae2054bc08 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f292bfff640 1 -- 172.21.15.7:0/3599292853 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29240a7070 con 0x7f29340cbb70 2023-11-28T18:32:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f292bfff640 1 -- 172.21.15.7:0/3599292853 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29240ab430 con 0x7f29340cbb70 2023-11-28T18:32:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f292bfff640 1 -- 172.21.15.7:0/3599292853 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29240ab7c0 con 0x7f29340cbb70 2023-11-28T18:32:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 -- 172.21.15.7:0/3599292853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cbb70 msgr2=0x7f29340cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3599292853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cbb70 0x7f29340cc2d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f29240098a0 tx=0x7f292401c520 comp rx=0 tx=0).stop 2023-11-28T18:32:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 -- 172.21.15.7:0/3599292853 shutdown_connections 2023-11-28T18:32:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3599292853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cc810 0x7f2934072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3599292853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cbb70 0x7f29340cc2d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 -- 172.21.15.7:0/3599292853 >> 172.21.15.7:0/3599292853 conn(0x7f29340717e0 msgr2=0x7f29340662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 -- 172.21.15.7:0/3599292853 shutdown_connections 2023-11-28T18:32:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 -- 172.21.15.7:0/3599292853 wait complete. 2023-11-28T18:32:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 Processor -- start 2023-11-28T18:32:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.517+0000 7f293b2c9640 1 -- start start 2023-11-28T18:32:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cbb70 0x7f29340d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cc810 0x7f293413f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29341542e0 0x7f29341417f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29340cf310 con 0x7f29340cc810 2023-11-28T18:32:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29340cda80 con 0x7f29341542e0 2023-11-28T18:32:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29340d0650 con 0x7f29340cbb70 2023-11-28T18:32:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cc810 0x7f293413f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cc810 0x7f293413f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56950/0 (socket says 172.21.15.7:56950) 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 -- 172.21.15.7:0/3861350633 learned_addr learned my addr 172.21.15.7:0/3861350633 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2939065640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cbb70 0x7f29340d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2939866640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29341542e0 0x7f29341417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 -- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cbb70 msgr2=0x7f29340d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cbb70 0x7f29340d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 -- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29341542e0 msgr2=0x7f29341417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29341542e0 0x7f29341417f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 -- 172.21.15.7:0/3861350633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29240094d0 con 0x7f29340cc810 2023-11-28T18:32:45.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2939866640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29341542e0 0x7f29341417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:32:45.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2938864640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cc810 0x7f293413f510 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f29240ad040 tx=0x7f29240192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:45.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2929ffb640 1 -- 172.21.15.7:0/3861350633 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29240b4830 con 0x7f29340cc810 2023-11-28T18:32:45.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2929ffb640 1 -- 172.21.15.7:0/3861350633 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29240b49b0 con 0x7f29340cc810 2023-11-28T18:32:45.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29340cea70 con 0x7f29340cc810 2023-11-28T18:32:45.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f2929ffb640 1 -- 172.21.15.7:0/3861350633 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29240b4d60 con 0x7f29340cc810 2023-11-28T18:32:45.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29340cf030 con 0x7f29340cc810 2023-11-28T18:32:45.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.521+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293406c670 con 0x7f29340cc810 2023-11-28T18:32:45.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.525+0000 7f2929ffb640 1 -- 172.21.15.7:0/3861350633 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f29240bf3a0 con 0x7f29340cc810 2023-11-28T18:32:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.525+0000 7f2929ffb640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f29180668d0 0x7f2918068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.525+0000 7f2929ffb640 1 -- 172.21.15.7:0/3861350633 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f292411df10 con 0x7f29340cc810 2023-11-28T18:32:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.525+0000 7f2939065640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f29180668d0 0x7f2918068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.525+0000 7f2939065640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f29180668d0 0x7f2918068d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2930000f70 tx=0x7f2930009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.529+0000 7f2929ffb640 1 -- 172.21.15.7:0/3861350633 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f29240eaf70 con 0x7f29340cc810 2023-11-28T18:32:45.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.717+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f293404e890 con 0x7f29340cc810 2023-11-28T18:32:45.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.717+0000 7f2929ffb640 1 -- 172.21.15.7:0/3861350633 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f29240ee590 con 0x7f29340cc810 2023-11-28T18:32:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:45.720 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:45.720 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 47 2023-11-28T18:32:45.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f29180668d0 msgr2=0x7f2918068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:45.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f29180668d0 0x7f2918068d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2930000f70 tx=0x7f2930009040 comp rx=0 tx=0).stop 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cc810 msgr2=0x7f293413f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cc810 0x7f293413f510 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f29240ad040 tx=0x7f29240192c0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 shutdown_connections 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f29180668d0 0x7f2918068d90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29341542e0 0x7f29341417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29340cc810 0x7f293413f510 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 --2- 172.21.15.7:0/3861350633 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f29340cbb70 0x7f29340d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 >> 172.21.15.7:0/3861350633 conn(0x7f29340717e0 msgr2=0x7f29341597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 shutdown_connections 2023-11-28T18:32:45.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:45.721+0000 7f293b2c9640 1 -- 172.21.15.7:0/3861350633 wait complete. 2023-11-28T18:32:45.740 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:32:47.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:47.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc7279640 1 Processor -- start 2023-11-28T18:32:47.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc7279640 1 -- start start 2023-11-28T18:32:47.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc7279640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fabc00c7600 0x7fabc015a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:47.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc7279640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00c1b50 con 0x7fabc0150840 2023-11-28T18:32:47.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc7279640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc006ccc0 con 0x7fabc00c1700 2023-11-28T18:32:47.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc7279640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc006cf90 con 0x7fabc00c7600 2023-11-28T18:32:47.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc5015640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fabc0150840 0x7fabc00c7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53552/0 (socket says 172.21.15.7:53552) 2023-11-28T18:32:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc5015640 1 -- 172.21.15.7:0/3431415142 learned_addr learned my addr 172.21.15.7:0/3431415142 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:32:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabc4814640 1 --2- 172.21.15.7:0/3431415142 >> v2:172.21.15.193:3300/0 conn(0x7fabc00c7600 0x7fabc015a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabb7fff640 1 -- 172.21.15.7:0/3431415142 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629484959 0 0) 0x7fabc00c1b50 con 0x7fabc0150840 2023-11-28T18:32:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.897+0000 7fabb7fff640 1 -- 172.21.15.7:0/3431415142 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac004640 con 0x7fabc0150840 2023-11-28T18:32:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabc4814640 1 -- 172.21.15.7:0/3431415142 >> v1:172.21.15.193:6789/0 conn(0x7fabc00c1700 legacy=0x7fabc006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabc4814640 1 -- 172.21.15.7:0/3431415142 >> v1:172.21.15.7:6789/0 conn(0x7fabc0150840 legacy=0x7fabc00c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabc4814640 1 -- 172.21.15.7:0/3431415142 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc006d270 con 0x7fabc00c7600 2023-11-28T18:32:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabc4814640 1 --2- 172.21.15.7:0/3431415142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 0x7fabc015a3a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fabb00097c0 tx=0x7fabb001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b88e0d0789bd652a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:47.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabb7fff640 1 -- 172.21.15.7:0/3431415142 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb00a8970 con 0x7fabc00c7600 2023-11-28T18:32:47.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabb7fff640 1 -- 172.21.15.7:0/3431415142 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb00a8ad0 con 0x7fabc00c7600 2023-11-28T18:32:47.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabb7fff640 1 -- 172.21.15.7:0/3431415142 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb00a8e60 con 0x7fabc00c7600 2023-11-28T18:32:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabc7279640 1 -- 172.21.15.7:0/3431415142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 msgr2=0x7fabc015a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.901+0000 7fabc7279640 1 --2- 172.21.15.7:0/3431415142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 0x7fabc015a3a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fabb00097c0 tx=0x7fabb001ec70 comp rx=0 tx=0).stop 2023-11-28T18:32:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 -- 172.21.15.7:0/3431415142 shutdown_connections 2023-11-28T18:32:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 --2- 172.21.15.7:0/3431415142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 0x7fabc015a3a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 -- 172.21.15.7:0/3431415142 >> 172.21.15.7:0/3431415142 conn(0x7fabc00cbf80 msgr2=0x7fabc00cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 -- 172.21.15.7:0/3431415142 shutdown_connections 2023-11-28T18:32:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 -- 172.21.15.7:0/3431415142 wait complete. 2023-11-28T18:32:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 Processor -- start 2023-11-28T18:32:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 -- start start 2023-11-28T18:32:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc00c1700 0x7fabc00cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 0x7fabc014e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc0150840 0x7fabc01458f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01f03a0 con 0x7fabc00c1700 2023-11-28T18:32:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc014d490 con 0x7fabc00c7600 2023-11-28T18:32:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc7279640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc014d730 con 0x7fabc0150840 2023-11-28T18:32:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc00c1700 0x7fabc00cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc00c1700 0x7fabc00cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56962/0 (socket says 172.21.15.7:56962) 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 -- 172.21.15.7:0/1335189077 learned_addr learned my addr 172.21.15.7:0/1335189077 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc4814640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 0x7fabc014e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5816640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc0150840 0x7fabc01458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 -- 172.21.15.7:0/1335189077 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc0150840 msgr2=0x7fabc01458f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc0150840 0x7fabc01458f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 -- 172.21.15.7:0/1335189077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 msgr2=0x7fabc014e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 0x7fabc014e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 -- 172.21.15.7:0/1335189077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb0009420 con 0x7fabc00c1700 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.905+0000 7fabc5015640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc00c1700 0x7fabc00cc5c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00c420 tx=0x7fabbc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabb5ffb640 1 -- 172.21.15.7:0/1335189077 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc005cf0 con 0x7fabc00c1700 2023-11-28T18:32:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabb5ffb640 1 -- 172.21.15.7:0/1335189077 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc005e50 con 0x7fabc00c1700 2023-11-28T18:32:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabb5ffb640 1 -- 172.21.15.7:0/1335189077 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc0072d0 con 0x7fabc00c1700 2023-11-28T18:32:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc014da10 con 0x7fabc00c1700 2023-11-28T18:32:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc014dfd0 con 0x7fabc00c1700 2023-11-28T18:32:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc004e890 con 0x7fabc00c1700 2023-11-28T18:32:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabb5ffb640 1 -- 172.21.15.7:0/1335189077 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fabbc007430 con 0x7fabc00c1700 2023-11-28T18:32:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabb5ffb640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faba40668d0 0x7faba4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabb5ffb640 1 -- 172.21.15.7:0/1335189077 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fabbc083c30 con 0x7fabc00c1700 2023-11-28T18:32:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabc4814640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faba40668d0 0x7faba4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.909+0000 7fabc4814640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faba40668d0 0x7faba4068d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabb00b2000 tx=0x7fabb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:47.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:47.913+0000 7fabb5ffb640 1 -- 172.21.15.7:0/1335189077 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fabbc050cb0 con 0x7fabc00c1700 2023-11-28T18:32:48.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.101+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabc00c7b40 con 0x7fabc00c1700 2023-11-28T18:32:48.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabb5ffb640 1 -- 172.21.15.7:0/1335189077 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6551 (secure 0 0 0) 0x7fabbc017020 con 0x7fabc00c1700 2023-11-28T18:32:48.106 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:48.106 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:48.107 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 47 2023-11-28T18:32:48.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faba40668d0 msgr2=0x7faba4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:48.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faba40668d0 0x7faba4068d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabb00b2000 tx=0x7fabb0008040 comp rx=0 tx=0).stop 2023-11-28T18:32:48.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc00c1700 msgr2=0x7fabc00cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:48.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc00c1700 0x7fabc00cc5c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00c420 tx=0x7fabbc00c8e0 comp rx=0 tx=0).stop 2023-11-28T18:32:48.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 shutdown_connections 2023-11-28T18:32:48.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faba40668d0 0x7faba4068d90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:48.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc0150840 0x7fabc01458f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:48.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc00c7600 0x7fabc014e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:48.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 --2- 172.21.15.7:0/1335189077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc00c1700 0x7fabc00cc5c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:48.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 >> 172.21.15.7:0/1335189077 conn(0x7fabc00cbf80 msgr2=0x7fabc00ca8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:48.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 shutdown_connections 2023-11-28T18:32:48.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:48.105+0000 7fabc7279640 1 -- 172.21.15.7:0/1335189077 wait complete. 2023-11-28T18:32:48.126 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:32:50.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:50.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b90fe640 1 Processor -- start 2023-11-28T18:32:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b90fe640 1 -- start start 2023-11-28T18:32:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b90fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b40ca4d0 0x7f85b415a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b90fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f85b40cf6c0 0x7f85b40cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b90fe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b415e380 con 0x7f85b41540e0 2023-11-28T18:32:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b90fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b415ade0 con 0x7f85b40cf6c0 2023-11-28T18:32:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b90fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b415d7e0 con 0x7f85b40ca4d0 2023-11-28T18:32:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b2d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f85b41540e0 0x7f85b415dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53554/0 (socket says 172.21.15.7:53554) 2023-11-28T18:32:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b2d9d640 1 -- 172.21.15.7:0/3643974975 learned_addr learned my addr 172.21.15.7:0/3643974975 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:32:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936019740 0 0) 0x7f85a4001120 con 0x7f85b41540e0 2023-11-28T18:32:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a0003530 con 0x7f85b41540e0 2023-11-28T18:32:50.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85abfff640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b40ca4d0 0x7f85b415a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b359e640 1 --2- 172.21.15.7:0/3643974975 >> v2:172.21.15.193:3300/0 conn(0x7f85b40cf6c0 0x7f85b40cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1303836817 0 0) 0x7f85a0003530 con 0x7f85b41540e0 2023-11-28T18:32:50.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b40ca4d0 msgr2=0x7f85b415a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:50.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b40ca4d0 0x7f85b415a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 >> v2:172.21.15.193:3300/0 conn(0x7f85b40cf6c0 msgr2=0x7f85b40cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:50.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 --2- 172.21.15.7:0/3643974975 >> v2:172.21.15.193:3300/0 conn(0x7f85b40cf6c0 0x7f85b40cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b40ca8f0 con 0x7f85b41540e0 2023-11-28T18:32:50.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f85a4001810 con 0x7f85b41540e0 2023-11-28T18:32:50.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 >> v1:172.21.15.7:6789/0 conn(0x7f85b41540e0 legacy=0x7f85b415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:50.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.281+0000 7f85b259c640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b40ca4d0 0x7f85a0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b259c640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0003fb0 0x7f85a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b259c640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0004d10 0x7f85a0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b415ade0 con 0x7f85a0004d10 2023-11-28T18:32:50.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b415d7e0 con 0x7f85a0003fb0 2023-11-28T18:32:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a000ba90 con 0x7f85b40ca4d0 2023-11-28T18:32:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b359e640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0004d10 0x7f85a0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b2d9d640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b40ca4d0 0x7f85a0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85abfff640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0003fb0 0x7f85a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b359e640 1 -- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b40ca4d0 msgr2=0x7f85a0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b359e640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b40ca4d0 0x7f85a0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b359e640 1 -- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0003fb0 msgr2=0x7f85a00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:50.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b359e640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0003fb0 0x7f85a00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b359e640 1 -- 172.21.15.7:0/3643974975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a0003530 con 0x7f85a0004d10 2023-11-28T18:32:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b359e640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0004d10 0x7f85a0005190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f85ac007360 tx=0x7f85ac01ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5315d240aee723f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0a5d60 con 0x7f85a0004d10 2023-11-28T18:32:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac0ad440 con 0x7f85a0004d10 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.285+0000 7f85b259c640 1 -- 172.21.15.7:0/3643974975 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0ad7d0 con 0x7f85a0004d10 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- 172.21.15.7:0/3643974975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0004d10 msgr2=0x7f85a0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 --2- 172.21.15.7:0/3643974975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0004d10 0x7f85a0005190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f85ac007360 tx=0x7f85ac01ce60 comp rx=0 tx=0).stop 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b359e640 1 -- 172.21.15.7:0/3643974975 reap_dead start 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- 172.21.15.7:0/3643974975 shutdown_connections 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- 172.21.15.7:0/3643974975 >> 172.21.15.7:0/3643974975 conn(0x7f85b40662e0 msgr2=0x7f85b40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- 172.21.15.7:0/3643974975 shutdown_connections 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- 172.21.15.7:0/3643974975 wait complete. 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 Processor -- start 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- start start 2023-11-28T18:32:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0004d10 0x7f85b4148010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85b4149c20 0x7f85b40b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b415cd90 0x7f85b415a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b4158750 con 0x7f85b4149c20 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b415a020 con 0x7f85a0004d10 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b90fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b4151410 con 0x7f85b415cd90 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85abfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85b4149c20 0x7f85b40b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85abfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85b4149c20 0x7f85b40b4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56984/0 (socket says 172.21.15.7:56984) 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85abfff640 1 -- 172.21.15.7:0/1933718270 learned_addr learned my addr 172.21.15.7:0/1933718270 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b2d9d640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0004d10 0x7f85b4148010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.289+0000 7f85b359e640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b415cd90 0x7f85b415a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85abfff640 1 -- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b415cd90 msgr2=0x7f85b415a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85abfff640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b415cd90 0x7f85b415a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85abfff640 1 -- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0004d10 msgr2=0x7f85b4148010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85abfff640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0004d10 0x7f85b4148010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85abfff640 1 -- 172.21.15.7:0/1933718270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b4147500 con 0x7f85b4149c20 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85b2d9d640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0004d10 0x7f85b4148010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85abfff640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85b4149c20 0x7f85b40b4e00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f859c00b820 tx=0x7f859c00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85ab7fe640 1 -- 172.21.15.7:0/1933718270 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c0040f0 con 0x7f85b4149c20 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85ab7fe640 1 -- 172.21.15.7:0/1933718270 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c0107d0 con 0x7f85b4149c20 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b41477a0 con 0x7f85b4149c20 2023-11-28T18:32:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85ab7fe640 1 -- 172.21.15.7:0/1933718270 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c010b60 con 0x7f85b4149c20 2023-11-28T18:32:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b4159960 con 0x7f85b4149c20 2023-11-28T18:32:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b404e890 con 0x7f85b4149c20 2023-11-28T18:32:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85ab7fe640 1 -- 172.21.15.7:0/1933718270 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f859c010cc0 con 0x7f85b4149c20 2023-11-28T18:32:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85ab7fe640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f85940668d0 0x7f8594068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85ab7fe640 1 -- 172.21.15.7:0/1933718270 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f859c084a80 con 0x7f85b4149c20 2023-11-28T18:32:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.293+0000 7f85b2d9d640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f85940668d0 0x7f8594068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.297+0000 7f85b2d9d640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f85940668d0 0x7f8594068d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f85a4001280 tx=0x7f85a4007a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.297+0000 7f85ab7fe640 1 -- 172.21.15.7:0/1933718270 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f859c051b00 con 0x7f85b4149c20 2023-11-28T18:32:50.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85b415de80 con 0x7f85b4149c20 2023-11-28T18:32:50.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85ab7fe640 1 -- 172.21.15.7:0/1933718270 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f859c022090 con 0x7f85b4149c20 2023-11-28T18:32:50.490 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:50.490 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:31:08.379745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.7:6838/2690556675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":2690556675},{"type":"v1","addr":"172.21.15.7:6838","nonce":2690556675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:50.491 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 47 2023-11-28T18:32:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f85940668d0 msgr2=0x7f8594068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f85940668d0 0x7f8594068d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f85a4001280 tx=0x7f85a4007a80 comp rx=0 tx=0).stop 2023-11-28T18:32:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85b4149c20 msgr2=0x7f85b40b4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85b4149c20 0x7f85b40b4e00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f859c00b820 tx=0x7f859c00bc20 comp rx=0 tx=0).stop 2023-11-28T18:32:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 shutdown_connections 2023-11-28T18:32:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f85940668d0 0x7f8594068d90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85b415cd90 0x7f85b415a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85b4149c20 0x7f85b40b4e00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 --2- 172.21.15.7:0/1933718270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0004d10 0x7f85b4148010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:50.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 >> 172.21.15.7:0/1933718270 conn(0x7f85b40662e0 msgr2=0x7f85b4071e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:50.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 shutdown_connections 2023-11-28T18:32:50.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:50.489+0000 7f85b90fe640 1 -- 172.21.15.7:0/1933718270 wait complete. 2023-11-28T18:32:50.511 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:32:52.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.653+0000 7fb0e813e640 1 Processor -- start 2023-11-28T18:32:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.653+0000 7fb0e813e640 1 -- start start 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e813e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e0151cd0 0x7fb0e015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e813e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0e006e560 0x7fb0e0157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e813e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fb0e00701f0 0x7fb0e00c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e813e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e0071e60 con 0x7fb0e006e560 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e813e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e0070930 con 0x7fb0e00701f0 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e813e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e0070660 con 0x7fb0e0151cd0 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e56d9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0e006e560 0x7fb0e0157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e56d9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0e006e560 0x7fb0e0157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56990/0 (socket says 172.21.15.7:56990) 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e56d9640 1 -- 172.21.15.7:0/3277680432 learned_addr learned my addr 172.21.15.7:0/3277680432 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e5eda640 1 --2- 172.21.15.7:0/3277680432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e0151cd0 0x7fb0e015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e66db640 1 --2- 172.21.15.7:0/3277680432 >> v2:172.21.15.193:3300/0 conn(0x7fb0e00701f0 0x7fb0e00c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e56d9640 1 -- 172.21.15.7:0/3277680432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e0151cd0 msgr2=0x7fb0e015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e56d9640 1 --2- 172.21.15.7:0/3277680432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e0151cd0 0x7fb0e015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e56d9640 1 -- 172.21.15.7:0/3277680432 >> v2:172.21.15.193:3300/0 conn(0x7fb0e00701f0 msgr2=0x7fb0e00c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e56d9640 1 --2- 172.21.15.7:0/3277680432 >> v2:172.21.15.193:3300/0 conn(0x7fb0e00701f0 0x7fb0e00c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.657+0000 7fb0e56d9640 1 -- 172.21.15.7:0/3277680432 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0070bf0 con 0x7fb0e006e560 2023-11-28T18:32:52.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e56d9640 1 --2- 172.21.15.7:0/3277680432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0157b90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb0e00c95f0 tx=0x7fb0d001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42ea7d8f5645c55 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e4ed8640 1 -- 172.21.15.7:0/3277680432 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00ab600 con 0x7fb0e006e560 2023-11-28T18:32:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e4ed8640 1 -- 172.21.15.7:0/3277680432 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d00a7050 con 0x7fb0e006e560 2023-11-28T18:32:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e4ed8640 1 -- 172.21.15.7:0/3277680432 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00ab980 con 0x7fb0e006e560 2023-11-28T18:32:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- 172.21.15.7:0/3277680432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 msgr2=0x7fb0e0157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 --2- 172.21.15.7:0/3277680432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0157b90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb0e00c95f0 tx=0x7fb0d001c470 comp rx=0 tx=0).stop 2023-11-28T18:32:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- 172.21.15.7:0/3277680432 shutdown_connections 2023-11-28T18:32:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 --2- 172.21.15.7:0/3277680432 >> v2:172.21.15.193:3300/0 conn(0x7fb0e00701f0 0x7fb0e00c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 --2- 172.21.15.7:0/3277680432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0157b90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 --2- 172.21.15.7:0/3277680432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e0151cd0 0x7fb0e015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- 172.21.15.7:0/3277680432 >> 172.21.15.7:0/3277680432 conn(0x7fb0e00662e0 msgr2=0x7fb0e00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- 172.21.15.7:0/3277680432 shutdown_connections 2023-11-28T18:32:52.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- 172.21.15.7:0/3277680432 wait complete. 2023-11-28T18:32:52.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 Processor -- start 2023-11-28T18:32:52.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- start start 2023-11-28T18:32:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e00701f0 0x7fb0e01f4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0e0151cd0 0x7fb0e01528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e01553d0 con 0x7fb0e006e560 2023-11-28T18:32:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00c67c0 con 0x7fb0e0151cd0 2023-11-28T18:32:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.661+0000 7fb0e813e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e0154e80 con 0x7fb0e00701f0 2023-11-28T18:32:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57004/0 (socket says 172.21.15.7:57004) 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 -- 172.21.15.7:0/1158859202 learned_addr learned my addr 172.21.15.7:0/1158859202 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e56d9640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e00701f0 0x7fb0e01f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e66db640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0e0151cd0 0x7fb0e01528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 -- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e00701f0 msgr2=0x7fb0e01f4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e00701f0 0x7fb0e01f4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 -- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0e0151cd0 msgr2=0x7fb0e01528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0e0151cd0 0x7fb0e01528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 -- 172.21.15.7:0/1158859202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d00094d0 con 0x7fb0e006e560 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e66db640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0e0151cd0 0x7fb0e01528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e5eda640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0150ff0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00d620 tx=0x7fb0dc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0d67fc640 1 -- 172.21.15.7:0/1158859202 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc00b620 con 0x7fb0e006e560 2023-11-28T18:32:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0d67fc640 1 -- 172.21.15.7:0/1158859202 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0dc00b780 con 0x7fb0e006e560 2023-11-28T18:32:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e0152df0 con 0x7fb0e006e560 2023-11-28T18:32:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0d67fc640 1 -- 172.21.15.7:0/1158859202 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc00bb10 con 0x7fb0e006e560 2023-11-28T18:32:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e01f0a80 con 0x7fb0e006e560 2023-11-28T18:32:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e01f08e0 con 0x7fb0e006e560 2023-11-28T18:32:52.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0d67fc640 1 -- 172.21.15.7:0/1158859202 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb0dc00bc70 con 0x7fb0e006e560 2023-11-28T18:32:52.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.665+0000 7fb0d67fc640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb0c40668d0 0x7fb0c4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:52.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.669+0000 7fb0d67fc640 1 -- 172.21.15.7:0/1158859202 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fb0dc084b40 con 0x7fb0e006e560 2023-11-28T18:32:52.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.669+0000 7fb0e56d9640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb0c40668d0 0x7fb0c4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:52.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.669+0000 7fb0e56d9640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb0c40668d0 0x7fb0c4068d90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00ad000 tx=0x7fb0d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:52.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.669+0000 7fb0d67fc640 1 -- 172.21.15.7:0/1158859202 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb0e01f08e0 con 0x7fb0e006e560 2023-11-28T18:32:52.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0e01f08e0 con 0x7fb0e006e560 2023-11-28T18:32:52.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0d67fc640 1 -- 172.21.15.7:0/1158859202 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5765 (secure 0 0 0) 0x7fb0e01f08e0 con 0x7fb0e006e560 2023-11-28T18:32:52.863 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:52.863 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:32:52.248633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:52.863 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 48 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb0c40668d0 msgr2=0x7fb0c4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb0c40668d0 0x7fb0c4068d90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00ad000 tx=0x7fb0d0008040 comp rx=0 tx=0).stop 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 msgr2=0x7fb0e0150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0150ff0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00d620 tx=0x7fb0dc00dae0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 shutdown_connections 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb0c40668d0 0x7fb0c4068d90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0e0151cd0 0x7fb0e01528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb0e00701f0 0x7fb0e01f4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 --2- 172.21.15.7:0/1158859202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e006e560 0x7fb0e0150ff0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:52.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.861+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 >> 172.21.15.7:0/1158859202 conn(0x7fb0e00662e0 msgr2=0x7fb0e014b180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:52.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.865+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 shutdown_connections 2023-11-28T18:32:52.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:52.865+0000 7fb0e813e640 1 -- 172.21.15.7:0/1158859202 wait complete. 2023-11-28T18:32:52.884 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-11-28T18:32:52.884 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:32:52.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.025+0000 7fa19a30f640 1 Processor -- start 2023-11-28T18:32:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.025+0000 7fa19a30f640 1 -- start start 2023-11-28T18:32:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa19a30f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa19415bd00 0x7fa194157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa19a30f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19415c110 con 0x7fa194151cd0 2023-11-28T18:32:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa19a30f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa194071e60 con 0x7fa19406e560 2023-11-28T18:32:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa19a30f640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa194070930 con 0x7fa19415bd00 2023-11-28T18:32:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa193fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa194151cd0 0x7fa1940c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53568/0 (socket says 172.21.15.7:53568) 2023-11-28T18:32:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa193fff640 1 -- 172.21.15.7:0/746298791 learned_addr learned my addr 172.21.15.7:0/746298791 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1988ac640 1 --2- 172.21.15.7:0/746298791 >> v2:172.21.15.193:3300/0 conn(0x7fa19415bd00 0x7fa194157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062268986 0 0) 0x7fa19415c110 con 0x7fa194151cd0 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c003530 con 0x7fa194151cd0 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548302757 0 0) 0x7fa17c003530 con 0x7fa194151cd0 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 >> v2:172.21.15.193:3300/0 conn(0x7fa19415bd00 msgr2=0x7fa194157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 --2- 172.21.15.7:0/746298791 >> v2:172.21.15.193:3300/0 conn(0x7fa19415bd00 0x7fa194157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 >> v1:172.21.15.193:6789/0 conn(0x7fa19406e560 legacy=0x7fa194160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa194070bf0 con 0x7fa194151cd0 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fa184001680 con 0x7fa194151cd0 2023-11-28T18:32:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 >> v1:172.21.15.7:6789/0 conn(0x7fa194151cd0 legacy=0x7fa1940c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0039d0 0x7fa17c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa17c0047d0 0x7fa17c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa17c005190 0x7fa17c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa194070930 con 0x7fa17c0047d0 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c00c090 con 0x7fa17c0039d0 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c00c250 con 0x7fa17c005190 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1937fe640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa17c0047d0 0x7fa17c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1988ac640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa17c005190 0x7fa17c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa193fff640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0039d0 0x7fa17c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1937fe640 1 -- 172.21.15.7:0/746298791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa17c005190 msgr2=0x7fa17c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1937fe640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa17c005190 0x7fa17c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1937fe640 1 -- 172.21.15.7:0/746298791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0039d0 msgr2=0x7fa17c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1937fe640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0039d0 0x7fa17c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1937fe640 1 -- 172.21.15.7:0/746298791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c003530 con 0x7fa17c0047d0 2023-11-28T18:32:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.029+0000 7fa1988ac640 1 -- 172.21.15.7:0/746298791 reap_dead start 2023-11-28T18:32:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa1937fe640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa17c0047d0 0x7fa17c004c50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa188001780 tx=0x7fa18801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3a9eea9247b4c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1880052d0 con 0x7fa17c0047d0 2023-11-28T18:32:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1880a7050 con 0x7fa17c0047d0 2023-11-28T18:32:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa192ffd640 1 -- 172.21.15.7:0/746298791 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188003600 con 0x7fa17c0047d0 2023-11-28T18:32:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa19a30f640 1 -- 172.21.15.7:0/746298791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa17c0047d0 msgr2=0x7fa17c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa19a30f640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa17c0047d0 0x7fa17c004c50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa188001780 tx=0x7fa18801c3e0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa19a30f640 1 -- 172.21.15.7:0/746298791 shutdown_connections 2023-11-28T18:32:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa19a30f640 1 --2- 172.21.15.7:0/746298791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa17c0047d0 0x7fa17c004c50 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa19a30f640 1 -- 172.21.15.7:0/746298791 >> 172.21.15.7:0/746298791 conn(0x7fa1940662e0 msgr2=0x7fa1940666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa19a30f640 1 -- 172.21.15.7:0/746298791 shutdown_connections 2023-11-28T18:32:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa19a30f640 1 -- 172.21.15.7:0/746298791 wait complete. 2023-11-28T18:32:53.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.033+0000 7fa19a30f640 1 Processor -- start 2023-11-28T18:32:53.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 -- start start 2023-11-28T18:32:53.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0047d0 0x7fa194147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:53.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa19414d2b0 0x7fa19414d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:53.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19414e1e0 0x7fa19414e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:53.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1941f3930 con 0x7fa19414e1e0 2023-11-28T18:32:53.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa194147740 con 0x7fa17c0047d0 2023-11-28T18:32:53.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19414c120 con 0x7fa19414d2b0 2023-11-28T18:32:53.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19414e1e0 0x7fa19414e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:53.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19414e1e0 0x7fa19414e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57026/0 (socket says 172.21.15.7:57026) 2023-11-28T18:32:53.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 -- 172.21.15.7:0/4018512162 learned_addr learned my addr 172.21.15.7:0/4018512162 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:53.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1937fe640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa19414d2b0 0x7fa19414d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:53.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa193fff640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0047d0 0x7fa194147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:53.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 -- 172.21.15.7:0/4018512162 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa19414d2b0 msgr2=0x7fa19414d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:53.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa19414d2b0 0x7fa19414d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 -- 172.21.15.7:0/4018512162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0047d0 msgr2=0x7fa194147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:53.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0047d0 0x7fa194147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 -- 172.21.15.7:0/4018512162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19414c400 con 0x7fa19414e1e0 2023-11-28T18:32:53.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa1988ac640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19414e1e0 0x7fa19414e660 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa19406e510 tx=0x7fa18c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:53.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa190ff9640 1 -- 172.21.15.7:0/4018512162 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c00e070 con 0x7fa19414e1e0 2023-11-28T18:32:53.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa190ff9640 1 -- 172.21.15.7:0/4018512162 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c011db0 con 0x7fa19414e1e0 2023-11-28T18:32:53.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19414c6e0 con 0x7fa19414e1e0 2023-11-28T18:32:53.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa190ff9640 1 -- 172.21.15.7:0/4018512162 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0039a0 con 0x7fa19414e1e0 2023-11-28T18:32:53.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19414f730 con 0x7fa19414e1e0 2023-11-28T18:32:53.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.037+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1941f3aa0 con 0x7fa19414e1e0 2023-11-28T18:32:53.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.041+0000 7fa190ff9640 1 -- 172.21.15.7:0/4018512162 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa18c003b00 con 0x7fa19414e1e0 2023-11-28T18:32:53.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.041+0000 7fa190ff9640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa180066900 0x7fa180068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:53.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.041+0000 7fa190ff9640 1 -- 172.21.15.7:0/4018512162 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fa18c084d30 con 0x7fa19414e1e0 2023-11-28T18:32:53.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.041+0000 7fa193fff640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa180066900 0x7fa180068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:53.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.041+0000 7fa193fff640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa180066900 0x7fa180068dc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa1840055c0 tx=0x7fa184001570 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:53.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.045+0000 7fa190ff9640 1 -- 172.21.15.7:0/4018512162 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa1941f3aa0 con 0x7fa19414e1e0 2023-11-28T18:32:53.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1941f3a70 con 0x7fa19414e1e0 2023-11-28T18:32:53.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa190ff9640 1 -- 172.21.15.7:0/4018512162 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5765 (secure 0 0 0) 0x7fa18c051df0 con 0x7fa19414e1e0 2023-11-28T18:32:53.242 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:53.242 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:32:52.248633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:53.243 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 48 2023-11-28T18:32:53.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa180066900 msgr2=0x7fa180068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:53.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa180066900 0x7fa180068dc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa1840055c0 tx=0x7fa184001570 comp rx=0 tx=0).stop 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19414e1e0 msgr2=0x7fa19414e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19414e1e0 0x7fa19414e660 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa19406e510 tx=0x7fa18c0115a0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 shutdown_connections 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa180066900 0x7fa180068dc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19414e1e0 0x7fa19414e660 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa19414d2b0 0x7fa19414d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 --2- 172.21.15.7:0/4018512162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa17c0047d0 0x7fa194147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 >> 172.21.15.7:0/4018512162 conn(0x7fa1940662e0 msgr2=0x7fa1940728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 shutdown_connections 2023-11-28T18:32:53.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:53.241+0000 7fa19a30f640 1 -- 172.21.15.7:0/4018512162 wait complete. 2023-11-28T18:32:53.266 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:32:52.248633+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 5359, "mds_2": 5307, "mds_3": 5086, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:32:55.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:55.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.421+0000 7f84d357d640 1 Processor -- start 2023-11-28T18:32:55.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.421+0000 7f84d357d640 1 -- start start 2023-11-28T18:32:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d357d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f84cc15ce10 0x7f84cc0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d357d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc0cafe0 0x7f84cc0ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d357d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84cc15a3a0 con 0x7f84cc15ca30 2023-11-28T18:32:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d357d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84cc15a620 con 0x7f84cc15ce10 2023-11-28T18:32:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d357d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc15a900 con 0x7f84cc0cafe0 2023-11-28T18:32:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d1d7a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f84cc15ce10 0x7f84cc0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:55.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d1d7a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f84cc15ce10 0x7f84cc0ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:51570/0 (socket says 172.21.15.7:51570) 2023-11-28T18:32:55.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d1d7a640 1 -- 172.21.15.7:0/3296162655 learned_addr learned my addr 172.21.15.7:0/3296162655 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:55.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d2d7c640 1 --2- 172.21.15.7:0/3296162655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc0cafe0 0x7f84cc0ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d1579640 1 -- 172.21.15.7:0/3296162655 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126368812 0 0) 0x7f84cc15a3a0 con 0x7f84cc15ca30 2023-11-28T18:32:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d1579640 1 -- 172.21.15.7:0/3296162655 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b8003530 con 0x7f84cc15ca30 2023-11-28T18:32:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d2d7c640 1 -- 172.21.15.7:0/3296162655 >> v2:172.21.15.193:3300/0 conn(0x7f84cc15ce10 msgr2=0x7f84cc0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d2d7c640 1 --2- 172.21.15.7:0/3296162655 >> v2:172.21.15.193:3300/0 conn(0x7f84cc15ce10 0x7f84cc0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d2d7c640 1 -- 172.21.15.7:0/3296162655 >> v1:172.21.15.193:6789/0 conn(0x7f84cc15ca30 legacy=0x7f84cc159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d2d7c640 1 -- 172.21.15.7:0/3296162655 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc15abe0 con 0x7f84cc0cafe0 2023-11-28T18:32:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.425+0000 7f84d1d7a640 1 --2- 172.21.15.7:0/3296162655 >> v2:172.21.15.193:3300/0 conn(0x7f84cc15ce10 0x7f84cc0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:32:55.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d2d7c640 1 --2- 172.21.15.7:0/3296162655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc0cafe0 0x7f84cc0ca710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009d90 tx=0x7f84c401f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e309f29ab808eb2e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:55.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d1579640 1 -- 172.21.15.7:0/3296162655 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40af600 con 0x7f84cc0cafe0 2023-11-28T18:32:55.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d1579640 1 -- 172.21.15.7:0/3296162655 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c40ab050 con 0x7f84cc0cafe0 2023-11-28T18:32:55.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d1579640 1 -- 172.21.15.7:0/3296162655 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40af980 con 0x7f84cc0cafe0 2023-11-28T18:32:55.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d357d640 1 -- 172.21.15.7:0/3296162655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc0cafe0 msgr2=0x7f84cc0ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:55.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d357d640 1 --2- 172.21.15.7:0/3296162655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc0cafe0 0x7f84cc0ca710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009d90 tx=0x7f84c401f130 comp rx=0 tx=0).stop 2023-11-28T18:32:55.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d357d640 1 -- 172.21.15.7:0/3296162655 shutdown_connections 2023-11-28T18:32:55.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d357d640 1 --2- 172.21.15.7:0/3296162655 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc0cafe0 0x7f84cc0ca710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d357d640 1 --2- 172.21.15.7:0/3296162655 >> v2:172.21.15.193:3300/0 conn(0x7f84cc15ce10 0x7f84cc0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d357d640 1 -- 172.21.15.7:0/3296162655 >> 172.21.15.7:0/3296162655 conn(0x7f84cc06c060 msgr2=0x7f84cc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:55.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d357d640 1 -- 172.21.15.7:0/3296162655 shutdown_connections 2023-11-28T18:32:55.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.429+0000 7f84d357d640 1 -- 172.21.15.7:0/3296162655 wait complete. 2023-11-28T18:32:55.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 Processor -- start 2023-11-28T18:32:55.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 -- start start 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84cc0cafe0 0x7f84cc1501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc15ca30 0x7f84cc0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84cc15ce10 0x7f84cc151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc1576b0 con 0x7f84cc15ce10 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0c3630 con 0x7f84cc0cafe0 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc1f2c40 con 0x7f84cc15ca30 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84cc15ce10 0x7f84cc151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84cc15ce10 0x7f84cc151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38008/0 (socket says 172.21.15.7:38008) 2023-11-28T18:32:55.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 -- 172.21.15.7:0/1346155723 learned_addr learned my addr 172.21.15.7:0/1346155723 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d1d7a640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc15ca30 0x7f84cc0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d257b640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84cc0cafe0 0x7f84cc1501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 -- 172.21.15.7:0/1346155723 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc15ca30 msgr2=0x7f84cc0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc15ca30 0x7f84cc0cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 -- 172.21.15.7:0/1346155723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84cc0cafe0 msgr2=0x7f84cc1501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84cc0cafe0 0x7f84cc1501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 -- 172.21.15.7:0/1346155723 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0094d0 con 0x7f84cc15ce10 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d2d7c640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84cc15ce10 0x7f84cc151330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f84c40b2040 tx=0x7f84c401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84c2ffd640 1 -- 172.21.15.7:0/1346155723 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4009450 con 0x7f84cc15ce10 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84c2ffd640 1 -- 172.21.15.7:0/1346155723 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c40b9410 con 0x7f84cc15ce10 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84c2ffd640 1 -- 172.21.15.7:0/1346155723 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40b97c0 con 0x7f84cc15ce10 2023-11-28T18:32:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c4009a70 con 0x7f84cc15ce10 2023-11-28T18:32:55.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.433+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84cc1f37c0 con 0x7f84cc15ce10 2023-11-28T18:32:55.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.437+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84cc1f3950 con 0x7f84cc15ce10 2023-11-28T18:32:55.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.437+0000 7f84c2ffd640 1 -- 172.21.15.7:0/1346155723 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f84c40b6050 con 0x7f84cc15ce10 2023-11-28T18:32:55.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.437+0000 7f84c2ffd640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84b00668d0 0x7f84b0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:55.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.437+0000 7f84c2ffd640 1 -- 172.21.15.7:0/1346155723 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f84c4122930 con 0x7f84cc15ce10 2023-11-28T18:32:55.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.437+0000 7f84d257b640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84b00668d0 0x7f84b0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:55.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.437+0000 7f84d257b640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84b00668d0 0x7f84b0068d90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f84c80011f0 tx=0x7f84c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:55.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.441+0000 7f84c2ffd640 1 -- 172.21.15.7:0/1346155723 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f84c40ef960 con 0x7f84cc15ce10 2023-11-28T18:32:55.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84cc04e890 con 0x7f84cc15ce10 2023-11-28T18:32:55.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:55.639 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:32:52.248633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:55.639 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 48 2023-11-28T18:32:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84c2ffd640 1 -- 172.21.15.7:0/1346155723 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f84c40b4090 con 0x7f84cc15ce10 2023-11-28T18:32:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84b00668d0 msgr2=0x7f84b0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84b00668d0 0x7f84b0068d90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f84c80011f0 tx=0x7f84c8009040 comp rx=0 tx=0).stop 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84cc15ce10 msgr2=0x7f84cc151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84cc15ce10 0x7f84cc151330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f84c40b2040 tx=0x7f84c401bed0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 shutdown_connections 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84b00668d0 0x7f84b0068d90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84cc15ce10 0x7f84cc151330 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f84cc15ca30 0x7f84cc0cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 --2- 172.21.15.7:0/1346155723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84cc0cafe0 0x7f84cc1501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 >> 172.21.15.7:0/1346155723 conn(0x7f84cc06c060 msgr2=0x7f84cc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 shutdown_connections 2023-11-28T18:32:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:55.637+0000 7f84d357d640 1 -- 172.21.15.7:0/1346155723 wait complete. 2023-11-28T18:32:55.765 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:32:57.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:32:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4759e640 1 Processor -- start 2023-11-28T18:32:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4759e640 1 -- start start 2023-11-28T18:32:57.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4759e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:57.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4759e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2c480cafe0 0x7f2c480cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:57.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4759e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4815a620 con 0x7f2c4815ce10 2023-11-28T18:32:57.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4759e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4815a900 con 0x7f2c480cafe0 2023-11-28T18:32:57.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4759e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4815abe0 con 0x7f2c4815ca30 2023-11-28T18:32:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c46d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2c480cafe0 0x7f2c480cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c45d9b640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f2c4815ce10 0x7f2c480ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:54720/0 (socket says 172.21.15.7:54720) 2023-11-28T18:32:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:38368/0 (socket says 172.21.15.7:38368) 2023-11-28T18:32:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c46d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2c480cafe0 0x7f2c480cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:51586/0 (socket says 172.21.15.7:51586) 2023-11-28T18:32:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c45d9b640 1 -- 172.21.15.7:0/3551376341 learned_addr learned my addr 172.21.15.7:0/3551376341 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:32:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c46d9d640 1 -- 172.21.15.7:0/3551376341 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 msgr2=0x7f2c48159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:32:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c46d9d640 1 --2- 172.21.15.7:0/3551376341 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c46d9d640 1 -- 172.21.15.7:0/3551376341 >> v1:172.21.15.193:6789/0 conn(0x7f2c4815ce10 legacy=0x7f2c480ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c46d9d640 1 -- 172.21.15.7:0/3551376341 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c480cbb10 con 0x7f2c480cafe0 2023-11-28T18:32:57.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4559a640 1 -- 172.21.15.7:0/3551376341 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955876515 0 0) 0x7f2c4815a620 con 0x7f2c4815ce10 2023-11-28T18:32:57.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.921+0000 7f2c4659c640 1 --2- 172.21.15.7:0/3551376341 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:32:57.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c46d9d640 1 --2- 172.21.15.7:0/3551376341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c480cafe0 0x7f2c480cc5f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009d90 tx=0x7f2c4001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5699fd2dd54a7cd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:57.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4559a640 1 -- 172.21.15.7:0/3551376341 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c400af600 con 0x7f2c480cafe0 2023-11-28T18:32:57.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4559a640 1 -- 172.21.15.7:0/3551376341 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c400ab050 con 0x7f2c480cafe0 2023-11-28T18:32:57.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4559a640 1 -- 172.21.15.7:0/3551376341 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c400af980 con 0x7f2c480cafe0 2023-11-28T18:32:57.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4759e640 1 -- 172.21.15.7:0/3551376341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c480cafe0 msgr2=0x7f2c480cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:57.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4759e640 1 --2- 172.21.15.7:0/3551376341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c480cafe0 0x7f2c480cc5f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009d90 tx=0x7f2c4001f130 comp rx=0 tx=0).stop 2023-11-28T18:32:57.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4759e640 1 -- 172.21.15.7:0/3551376341 shutdown_connections 2023-11-28T18:32:57.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4759e640 1 --2- 172.21.15.7:0/3551376341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c480cafe0 0x7f2c480cc5f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:57.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4759e640 1 --2- 172.21.15.7:0/3551376341 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:57.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4759e640 1 -- 172.21.15.7:0/3551376341 >> 172.21.15.7:0/3551376341 conn(0x7f2c4806c060 msgr2=0x7f2c480662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:57.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4759e640 1 -- 172.21.15.7:0/3551376341 shutdown_connections 2023-11-28T18:32:57.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.925+0000 7f2c4759e640 1 -- 172.21.15.7:0/3551376341 wait complete. 2023-11-28T18:32:57.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 Processor -- start 2023-11-28T18:32:57.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 -- start start 2023-11-28T18:32:57.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c480cafe0 0x7f2c480c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:57.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:57.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c4815ce10 0x7f2c481564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:57.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c480c4810 con 0x7f2c480cafe0 2023-11-28T18:32:57.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c480c6060 con 0x7f2c4815ce10 2023-11-28T18:32:57.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c480c54e0 con 0x7f2c4815ca30 2023-11-28T18:32:57.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c480cafe0 0x7f2c480c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:57.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c480cafe0 0x7f2c480c2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38024/0 (socket says 172.21.15.7:38024) 2023-11-28T18:32:57.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 -- 172.21.15.7:0/2336217258 learned_addr learned my addr 172.21.15.7:0/2336217258 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:32:57.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c45d9b640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:57.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c46d9d640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c4815ce10 0x7f2c481564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 -- 172.21.15.7:0/2336217258 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 msgr2=0x7f2c48154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 -- 172.21.15.7:0/2336217258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c4815ce10 msgr2=0x7f2c481564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c4815ce10 0x7f2c481564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 -- 172.21.15.7:0/2336217258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38009420 con 0x7f2c480cafe0 2023-11-28T18:32:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4659c640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c480cafe0 0x7f2c480c2df0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800ebe0 tx=0x7f2c3800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c36ffd640 1 -- 172.21.15.7:0/2336217258 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3800eee0 con 0x7f2c480cafe0 2023-11-28T18:32:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c36ffd640 1 -- 172.21.15.7:0/2336217258 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3800cc30 con 0x7f2c480cafe0 2023-11-28T18:32:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c36ffd640 1 -- 172.21.15.7:0/2336217258 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38013470 con 0x7f2c480cafe0 2023-11-28T18:32:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c40009a70 con 0x7f2c480cafe0 2023-11-28T18:32:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.929+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c480c5aa0 con 0x7f2c480cafe0 2023-11-28T18:32:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.933+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4804e890 con 0x7f2c480cafe0 2023-11-28T18:32:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.933+0000 7f2c36ffd640 1 -- 172.21.15.7:0/2336217258 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2c380135d0 con 0x7f2c480cafe0 2023-11-28T18:32:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.933+0000 7f2c36ffd640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2c280668d0 0x7f2c28068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:32:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.933+0000 7f2c36ffd640 1 -- 172.21.15.7:0/2336217258 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f2c38084a40 con 0x7f2c480cafe0 2023-11-28T18:32:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.933+0000 7f2c45d9b640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2c280668d0 0x7f2c28068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:32:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.933+0000 7f2c45d9b640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2c280668d0 0x7f2c28068d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c480c9a90 tx=0x7f2c3c007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:32:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:57.937+0000 7f2c36ffd640 1 -- 172.21.15.7:0/2336217258 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2c380519e0 con 0x7f2c480cafe0 2023-11-28T18:32:58.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c4815c280 con 0x7f2c480cafe0 2023-11-28T18:32:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c36ffd640 1 -- 172.21.15.7:0/2336217258 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f2c38021090 con 0x7f2c480cafe0 2023-11-28T18:32:58.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:32:58.139 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:32:52.248633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:32:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 48 2023-11-28T18:32:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2c280668d0 msgr2=0x7f2c28068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2c280668d0 0x7f2c28068d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c480c9a90 tx=0x7f2c3c007df0 comp rx=0 tx=0).stop 2023-11-28T18:32:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c480cafe0 msgr2=0x7f2c480c2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:32:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c480cafe0 0x7f2c480c2df0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800ebe0 tx=0x7f2c3800c520 comp rx=0 tx=0).stop 2023-11-28T18:32:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 shutdown_connections 2023-11-28T18:32:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2c280668d0 0x7f2c28068d90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c4815ce10 0x7f2c481564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2c4815ca30 0x7f2c48154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 --2- 172.21.15.7:0/2336217258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c480cafe0 0x7f2c480c2df0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:32:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 >> 172.21.15.7:0/2336217258 conn(0x7f2c4806c060 msgr2=0x7f2c481595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:32:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 shutdown_connections 2023-11-28T18:32:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:32:58.137+0000 7f2c4759e640 1 -- 172.21.15.7:0/2336217258 wait complete. 2023-11-28T18:32:58.161 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:33:00.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:00.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f1235957640 1 Processor -- start 2023-11-28T18:33:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f1235957640 1 -- start start 2023-11-28T18:33:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f1235957640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3bc0 0x7f12300c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f1235957640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12300c3490 0x7f12300c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f1235957640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12300c3920 con 0x7f12301503c0 2023-11-28T18:33:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f1235957640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1230157240 con 0x7f12300c3490 2023-11-28T18:33:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f1235957640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12300c2ce0 con 0x7f12300c3bc0 2023-11-28T18:33:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f122effd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f12301503c0 0x7f123006d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42526/0 (socket says 172.21.15.7:42526) 2023-11-28T18:33:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f122effd640 1 -- 172.21.15.7:0/4051596673 learned_addr learned my addr 172.21.15.7:0/4051596673 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:33:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f122f7fe640 1 --2- 172.21.15.7:0/4051596673 >> v2:172.21.15.7:3300/0 conn(0x7f12300c3490 0x7f12300c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.305+0000 7f1227fff640 1 --2- 172.21.15.7:0/4051596673 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3bc0 0x7f12300c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122e7fc640 1 -- 172.21.15.7:0/4051596673 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899435646 0 0) 0x7f1220001120 con 0x7f12301503c0 2023-11-28T18:33:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122e7fc640 1 -- 172.21.15.7:0/4051596673 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f121c003530 con 0x7f12301503c0 2023-11-28T18:33:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122f7fe640 1 -- 172.21.15.7:0/4051596673 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3bc0 msgr2=0x7f12300c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122f7fe640 1 --2- 172.21.15.7:0/4051596673 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3bc0 0x7f12300c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122f7fe640 1 -- 172.21.15.7:0/4051596673 >> v1:172.21.15.7:6789/0 conn(0x7f12301503c0 legacy=0x7f123006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122f7fe640 1 -- 172.21.15.7:0/4051596673 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12300c2f80 con 0x7f12300c3490 2023-11-28T18:33:00.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122f7fe640 1 --2- 172.21.15.7:0/4051596673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3490 0x7f12300c46f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1228009d90 tx=0x7f122801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa92451ba7a7c89a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:00.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122e7fc640 1 -- 172.21.15.7:0/4051596673 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12280af600 con 0x7f12300c3490 2023-11-28T18:33:00.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122e7fc640 1 -- 172.21.15.7:0/4051596673 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12280ab050 con 0x7f12300c3490 2023-11-28T18:33:00.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f122e7fc640 1 -- 172.21.15.7:0/4051596673 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12280af980 con 0x7f12300c3490 2023-11-28T18:33:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f1235957640 1 -- 172.21.15.7:0/4051596673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3490 msgr2=0x7f12300c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.309+0000 7f1235957640 1 --2- 172.21.15.7:0/4051596673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3490 0x7f12300c46f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1228009d90 tx=0x7f122801f130 comp rx=0 tx=0).stop 2023-11-28T18:33:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 -- 172.21.15.7:0/4051596673 shutdown_connections 2023-11-28T18:33:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 --2- 172.21.15.7:0/4051596673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3490 0x7f12300c46f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 --2- 172.21.15.7:0/4051596673 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3bc0 0x7f12300c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 -- 172.21.15.7:0/4051596673 >> 172.21.15.7:0/4051596673 conn(0x7f12300662e0 msgr2=0x7f12300666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 -- 172.21.15.7:0/4051596673 shutdown_connections 2023-11-28T18:33:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 -- 172.21.15.7:0/4051596673 wait complete. 2023-11-28T18:33:00.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 Processor -- start 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 -- start start 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3490 0x7f123014ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3bc0 0x7f12300715d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12301503c0 0x7f12301eb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12301eb2a0 con 0x7f12300c3bc0 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123014d540 con 0x7f12301503c0 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1235957640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123014c270 con 0x7f12300c3490 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3bc0 0x7f12300715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3bc0 0x7f12300715d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38028/0 (socket says 172.21.15.7:38028) 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 -- 172.21.15.7:0/2565880332 learned_addr learned my addr 172.21.15.7:0/2565880332 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:00.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 -- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3490 msgr2=0x7f123014ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f122effd640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3490 0x7f123014ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f122f7fe640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12301503c0 0x7f12301eb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3490 0x7f123014ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 -- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12301503c0 msgr2=0x7f12301eb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12301503c0 0x7f12301eb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 -- 172.21.15.7:0/2565880332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1228009a70 con 0x7f12300c3bc0 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f122f7fe640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12301503c0 0x7f12301eb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f122effd640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3490 0x7f123014ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f1227fff640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3bc0 0x7f12300715d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f121800e910 tx=0x7f121800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.313+0000 7f122cff9640 1 -- 172.21.15.7:0/2565880332 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121800cc40 con 0x7f12300c3bc0 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f122cff9640 1 -- 172.21.15.7:0/2565880332 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121800cda0 con 0x7f12300c3bc0 2023-11-28T18:33:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f122cff9640 1 -- 172.21.15.7:0/2565880332 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218011610 con 0x7f12300c3bc0 2023-11-28T18:33:00.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123014c550 con 0x7f12300c3bc0 2023-11-28T18:33:00.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123014bf40 con 0x7f12300c3bc0 2023-11-28T18:33:00.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12301eb3e0 con 0x7f12300c3bc0 2023-11-28T18:33:00.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f122cff9640 1 -- 172.21.15.7:0/2565880332 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1218011770 con 0x7f12300c3bc0 2023-11-28T18:33:00.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f122cff9640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12100668d0 0x7f1210068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:00.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f122cff9640 1 -- 172.21.15.7:0/2565880332 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f1218084630 con 0x7f12300c3bc0 2023-11-28T18:33:00.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.317+0000 7f122effd640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12100668d0 0x7f1210068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:00.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.321+0000 7f122effd640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12100668d0 0x7f1210068d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1220007fd0 tx=0x7f1220009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:00.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.321+0000 7f122cff9640 1 -- 172.21.15.7:0/2565880332 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f12301eb3e0 con 0x7f12300c3bc0 2023-11-28T18:33:00.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12301eb3e0 con 0x7f12300c3bc0 2023-11-28T18:33:00.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f122cff9640 1 -- 172.21.15.7:0/2565880332 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5767 (secure 0 0 0) 0x7f12301eb3e0 con 0x7f12300c3bc0 2023-11-28T18:33:00.514 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:00.514 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:32:57.306053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:resolve","state_seq":27,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:00.515 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 49 2023-11-28T18:33:00.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12100668d0 msgr2=0x7f1210068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:00.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12100668d0 0x7f1210068d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1220007fd0 tx=0x7f1220009040 comp rx=0 tx=0).stop 2023-11-28T18:33:00.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3bc0 msgr2=0x7f12300715d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:00.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3bc0 0x7f12300715d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f121800e910 tx=0x7f121800edd0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 shutdown_connections 2023-11-28T18:33:00.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12100668d0 0x7f1210068d90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12301503c0 0x7f12301eb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12300c3bc0 0x7f12300715d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 --2- 172.21.15.7:0/2565880332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12300c3490 0x7f123014ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:00.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 >> 172.21.15.7:0/2565880332 conn(0x7f12300662e0 msgr2=0x7f1230147560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:00.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 shutdown_connections 2023-11-28T18:33:00.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:00.513+0000 7f1235957640 1 -- 172.21.15.7:0/2565880332 wait complete. 2023-11-28T18:33:00.527 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:33:00.532 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:33:00.537 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:33:02.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:02.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff26525a640 1 Processor -- start 2023-11-28T18:33:02.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff26525a640 1 -- start start 2023-11-28T18:33:02.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff26525a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2601542e0 0x7ff26015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:02.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff26525a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 0x7ff2600cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:02.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff26525a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff2600cc810 0x7ff260072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:02.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff26525a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff260071d50 con 0x7ff2601542e0 2023-11-28T18:33:02.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff26525a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff260072030 con 0x7ff2600cc810 2023-11-28T18:33:02.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff26525a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2600ca920 con 0x7ff2600cbb70 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25ed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2601542e0 0x7ff26015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25ed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2601542e0 0x7ff26015e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38042/0 (socket says 172.21.15.7:38042) 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25ed9d640 1 -- 172.21.15.7:0/1963508125 learned_addr learned my addr 172.21.15.7:0/1963508125 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25f59e640 1 --2- 172.21.15.7:0/1963508125 >> v2:172.21.15.193:3300/0 conn(0x7ff2600cc810 0x7ff260072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff257fff640 1 --2- 172.21.15.7:0/1963508125 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 0x7ff2600cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25ed9d640 1 -- 172.21.15.7:0/1963508125 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 msgr2=0x7ff2600cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25ed9d640 1 --2- 172.21.15.7:0/1963508125 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 0x7ff2600cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25ed9d640 1 -- 172.21.15.7:0/1963508125 >> v2:172.21.15.193:3300/0 conn(0x7ff2600cc810 msgr2=0x7ff260072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25ed9d640 1 --2- 172.21.15.7:0/1963508125 >> v2:172.21.15.193:3300/0 conn(0x7ff2600cc810 0x7ff260072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.693+0000 7ff25ed9d640 1 -- 172.21.15.7:0/1963508125 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26015e550 con 0x7ff2601542e0 2023-11-28T18:33:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff25ed9d640 1 --2- 172.21.15.7:0/1963508125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 0x7ff26015e010 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff260066fc0 tx=0x7ff25001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7827edcd2a68c4f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff25e59c640 1 -- 172.21.15.7:0/1963508125 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2500ac630 con 0x7ff2601542e0 2023-11-28T18:33:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff25e59c640 1 -- 172.21.15.7:0/1963508125 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2500ac790 con 0x7ff2601542e0 2023-11-28T18:33:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff25e59c640 1 -- 172.21.15.7:0/1963508125 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2500acb20 con 0x7ff2601542e0 2023-11-28T18:33:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 -- 172.21.15.7:0/1963508125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 msgr2=0x7ff26015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:02.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 --2- 172.21.15.7:0/1963508125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 0x7ff26015e010 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff260066fc0 tx=0x7ff25001c500 comp rx=0 tx=0).stop 2023-11-28T18:33:02.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 -- 172.21.15.7:0/1963508125 shutdown_connections 2023-11-28T18:33:02.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 --2- 172.21.15.7:0/1963508125 >> v2:172.21.15.193:3300/0 conn(0x7ff2600cc810 0x7ff260072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 --2- 172.21.15.7:0/1963508125 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 0x7ff2600cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 --2- 172.21.15.7:0/1963508125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 0x7ff26015e010 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 -- 172.21.15.7:0/1963508125 >> 172.21.15.7:0/1963508125 conn(0x7ff2600717e0 msgr2=0x7ff2600662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:02.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 -- 172.21.15.7:0/1963508125 shutdown_connections 2023-11-28T18:33:02.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.697+0000 7ff26525a640 1 -- 172.21.15.7:0/1963508125 wait complete. 2023-11-28T18:33:02.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff26525a640 1 Processor -- start 2023-11-28T18:33:02.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff26525a640 1 -- start start 2023-11-28T18:33:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff26525a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 0x7ff260159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff26525a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2600cc810 0x7ff2601552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff26525a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 0x7ff2600c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff26525a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2600c81b0 con 0x7ff2601542e0 2023-11-28T18:33:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff26525a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2601f2a10 con 0x7ff2600cc810 2023-11-28T18:33:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff26525a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2600c83c0 con 0x7ff2600cbb70 2023-11-28T18:33:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff25f59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 0x7ff2600c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff257fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2600cc810 0x7ff2601552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff25ed9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 0x7ff260159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff257fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2600cc810 0x7ff2601552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:51618/0 (socket says 172.21.15.7:51618) 2023-11-28T18:33:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff257fff640 1 -- 172.21.15.7:0/4057796298 learned_addr learned my addr 172.21.15.7:0/4057796298 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff25f59e640 1 -- 172.21.15.7:0/4057796298 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 msgr2=0x7ff260159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff25f59e640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 0x7ff260159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff25f59e640 1 -- 172.21.15.7:0/4057796298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2600cc810 msgr2=0x7ff2601552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff25f59e640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2600cc810 0x7ff2601552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff25f59e640 1 -- 172.21.15.7:0/4057796298 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2500094d0 con 0x7ff2601542e0 2023-11-28T18:33:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.701+0000 7ff25f59e640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 0x7ff2600c9180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff258007d90 tx=0x7ff25800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:02.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff2577fe640 1 -- 172.21.15.7:0/4057796298 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258018070 con 0x7ff2601542e0 2023-11-28T18:33:02.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff2577fe640 1 -- 172.21.15.7:0/4057796298 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25800edd0 con 0x7ff2601542e0 2023-11-28T18:33:02.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2600c86a0 con 0x7ff2601542e0 2023-11-28T18:33:02.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff2577fe640 1 -- 172.21.15.7:0/4057796298 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258003bc0 con 0x7ff2601542e0 2023-11-28T18:33:02.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2600c8c60 con 0x7ff2601542e0 2023-11-28T18:33:02.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26006c670 con 0x7ff2601542e0 2023-11-28T18:33:02.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff2577fe640 1 -- 172.21.15.7:0/4057796298 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff258014420 con 0x7ff2601542e0 2023-11-28T18:33:02.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff2577fe640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2400668d0 0x7ff240068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:02.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff2577fe640 1 -- 172.21.15.7:0/4057796298 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7ff2580848f0 con 0x7ff2601542e0 2023-11-28T18:33:02.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff25ed9d640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2400668d0 0x7ff240068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:02.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.705+0000 7ff25ed9d640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2400668d0 0x7ff240068d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff2500ae000 tx=0x7ff250008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:02.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.709+0000 7ff2577fe640 1 -- 172.21.15.7:0/4057796298 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff258051890 con 0x7ff2601542e0 2023-11-28T18:33:02.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.901+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2600c8e80 con 0x7ff2601542e0 2023-11-28T18:33:02.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.901+0000 7ff2577fe640 1 -- 172.21.15.7:0/4057796298 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5767 (secure 0 0 0) 0x7ff258054eb0 con 0x7ff2601542e0 2023-11-28T18:33:02.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:02.904 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:32:57.306053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:resolve","state_seq":27,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:02.904 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 49 2023-11-28T18:33:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2400668d0 msgr2=0x7ff240068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2400668d0 0x7ff240068d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff2500ae000 tx=0x7ff250008040 comp rx=0 tx=0).stop 2023-11-28T18:33:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 msgr2=0x7ff2600c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 0x7ff2600c9180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff258007d90 tx=0x7ff25800e4b0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 shutdown_connections 2023-11-28T18:33:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff2400668d0 0x7ff240068d90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2601542e0 0x7ff2600c9180 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2600cc810 0x7ff2601552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 --2- 172.21.15.7:0/4057796298 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2600cbb70 0x7ff260159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:02.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 >> 172.21.15.7:0/4057796298 conn(0x7ff2600717e0 msgr2=0x7ff260152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:02.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 shutdown_connections 2023-11-28T18:33:02.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:02.905+0000 7ff26525a640 1 -- 172.21.15.7:0/4057796298 wait complete. 2023-11-28T18:33:02.926 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:33:04.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:05.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac6126b640 1 Processor -- start 2023-11-28T18:33:05.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac6126b640 1 -- start start 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac6126b640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fac5c06e560 0x7fac5c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac6126b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c071e60 con 0x7fac5c157b90 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac6126b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c070930 con 0x7fac5c151cd0 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac6126b640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c157fa0 con 0x7fac5c06e560 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac5b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fac5c157b90 0x7fac5c162510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41012/0 (socket says 172.21.15.7:41012) 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac5b7fe640 1 -- 172.21.15.7:0/3514817075 learned_addr learned my addr 172.21.15.7:0/3514817075 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.081+0000 7fac527fc640 1 --2- 172.21.15.7:0/3514817075 >> v2:172.21.15.193:3300/0 conn(0x7fac5c06e560 0x7fac5c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac5a7fc640 1 -- 172.21.15.7:0/3514817075 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259112873 0 0) 0x7fac5c071e60 con 0x7fac5c157b90 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac5a7fc640 1 -- 172.21.15.7:0/3514817075 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48003530 con 0x7fac5c157b90 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac527fc640 1 -- 172.21.15.7:0/3514817075 >> v1:172.21.15.193:6789/0 conn(0x7fac5c151cd0 legacy=0x7fac5c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac527fc640 1 -- 172.21.15.7:0/3514817075 >> v1:172.21.15.7:6789/0 conn(0x7fac5c157b90 legacy=0x7fac5c162510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:05.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac527fc640 1 -- 172.21.15.7:0/3514817075 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c070bf0 con 0x7fac5c06e560 2023-11-28T18:33:05.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac527fc640 1 --2- 172.21.15.7:0/3514817075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 0x7fac5c15bd00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac4401c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b7e1715d84548fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:05.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac5a7fc640 1 -- 172.21.15.7:0/3514817075 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440a5870 con 0x7fac5c06e560 2023-11-28T18:33:05.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac5a7fc640 1 -- 172.21.15.7:0/3514817075 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac440a59d0 con 0x7fac5c06e560 2023-11-28T18:33:05.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.085+0000 7fac5a7fc640 1 -- 172.21.15.7:0/3514817075 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440a5d60 con 0x7fac5c06e560 2023-11-28T18:33:05.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- 172.21.15.7:0/3514817075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 msgr2=0x7fac5c15bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:05.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 --2- 172.21.15.7:0/3514817075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 0x7fac5c15bd00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac4401c8c0 comp rx=0 tx=0).stop 2023-11-28T18:33:05.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- 172.21.15.7:0/3514817075 shutdown_connections 2023-11-28T18:33:05.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 --2- 172.21.15.7:0/3514817075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 0x7fac5c15bd00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:05.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- 172.21.15.7:0/3514817075 >> 172.21.15.7:0/3514817075 conn(0x7fac5c0662e0 msgr2=0x7fac5c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:05.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- 172.21.15.7:0/3514817075 shutdown_connections 2023-11-28T18:33:05.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- 172.21.15.7:0/3514817075 wait complete. 2023-11-28T18:33:05.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 Processor -- start 2023-11-28T18:33:05.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- start start 2023-11-28T18:33:05.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 0x7fac5c157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:05.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac5c151cd0 0x7fac5c147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:05.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5c157b90 0x7fac5c13ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:05.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0c4190 con 0x7fac5c151cd0 2023-11-28T18:33:05.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0c38f0 con 0x7fac5c06e560 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac6126b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0c3bd0 con 0x7fac5c157b90 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac527fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac5c151cd0 0x7fac5c147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac527fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac5c151cd0 0x7fac5c147840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53396/0 (socket says 172.21.15.7:53396) 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac527fc640 1 -- 172.21.15.7:0/788420047 learned_addr learned my addr 172.21.15.7:0/788420047 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac5affd640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 0x7fac5c157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac5b7fe640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5c157b90 0x7fac5c13ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac527fc640 1 -- 172.21.15.7:0/788420047 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5c157b90 msgr2=0x7fac5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac527fc640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5c157b90 0x7fac5c13ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac527fc640 1 -- 172.21.15.7:0/788420047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 msgr2=0x7fac5c157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:05.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac527fc640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 0x7fac5c157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:05.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.089+0000 7fac527fc640 1 -- 172.21.15.7:0/788420047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac44009a70 con 0x7fac5c151cd0 2023-11-28T18:33:05.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac527fc640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac5c151cd0 0x7fac5c147840 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fac440194d0 tx=0x7fac44019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:05.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac53fff640 1 -- 172.21.15.7:0/788420047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440a5870 con 0x7fac5c151cd0 2023-11-28T18:33:05.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac53fff640 1 -- 172.21.15.7:0/788420047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac440b9860 con 0x7fac5c151cd0 2023-11-28T18:33:05.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac53fff640 1 -- 172.21.15.7:0/788420047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440b9bf0 con 0x7fac5c151cd0 2023-11-28T18:33:05.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c0c3eb0 con 0x7fac5c151cd0 2023-11-28T18:33:05.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c147320 con 0x7fac5c151cd0 2023-11-28T18:33:05.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c0c26c0 con 0x7fac5c151cd0 2023-11-28T18:33:05.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac53fff640 1 -- 172.21.15.7:0/788420047 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fac440b9d50 con 0x7fac5c151cd0 2023-11-28T18:33:05.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac53fff640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:05.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.093+0000 7fac53fff640 1 -- 172.21.15.7:0/788420047 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fac4411f870 con 0x7fac5c151cd0 2023-11-28T18:33:05.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.097+0000 7fac5affd640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:05.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.097+0000 7fac5affd640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac4c007fd0 tx=0x7fac4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:05.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.101+0000 7fac53fff640 1 -- 172.21.15.7:0/788420047 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fac440ec8a0 con 0x7fac5c151cd0 2023-11-28T18:33:05.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac5c147510 con 0x7fac5c151cd0 2023-11-28T18:33:05.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac53fff640 1 -- 172.21.15.7:0/788420047 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fac440efec0 con 0x7fac5c151cd0 2023-11-28T18:33:05.290 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:05.290 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:03.259545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:05.291 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 50 2023-11-28T18:33:05.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 msgr2=0x7fac3c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac4c007fd0 tx=0x7fac4c009040 comp rx=0 tx=0).stop 2023-11-28T18:33:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac5c151cd0 msgr2=0x7fac5c147840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac5c151cd0 0x7fac5c147840 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fac440194d0 tx=0x7fac44019660 comp rx=0 tx=0).stop 2023-11-28T18:33:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 shutdown_connections 2023-11-28T18:33:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5c157b90 0x7fac5c13ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac5c151cd0 0x7fac5c147840 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 --2- 172.21.15.7:0/788420047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5c06e560 0x7fac5c157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:05.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 >> 172.21.15.7:0/788420047 conn(0x7fac5c0662e0 msgr2=0x7fac5c0caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:05.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 shutdown_connections 2023-11-28T18:33:05.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:05.289+0000 7fac6126b640 1 -- 172.21.15.7:0/788420047 wait complete. 2023-11-28T18:33:05.314 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4291": { "addr": "172.21.15.7:6839/1819203827", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 1819203827, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 1819203827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4291, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:reconnect", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:33:03.259545+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4291, "mds_1": 5359, "mds_2": 5307, "mds_3": 5086, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:33:07.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:07.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f71c9a640 1 Processor -- start 2023-11-28T18:33:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f71c9a640 1 -- start start 2023-11-28T18:33:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f71c9a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2f6c151cd0 0x7f2f6c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f71c9a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2f6c157b90 0x7f2f6c0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f71c9a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c071e60 con 0x7f2f6c06e560 2023-11-28T18:33:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f71c9a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c070730 con 0x7f2f6c157b90 2023-11-28T18:33:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f71c9a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c070930 con 0x7f2f6c151cd0 2023-11-28T18:33:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6affd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2f6c06e560 0x7f2f6c0c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41026/0 (socket says 172.21.15.7:41026) 2023-11-28T18:33:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6affd640 1 -- 172.21.15.7:0/1859843713 learned_addr learned my addr 172.21.15.7:0/1859843713 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:33:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6bfff640 1 --2- 172.21.15.7:0/1859843713 >> v2:172.21.15.7:3300/0 conn(0x7f2f6c157b90 0x7f2f6c0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6b7fe640 1 --2- 172.21.15.7:0/1859843713 >> v2:172.21.15.193:3300/0 conn(0x7f2f6c151cd0 0x7f2f6c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6bfff640 1 -- 172.21.15.7:0/1859843713 >> v2:172.21.15.193:3300/0 conn(0x7f2f6c151cd0 msgr2=0x7f2f6c15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6bfff640 1 --2- 172.21.15.7:0/1859843713 >> v2:172.21.15.193:3300/0 conn(0x7f2f6c151cd0 0x7f2f6c15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6bfff640 1 -- 172.21.15.7:0/1859843713 >> v1:172.21.15.7:6789/0 conn(0x7f2f6c06e560 legacy=0x7f2f6c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6a7fc640 1 -- 172.21.15.7:0/1859843713 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485158176 0 0) 0x7f2f6c071e60 con 0x7f2f6c06e560 2023-11-28T18:33:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.469+0000 7f2f6bfff640 1 -- 172.21.15.7:0/1859843713 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c157fa0 con 0x7f2f6c157b90 2023-11-28T18:33:07.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f6bfff640 1 --2- 172.21.15.7:0/1859843713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c157b90 0x7f2f6c0701f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c06fee0 tx=0x7f2f6401c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5708de46b4e616 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:07.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f6a7fc640 1 -- 172.21.15.7:0/1859843713 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f640a5830 con 0x7f2f6c157b90 2023-11-28T18:33:07.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f6a7fc640 1 -- 172.21.15.7:0/1859843713 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f640a5990 con 0x7f2f6c157b90 2023-11-28T18:33:07.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f6a7fc640 1 -- 172.21.15.7:0/1859843713 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f640a5d20 con 0x7f2f6c157b90 2023-11-28T18:33:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f71c9a640 1 -- 172.21.15.7:0/1859843713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c157b90 msgr2=0x7f2f6c0701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/1859843713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c157b90 0x7f2f6c0701f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c06fee0 tx=0x7f2f6401c880 comp rx=0 tx=0).stop 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f71c9a640 1 -- 172.21.15.7:0/1859843713 shutdown_connections 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/1859843713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c157b90 0x7f2f6c0701f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/1859843713 >> v2:172.21.15.193:3300/0 conn(0x7f2f6c151cd0 0x7f2f6c15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f71c9a640 1 -- 172.21.15.7:0/1859843713 >> 172.21.15.7:0/1859843713 conn(0x7f2f6c0662e0 msgr2=0x7f2f6c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f71c9a640 1 -- 172.21.15.7:0/1859843713 shutdown_connections 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.473+0000 7f2f71c9a640 1 -- 172.21.15.7:0/1859843713 wait complete. 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 Processor -- start 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 -- start start 2023-11-28T18:33:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c06e560 0x7f2f6c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f6c151cd0 0x7f2f6c13bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2f6c157b90 0x7f2f6c13c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0c2c40 con 0x7f2f6c06e560 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1618d0 con 0x7f2f6c151cd0 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c161b90 con 0x7f2f6c157b90 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c06e560 0x7f2f6c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c06e560 0x7f2f6c1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53418/0 (socket says 172.21.15.7:53418) 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 -- 172.21.15.7:0/3564161600 learned_addr learned my addr 172.21.15.7:0/3564161600 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6affd640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f6c151cd0 0x7f2f6c13bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6bfff640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2f6c157b90 0x7f2f6c13c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 -- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2f6c157b90 msgr2=0x7f2f6c13c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2f6c157b90 0x7f2f6c13c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 -- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f6c151cd0 msgr2=0x7f2f6c13bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f6c151cd0 0x7f2f6c13bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 -- 172.21.15.7:0/3564161600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f64009a70 con 0x7f2f6c06e560 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6affd640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f6c151cd0 0x7f2f6c13bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f6b7fe640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c06e560 0x7f2f6c1570a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c00e930 tx=0x7f2f5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f53fff640 1 -- 172.21.15.7:0/3564161600 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c009690 con 0x7f2f6c06e560 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f53fff640 1 -- 172.21.15.7:0/3564161600 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c00ca00 con 0x7f2f6c06e560 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f53fff640 1 -- 172.21.15.7:0/3564161600 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c00cd70 con 0x7f2f6c06e560 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.477+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c0c2e30 con 0x7f2f6c06e560 2023-11-28T18:33:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.481+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c0c48d0 con 0x7f2f6c06e560 2023-11-28T18:33:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.481+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c0c26b0 con 0x7f2f6c06e560 2023-11-28T18:33:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.481+0000 7f2f53fff640 1 -- 172.21.15.7:0/3564161600 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2f5c013470 con 0x7f2f6c06e560 2023-11-28T18:33:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.481+0000 7f2f53fff640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2f58066900 0x7f2f58068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.481+0000 7f2f53fff640 1 -- 172.21.15.7:0/3564161600 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f2f5c0848a0 con 0x7f2f6c06e560 2023-11-28T18:33:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.481+0000 7f2f6affd640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2f58066900 0x7f2f58068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.481+0000 7f2f6affd640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2f58066900 0x7f2f58068dc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c1608e0 tx=0x7f2f60007c40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.485+0000 7f2f53fff640 1 -- 172.21.15.7:0/3564161600 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2f5c051990 con 0x7f2f6c06e560 2023-11-28T18:33:07.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.677+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f6c04e890 con 0x7f2f6c06e560 2023-11-28T18:33:07.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.677+0000 7f2f53fff640 1 -- 172.21.15.7:0/3564161600 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f2f5c054fb0 con 0x7f2f6c06e560 2023-11-28T18:33:07.679 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:07.679 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:03.259545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:07.680 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 50 2023-11-28T18:33:07.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2f58066900 msgr2=0x7f2f58068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:07.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2f58066900 0x7f2f58068dc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c1608e0 tx=0x7f2f60007c40 comp rx=0 tx=0).stop 2023-11-28T18:33:07.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c06e560 msgr2=0x7f2f6c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:07.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c06e560 0x7f2f6c1570a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c00e930 tx=0x7f2f5c00edf0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 shutdown_connections 2023-11-28T18:33:07.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2f58066900 0x7f2f58068dc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2f6c157b90 0x7f2f6c13c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f6c151cd0 0x7f2f6c13bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 --2- 172.21.15.7:0/3564161600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f6c06e560 0x7f2f6c1570a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:07.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 >> 172.21.15.7:0/3564161600 conn(0x7f2f6c0662e0 msgr2=0x7f2f6c072580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:07.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 shutdown_connections 2023-11-28T18:33:07.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:07.681+0000 7f2f71c9a640 1 -- 172.21.15.7:0/3564161600 wait complete. 2023-11-28T18:33:07.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:33:09.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:09.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a505f8640 1 Processor -- start 2023-11-28T18:33:09.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a505f8640 1 -- start start 2023-11-28T18:33:09.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a505f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a481542e0 0x7f5a4815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:09.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a505f8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5a48072450 0x7f5a480728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:09.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a505f8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a480ca920 con 0x7f5a480cbb70 2023-11-28T18:33:09.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a505f8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a48071d50 con 0x7f5a48072450 2023-11-28T18:33:09.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a505f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a48072030 con 0x7f5a481542e0 2023-11-28T18:33:09.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4eb95640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5a48072450 0x7f5a480728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:09.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4e394640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a481542e0 0x7f5a4815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:09.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4eb95640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5a48072450 0x7f5a480728e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:35348/0 (socket says 172.21.15.7:35348) 2023-11-28T18:33:09.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4eb95640 1 -- 172.21.15.7:0/625438618 learned_addr learned my addr 172.21.15.7:0/625438618 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:09.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4db93640 1 --1- 172.21.15.7:0/625438618 >> v1:172.21.15.193:6789/0 conn(0x7f5a480cbb70 0x7f5a480cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:32814/0 (socket says 172.21.15.7:32814) 2023-11-28T18:33:09.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4eb95640 1 -- 172.21.15.7:0/625438618 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a481542e0 msgr2=0x7f5a4815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:09.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4eb95640 1 --2- 172.21.15.7:0/625438618 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a481542e0 0x7f5a4815e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:09.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4eb95640 1 -- 172.21.15.7:0/625438618 >> v1:172.21.15.193:6789/0 conn(0x7f5a480cbb70 legacy=0x7f5a480cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:09.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4eb95640 1 -- 172.21.15.7:0/625438618 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4815e550 con 0x7f5a48072450 2023-11-28T18:33:09.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.857+0000 7f5a4e394640 1 --2- 172.21.15.7:0/625438618 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a481542e0 0x7f5a4815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:33:09.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a4eb95640 1 --2- 172.21.15.7:0/625438618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a48072450 0x7f5a480728e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009e50 tx=0x7f5a4001ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f18e25319033cb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:09.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a4d392640 1 -- 172.21.15.7:0/625438618 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a400a57f0 con 0x7f5a48072450 2023-11-28T18:33:09.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a4d392640 1 -- 172.21.15.7:0/625438618 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a400a5950 con 0x7f5a48072450 2023-11-28T18:33:09.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a4d392640 1 -- 172.21.15.7:0/625438618 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a400a5ce0 con 0x7f5a48072450 2023-11-28T18:33:09.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a505f8640 1 -- 172.21.15.7:0/625438618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a48072450 msgr2=0x7f5a480728e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a505f8640 1 --2- 172.21.15.7:0/625438618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a48072450 0x7f5a480728e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009e50 tx=0x7f5a4001ca10 comp rx=0 tx=0).stop 2023-11-28T18:33:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a505f8640 1 -- 172.21.15.7:0/625438618 shutdown_connections 2023-11-28T18:33:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a505f8640 1 --2- 172.21.15.7:0/625438618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a48072450 0x7f5a480728e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a505f8640 1 --2- 172.21.15.7:0/625438618 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a481542e0 0x7f5a4815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a505f8640 1 -- 172.21.15.7:0/625438618 >> 172.21.15.7:0/625438618 conn(0x7f5a480717e0 msgr2=0x7f5a480662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:09.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a505f8640 1 -- 172.21.15.7:0/625438618 shutdown_connections 2023-11-28T18:33:09.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.861+0000 7f5a505f8640 1 -- 172.21.15.7:0/625438618 wait complete. 2023-11-28T18:33:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 Processor -- start 2023-11-28T18:33:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 -- start start 2023-11-28T18:33:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a48072450 0x7f5a481f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a480cbb70 0x7f5a481534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a481542e0 0x7f5a4813e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a481f6430 con 0x7f5a48072450 2023-11-28T18:33:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a481f7380 con 0x7f5a481542e0 2023-11-28T18:33:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a481f69b0 con 0x7f5a480cbb70 2023-11-28T18:33:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a48072450 0x7f5a481f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a48072450 0x7f5a481f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53420/0 (socket says 172.21.15.7:53420) 2023-11-28T18:33:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 -- 172.21.15.7:0/333821541 learned_addr learned my addr 172.21.15.7:0/333821541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 -- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a480cbb70 msgr2=0x7f5a481534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4eb95640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a481542e0 0x7f5a4813e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4db93640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a480cbb70 0x7f5a481534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a480cbb70 0x7f5a481534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 -- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a481542e0 msgr2=0x7f5a4813e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a481542e0 0x7f5a4813e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 -- 172.21.15.7:0/333821541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a44009420 con 0x7f5a48072450 2023-11-28T18:33:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4eb95640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a481542e0 0x7f5a4813e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:33:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4db93640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a480cbb70 0x7f5a481534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a4e394640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a48072450 0x7f5a481f6e40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400e9b0 tx=0x7f5a4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a3effd640 1 -- 172.21.15.7:0/333821541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4400c9a0 con 0x7f5a48072450 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a3effd640 1 -- 172.21.15.7:0/333821541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4400cb00 con 0x7f5a48072450 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a3effd640 1 -- 172.21.15.7:0/333821541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4400ce90 con 0x7f5a48072450 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a40009b30 con 0x7f5a48072450 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.865+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4813fc20 con 0x7f5a48072450 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.869+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4806c670 con 0x7f5a48072450 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.869+0000 7f5a3effd640 1 -- 172.21.15.7:0/333821541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5a44013470 con 0x7f5a48072450 2023-11-28T18:33:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.869+0000 7f5a3effd640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5a2c0668d0 0x7f5a2c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.869+0000 7f5a3effd640 1 -- 172.21.15.7:0/333821541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f5a44084950 con 0x7f5a48072450 2023-11-28T18:33:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.869+0000 7f5a4db93640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5a2c0668d0 0x7f5a2c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.869+0000 7f5a4db93640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5a2c0668d0 0x7f5a2c068d90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5a481406e0 tx=0x7f5a38009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:09.873+0000 7f5a3effd640 1 -- 172.21.15.7:0/333821541 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5a440518d0 con 0x7f5a48072450 2023-11-28T18:33:10.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a4804e890 con 0x7f5a48072450 2023-11-28T18:33:10.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a3effd640 1 -- 172.21.15.7:0/333821541 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5766 (secure 0 0 0) 0x7f5a44054ef0 con 0x7f5a48072450 2023-11-28T18:33:10.066 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:10.066 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:08.447411+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 51 2023-11-28T18:33:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5a2c0668d0 msgr2=0x7f5a2c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5a2c0668d0 0x7f5a2c068d90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5a481406e0 tx=0x7f5a38009040 comp rx=0 tx=0).stop 2023-11-28T18:33:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a48072450 msgr2=0x7f5a481f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a48072450 0x7f5a481f6e40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400e9b0 tx=0x7f5a4400ee70 comp rx=0 tx=0).stop 2023-11-28T18:33:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 shutdown_connections 2023-11-28T18:33:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5a2c0668d0 0x7f5a2c068d90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a481542e0 0x7f5a4813e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5a480cbb70 0x7f5a481534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 --2- 172.21.15.7:0/333821541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a48072450 0x7f5a481f6e40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 >> 172.21.15.7:0/333821541 conn(0x7f5a480717e0 msgr2=0x7f5a48144db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:10.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 shutdown_connections 2023-11-28T18:33:10.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:10.065+0000 7f5a505f8640 1 -- 172.21.15.7:0/333821541 wait complete. 2023-11-28T18:33:10.088 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:33:12.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac7a32640 1 Processor -- start 2023-11-28T18:33:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac7a32640 1 -- start start 2023-11-28T18:33:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac7a32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c24f0 0x7ffac01575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac7a32640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ffac00c7980 0x7ffac00c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac7a32640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00c6140 con 0x7ffac0150b30 2023-11-28T18:33:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac7a32640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac00c72b0 con 0x7ffac00c7980 2023-11-28T18:33:12.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac7a32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac0070c00 con 0x7ffac00c24f0 2023-11-28T18:33:12.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac57ce640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ffac0150b30 0x7ffac00c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41040/0 (socket says 172.21.15.7:41040) 2023-11-28T18:33:12.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac57ce640 1 -- 172.21.15.7:0/1317403708 learned_addr learned my addr 172.21.15.7:0/1317403708 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:33:12.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac5fcf640 1 --2- 172.21.15.7:0/1317403708 >> v2:172.21.15.193:3300/0 conn(0x7ffac00c7980 0x7ffac00c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:12.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac4fcd640 1 --2- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c24f0 0x7ffac01575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:12.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffab7fff640 1 -- 172.21.15.7:0/1317403708 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065585123 0 0) 0x7ffac00c6140 con 0x7ffac0150b30 2023-11-28T18:33:12.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffab7fff640 1 -- 172.21.15.7:0/1317403708 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaac003530 con 0x7ffac0150b30 2023-11-28T18:33:12.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffab7fff640 1 -- 172.21.15.7:0/1317403708 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 60529983 0 0) 0x7ffaac003530 con 0x7ffac0150b30 2023-11-28T18:33:12.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac5fcf640 1 -- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c24f0 msgr2=0x7ffac01575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:12.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac5fcf640 1 --2- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c24f0 0x7ffac01575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac5fcf640 1 -- 172.21.15.7:0/1317403708 >> v1:172.21.15.7:6789/0 conn(0x7ffac0150b30 legacy=0x7ffac00c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac5fcf640 1 -- 172.21.15.7:0/1317403708 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0157b00 con 0x7ffac00c7980 2023-11-28T18:33:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.245+0000 7ffac4fcd640 1 --2- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c24f0 0x7ffac01575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:33:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac5fcf640 1 --2- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c7980 0x7ffac00c7e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009d90 tx=0x7ffab801f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9042ed6372d4e8c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffab7fff640 1 -- 172.21.15.7:0/1317403708 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc001680 con 0x7ffac00c7980 2023-11-28T18:33:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffab7fff640 1 -- 172.21.15.7:0/1317403708 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab80a8bb0 con 0x7ffac00c7980 2023-11-28T18:33:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffab7fff640 1 -- 172.21.15.7:0/1317403708 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab80b0440 con 0x7ffac00c7980 2023-11-28T18:33:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac7a32640 1 -- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c7980 msgr2=0x7ffac00c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac7a32640 1 --2- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c7980 0x7ffac00c7e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009d90 tx=0x7ffab801f060 comp rx=0 tx=0).stop 2023-11-28T18:33:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac7a32640 1 -- 172.21.15.7:0/1317403708 shutdown_connections 2023-11-28T18:33:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac7a32640 1 --2- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c24f0 0x7ffac01575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac7a32640 1 --2- 172.21.15.7:0/1317403708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c7980 0x7ffac00c7e10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac7a32640 1 -- 172.21.15.7:0/1317403708 >> 172.21.15.7:0/1317403708 conn(0x7ffac00662e0 msgr2=0x7ffac00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac7a32640 1 -- 172.21.15.7:0/1317403708 shutdown_connections 2023-11-28T18:33:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.249+0000 7ffac7a32640 1 -- 172.21.15.7:0/1317403708 wait complete. 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac7a32640 1 Processor -- start 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac7a32640 1 -- start start 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac7a32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c24f0 0x7ffac00be550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac7a32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c7980 0x7ffac0155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac7a32640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac0150b30 0x7ffac0152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac7a32640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00bea90 con 0x7ffac0150b30 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac7a32640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac0151e30 con 0x7ffac00c24f0 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac7a32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac0150fb0 con 0x7ffac00c7980 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac0150b30 0x7ffac0152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac0150b30 0x7ffac0152920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53424/0 (socket says 172.21.15.7:53424) 2023-11-28T18:33:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 -- 172.21.15.7:0/3577271750 learned_addr learned my addr 172.21.15.7:0/3577271750 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac4fcd640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c7980 0x7ffac0155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac57ce640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c24f0 0x7ffac00be550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 -- 172.21.15.7:0/3577271750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c7980 msgr2=0x7ffac0155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c7980 0x7ffac0155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 -- 172.21.15.7:0/3577271750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c24f0 msgr2=0x7ffac00be550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c24f0 0x7ffac00be550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 -- 172.21.15.7:0/3577271750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab0009420 con 0x7ffac0150b30 2023-11-28T18:33:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.253+0000 7ffac5fcf640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac0150b30 0x7ffac0152920 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009e80 tx=0x7ffab801be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:12.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffab5ffb640 1 -- 172.21.15.7:0/3577271750 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8009450 con 0x7ffac0150b30 2023-11-28T18:33:12.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab8009a70 con 0x7ffac0150b30 2023-11-28T18:33:12.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac01517e0 con 0x7ffac0150b30 2023-11-28T18:33:12.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffab5ffb640 1 -- 172.21.15.7:0/3577271750 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab80c8460 con 0x7ffac0150b30 2023-11-28T18:33:12.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffab5ffb640 1 -- 172.21.15.7:0/3577271750 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab80c87d0 con 0x7ffac0150b30 2023-11-28T18:33:12.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa0005260 con 0x7ffac0150b30 2023-11-28T18:33:12.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffab5ffb640 1 -- 172.21.15.7:0/3577271750 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ffab80b7050 con 0x7ffac0150b30 2023-11-28T18:33:12.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffab5ffb640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffaa40668d0 0x7ffaa4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:12.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffab5ffb640 1 -- 172.21.15.7:0/3577271750 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7ffab8122970 con 0x7ffac0150b30 2023-11-28T18:33:12.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.257+0000 7ffac57ce640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffaa40668d0 0x7ffaa4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:12.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.261+0000 7ffac57ce640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffaa40668d0 0x7ffaa4068d90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006b60 tx=0x7ffabc001dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:12.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.261+0000 7ffab5ffb640 1 -- 172.21.15.7:0/3577271750 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffab80efa30 con 0x7ffac0150b30 2023-11-28T18:33:12.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.453+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaa0005a90 con 0x7ffac0150b30 2023-11-28T18:33:12.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.453+0000 7ffab5ffb640 1 -- 172.21.15.7:0/3577271750 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5766 (secure 0 0 0) 0x7ffab80b5090 con 0x7ffac0150b30 2023-11-28T18:33:12.456 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:12.456 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:08.447411+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:12.456 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 51 2023-11-28T18:33:12.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffaa40668d0 msgr2=0x7ffaa4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:12.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffaa40668d0 0x7ffaa4068d90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006b60 tx=0x7ffabc001dd0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac0150b30 msgr2=0x7ffac0152920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:12.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac0150b30 0x7ffac0152920 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009e80 tx=0x7ffab801be00 comp rx=0 tx=0).stop 2023-11-28T18:33:12.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 shutdown_connections 2023-11-28T18:33:12.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffaa40668d0 0x7ffaa4068d90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac0150b30 0x7ffac0152920 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffac00c7980 0x7ffac0155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 --2- 172.21.15.7:0/3577271750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00c24f0 0x7ffac00be550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:12.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 >> 172.21.15.7:0/3577271750 conn(0x7ffac00662e0 msgr2=0x7ffac00c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:12.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 shutdown_connections 2023-11-28T18:33:12.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:12.457+0000 7ffac7a32640 1 -- 172.21.15.7:0/3577271750 wait complete. 2023-11-28T18:33:12.478 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:33:14.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:14.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.633+0000 7f42c44fe640 1 Processor -- start 2023-11-28T18:33:14.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.633+0000 7f42c44fe640 1 -- start start 2023-11-28T18:33:14.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c44fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc072950 0x7f42bc074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:14.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c44fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f42bc06c920 0x7f42bc06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:14.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c44fe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0c8490 con 0x7f42bc073070 2023-11-28T18:33:14.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c44fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42bc06d2e0 con 0x7f42bc06c920 2023-11-28T18:33:14.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c44fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0c81f0 con 0x7f42bc072950 2023-11-28T18:33:14.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c1a99640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f42bc073070 0x7f42bc0c19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49488/0 (socket says 172.21.15.7:49488) 2023-11-28T18:33:14.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c1a99640 1 -- 172.21.15.7:0/4108667387 learned_addr learned my addr 172.21.15.7:0/4108667387 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:33:14.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c2a9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f42bc06c920 0x7f42bc06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:14.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c229a640 1 --2- 172.21.15.7:0/4108667387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc072950 0x7f42bc074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:14.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c2a9b640 1 -- 172.21.15.7:0/4108667387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc072950 msgr2=0x7f42bc074030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:14.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c2a9b640 1 --2- 172.21.15.7:0/4108667387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc072950 0x7f42bc074030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c2a9b640 1 -- 172.21.15.7:0/4108667387 >> v1:172.21.15.7:6789/0 conn(0x7f42bc073070 legacy=0x7f42bc0c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:14.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c2a9b640 1 -- 172.21.15.7:0/4108667387 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc15ab00 con 0x7f42bc06c920 2023-11-28T18:33:14.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c229a640 1 --2- 172.21.15.7:0/4108667387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc072950 0x7f42bc074030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:33:14.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.637+0000 7f42c1298640 1 -- 172.21.15.7:0/4108667387 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215609673 0 0) 0x7f42bc0c8490 con 0x7f42bc073070 2023-11-28T18:33:14.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c2a9b640 1 --2- 172.21.15.7:0/4108667387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc06c920 0x7f42bc06cda0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009d90 tx=0x7f42b401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfcc0f9d7d274e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:14.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c1298640 1 -- 172.21.15.7:0/4108667387 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40a66e0 con 0x7f42bc06c920 2023-11-28T18:33:14.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c1298640 1 -- 172.21.15.7:0/4108667387 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b40a6840 con 0x7f42bc06c920 2023-11-28T18:33:14.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c1298640 1 -- 172.21.15.7:0/4108667387 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40a6bd0 con 0x7f42bc06c920 2023-11-28T18:33:14.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- 172.21.15.7:0/4108667387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc06c920 msgr2=0x7f42bc06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:14.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 --2- 172.21.15.7:0/4108667387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc06c920 0x7f42bc06cda0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009d90 tx=0x7f42b401c830 comp rx=0 tx=0).stop 2023-11-28T18:33:14.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- 172.21.15.7:0/4108667387 shutdown_connections 2023-11-28T18:33:14.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 --2- 172.21.15.7:0/4108667387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc06c920 0x7f42bc06cda0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 --2- 172.21.15.7:0/4108667387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc072950 0x7f42bc074030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- 172.21.15.7:0/4108667387 >> 172.21.15.7:0/4108667387 conn(0x7f42bc0cbf80 msgr2=0x7f42bc0cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:14.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- 172.21.15.7:0/4108667387 shutdown_connections 2023-11-28T18:33:14.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- 172.21.15.7:0/4108667387 wait complete. 2023-11-28T18:33:14.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 Processor -- start 2023-11-28T18:33:14.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- start start 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc06c920 0x7f42bc146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc072950 0x7f42bc0c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42bc073070 0x7f42bc0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc15d240 con 0x7f42bc072950 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc15ddf0 con 0x7f42bc073070 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.641+0000 7f42c44fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc147230 con 0x7f42bc06c920 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc072950 0x7f42bc0c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc072950 0x7f42bc0c8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34228/0 (socket says 172.21.15.7:34228) 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 -- 172.21.15.7:0/1554647143 learned_addr learned my addr 172.21.15.7:0/1554647143 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:14.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c229a640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc06c920 0x7f42bc146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:14.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c2a9b640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42bc073070 0x7f42bc0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 -- 172.21.15.7:0/1554647143 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc06c920 msgr2=0x7f42bc146cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc06c920 0x7f42bc146cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 -- 172.21.15.7:0/1554647143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42bc073070 msgr2=0x7f42bc0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42bc073070 0x7f42bc0c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 -- 172.21.15.7:0/1554647143 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b4009a70 con 0x7f42bc072950 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c1a99640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc072950 0x7f42bc0c8a80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00c450 tx=0x7f42ac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42b2ffd640 1 -- 172.21.15.7:0/1554647143 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0074e0 con 0x7f42bc072950 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42b2ffd640 1 -- 172.21.15.7:0/1554647143 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ac007640 con 0x7f42bc072950 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42b2ffd640 1 -- 172.21.15.7:0/1554647143 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0049f0 con 0x7f42bc072950 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc1f6650 con 0x7f42bc072950 2023-11-28T18:33:14.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc1f5ad0 con 0x7f42bc072950 2023-11-28T18:33:14.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc15b140 con 0x7f42bc072950 2023-11-28T18:33:14.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.645+0000 7f42b2ffd640 1 -- 172.21.15.7:0/1554647143 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f42ac004b50 con 0x7f42bc072950 2023-11-28T18:33:14.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.649+0000 7f42b2ffd640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42a00668d0 0x7f42a0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:14.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.649+0000 7f42b2ffd640 1 -- 172.21.15.7:0/1554647143 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f42ac0837d0 con 0x7f42bc072950 2023-11-28T18:33:14.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.649+0000 7f42c229a640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42a00668d0 0x7f42a0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:14.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.649+0000 7f42c229a640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42a00668d0 0x7f42a0068d90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f42b8009950 tx=0x7f42b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.653+0000 7f42b2ffd640 1 -- 172.21.15.7:0/1554647143 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f42bc15b140 con 0x7f42bc072950 2023-11-28T18:33:14.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42bc15b140 con 0x7f42bc072950 2023-11-28T18:33:14.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42b2ffd640 1 -- 172.21.15.7:0/1554647143 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5766 (secure 0 0 0) 0x7f42bc15b140 con 0x7f42bc072950 2023-11-28T18:33:14.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:14.846 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:14.848 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 52 2023-11-28T18:33:14.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42a00668d0 msgr2=0x7f42a0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:14.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42a00668d0 0x7f42a0068d90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f42b8009950 tx=0x7f42b8008040 comp rx=0 tx=0).stop 2023-11-28T18:33:14.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc072950 msgr2=0x7f42bc0c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc072950 0x7f42bc0c8a80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00c450 tx=0x7f42ac00c910 comp rx=0 tx=0).stop 2023-11-28T18:33:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 shutdown_connections 2023-11-28T18:33:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42a00668d0 0x7f42a0068d90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42bc073070 0x7f42bc0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42bc072950 0x7f42bc0c8a80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 --2- 172.21.15.7:0/1554647143 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f42bc06c920 0x7f42bc146cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:14.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 >> 172.21.15.7:0/1554647143 conn(0x7f42bc0cbf80 msgr2=0x7f42bc0c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:14.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 shutdown_connections 2023-11-28T18:33:14.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:14.845+0000 7f42c44fe640 1 -- 172.21.15.7:0/1554647143 wait complete. 2023-11-28T18:33:14.867 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=1, replacing gid=4288 2023-11-28T18:33:14.867 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.c 2023-11-28T18:33:16.998 INFO:tasks.ceph.mds.e.smithi007.stderr:2023-11-28T18:33:16.997+0000 7f7e1dce5640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-11-28T18:33:21.868 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-11-28T18:33:21.868 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T18:33:21.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T18:33:21.873 INFO:tasks.ceph.mds.c:Started 2023-11-28T18:33:21.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:21.966 INFO:tasks.ceph.mds.c.smithi007.stdout:starting mds.c at 2023-11-28T18:33:22.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.021+0000 7fa2b4b2a640 1 Processor -- start 2023-11-28T18:33:22.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.021+0000 7fa2b4b2a640 1 -- start start 2023-11-28T18:33:22.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2b4b2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa2b006e560 0x7fa2b015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2b4b2a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa2b0157b90 0x7fa2b00701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2b4b2a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b0158000 con 0x7fa2b0151cd0 2023-11-28T18:33:22.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2b4b2a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b0070730 con 0x7fa2b0157b90 2023-11-28T18:33:22.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2b4b2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0071e60 con 0x7fa2b006e560 2023-11-28T18:33:22.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ae59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa2b0151cd0 0x7fa2b00c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49508/0 (socket says 172.21.15.7:49508) 2023-11-28T18:33:22.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ae59c640 1 -- 172.21.15.7:0/837137051 learned_addr learned my addr 172.21.15.7:0/837137051 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:33:22.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2aed9d640 1 --2- 172.21.15.7:0/837137051 >> v2:172.21.15.193:3300/0 conn(0x7fa2b0157b90 0x7fa2b00701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2add9b640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa2b006e560 0x7fa2b015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169124479 0 0) 0x7fa2b0158000 con 0x7fa2b0151cd0 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa298003530 con 0x7fa2b0151cd0 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1528967016 0 0) 0x7fa298003530 con 0x7fa2b0151cd0 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa2b006e560 msgr2=0x7fa2b015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa2b006e560 0x7fa2b015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 >> v2:172.21.15.193:3300/0 conn(0x7fa2b0157b90 msgr2=0x7fa2b00701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 --2- 172.21.15.7:0/837137051 >> v2:172.21.15.193:3300/0 conn(0x7fa2b0157b90 0x7fa2b00701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0070930 con 0x7fa2b0151cd0 2023-11-28T18:33:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fa2a0001680 con 0x7fa2b0151cd0 2023-11-28T18:33:22.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 >> v1:172.21.15.7:6789/0 conn(0x7fa2b0151cd0 legacy=0x7fa2b00c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:22.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298003980 0x7fa298003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2980047d0 0x7fa298004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa298005190 0x7fa29800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0070730 con 0x7fa298005190 2023-11-28T18:33:22.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0071e60 con 0x7fa2980047d0 2023-11-28T18:33:22.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29800bf50 con 0x7fa298003980 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2add9b640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa2b006e560 0x7fa2b015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2aed9d640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa298005190 0x7fa29800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2ae59c640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298003980 0x7fa298003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2add9b640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2980047d0 0x7fa298004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2aed9d640 1 -- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298003980 msgr2=0x7fa298003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2aed9d640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298003980 0x7fa298003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2aed9d640 1 -- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2980047d0 msgr2=0x7fa298004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2aed9d640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2980047d0 0x7fa298004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2aed9d640 1 -- 172.21.15.7:0/837137051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa298003530 con 0x7fa298005190 2023-11-28T18:33:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.025+0000 7fa2add9b640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2980047d0 0x7fa298004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2aed9d640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa298005190 0x7fa29800b990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009fc0 tx=0x7fa2a801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e296f410be8ab8d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80a5cb0 con 0x7fa298005190 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a800f040 con 0x7fa298005190 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2ad59a640 1 -- 172.21.15.7:0/837137051 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80ad680 con 0x7fa298005190 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/837137051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa298005190 msgr2=0x7fa29800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2b4b2a640 1 --2- 172.21.15.7:0/837137051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa298005190 0x7fa29800b990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009fc0 tx=0x7fa2a801cc60 comp rx=0 tx=0).stop 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2aed9d640 1 -- 172.21.15.7:0/837137051 reap_dead start 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/837137051 shutdown_connections 2023-11-28T18:33:22.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/837137051 >> 172.21.15.7:0/837137051 conn(0x7fa2b00662e0 msgr2=0x7fa2b00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:22.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/837137051 shutdown_connections 2023-11-28T18:33:22.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.029+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/837137051 wait complete. 2023-11-28T18:33:22.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 Processor -- start 2023-11-28T18:33:22.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 -- start start 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298005190 0x7fa2b014a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2b0150890 0x7fa2b01ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2b01efa20 0x7fa2b0146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b01f2a00 con 0x7fa2b01efa20 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b01f39b0 con 0x7fa2b0150890 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b014acb0 con 0x7fa298005190 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2b01efa20 0x7fa2b0146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2b01efa20 0x7fa2b0146950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34256/0 (socket says 172.21.15.7:34256) 2023-11-28T18:33:22.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 -- 172.21.15.7:0/1307008018 learned_addr learned my addr 172.21.15.7:0/1307008018 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2ae59c640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298005190 0x7fa2b014a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2add9b640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2b0150890 0x7fa2b01ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 -- 172.21.15.7:0/1307008018 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298005190 msgr2=0x7fa2b014a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298005190 0x7fa2b014a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 -- 172.21.15.7:0/1307008018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2b0150890 msgr2=0x7fa2b01ef4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2b0150890 0x7fa2b01ef4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 -- 172.21.15.7:0/1307008018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b01f2bf0 con 0x7fa2b01efa20 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2aed9d640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2b01efa20 0x7fa2b0146950 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa2a80b0040 tx=0x7fa2a8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa29effd640 1 -- 172.21.15.7:0/1307008018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80039a0 con 0x7fa2b01efa20 2023-11-28T18:33:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa29effd640 1 -- 172.21.15.7:0/1307008018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8003b20 con 0x7fa2b01efa20 2023-11-28T18:33:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa29effd640 1 -- 172.21.15.7:0/1307008018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80ad650 con 0x7fa2b01efa20 2023-11-28T18:33:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b01f2ed0 con 0x7fa2b01efa20 2023-11-28T18:33:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b01f3490 con 0x7fa2b01efa20 2023-11-28T18:33:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.033+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b01f3680 con 0x7fa2b01efa20 2023-11-28T18:33:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.037+0000 7fa29effd640 1 -- 172.21.15.7:0/1307008018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa2a80b4040 con 0x7fa2b01efa20 2023-11-28T18:33:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.037+0000 7fa29effd640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa290066900 0x7fa290068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.037+0000 7fa29effd640 1 -- 172.21.15.7:0/1307008018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fa2a811e190 con 0x7fa2b01efa20 2023-11-28T18:33:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.037+0000 7fa2ae59c640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa290066900 0x7fa290068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.037+0000 7fa2ae59c640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa290066900 0x7fa290068dc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0005540 tx=0x7fa2a00014f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:22.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.041+0000 7fa29effd640 1 -- 172.21.15.7:0/1307008018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa2a80eb1c0 con 0x7fa2b01efa20 2023-11-28T18:33:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.233+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2b004e890 con 0x7fa2b01efa20 2023-11-28T18:33:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.233+0000 7fa29effd640 1 -- 172.21.15.7:0/1307008018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5766 (secure 0 0 0) 0x7fa2a80ee7e0 con 0x7fa2b01efa20 2023-11-28T18:33:22.235 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:22.235 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:22.235 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 52 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa290066900 msgr2=0x7fa290068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa290066900 0x7fa290068dc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0005540 tx=0x7fa2a00014f0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2b01efa20 msgr2=0x7fa2b0146950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2b01efa20 0x7fa2b0146950 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa2a80b0040 tx=0x7fa2a8009750 comp rx=0 tx=0).stop 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 shutdown_connections 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa290066900 0x7fa290068dc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2b01efa20 0x7fa2b0146950 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2b0150890 0x7fa2b01ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 --2- 172.21.15.7:0/1307008018 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa298005190 0x7fa2b014a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:22.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 >> 172.21.15.7:0/1307008018 conn(0x7fa2b00662e0 msgr2=0x7fa2b00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:22.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 shutdown_connections 2023-11-28T18:33:22.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:22.237+0000 7fa2b4b2a640 1 -- 172.21.15.7:0/1307008018 wait complete. 2023-11-28T18:33:22.261 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-11-28T18:33:24.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:33:24.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac55679640 1 Processor -- start 2023-11-28T18:33:24.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac55679640 1 -- start start 2023-11-28T18:33:24.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac55679640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac50154300 0x7fac50159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:24.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac55679640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5015d490 con 0x7fac5015dc50 2023-11-28T18:33:24.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac55679640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5015a480 con 0x7fac5015e090 2023-11-28T18:33:24.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac55679640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5015a760 con 0x7fac50154300 2023-11-28T18:33:24.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac4effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac50154300 0x7fac50159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:24.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac4effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac50154300 0x7fac50159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42514/0 (socket says 172.21.15.7:42514) 2023-11-28T18:33:24.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac4effd640 1 -- 172.21.15.7:0/2406286034 learned_addr learned my addr 172.21.15.7:0/2406286034 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:24.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.429+0000 7fac4e7fc640 1 --1- 172.21.15.7:0/2406286034 >> v1:172.21.15.7:6789/0 conn(0x7fac5015dc50 0x7fac500cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56182/0 (socket says 172.21.15.7:56182) 2023-11-28T18:33:24.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.433+0000 7fac4effd640 1 -- 172.21.15.7:0/2406286034 >> v1:172.21.15.193:6789/0 conn(0x7fac5015e090 legacy=0x7fac500ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:24.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.433+0000 7fac4effd640 1 -- 172.21.15.7:0/2406286034 >> v1:172.21.15.7:6789/0 conn(0x7fac5015dc50 legacy=0x7fac500cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:24.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.433+0000 7fac4effd640 1 -- 172.21.15.7:0/2406286034 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5015aa40 con 0x7fac50154300 2023-11-28T18:33:24.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac4effd640 1 --2- 172.21.15.7:0/2406286034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50159e40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fac40009870 tx=0x7fac4001cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8eb30e8ae06ddc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:24.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac4dffb640 1 -- 172.21.15.7:0/2406286034 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac400a95f0 con 0x7fac50154300 2023-11-28T18:33:24.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac4dffb640 1 -- 172.21.15.7:0/2406286034 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac400a9750 con 0x7fac50154300 2023-11-28T18:33:24.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac4dffb640 1 -- 172.21.15.7:0/2406286034 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac400a9ae0 con 0x7fac50154300 2023-11-28T18:33:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 -- 172.21.15.7:0/2406286034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 msgr2=0x7fac50159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 --2- 172.21.15.7:0/2406286034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50159e40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fac40009870 tx=0x7fac4001cf70 comp rx=0 tx=0).stop 2023-11-28T18:33:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 -- 172.21.15.7:0/2406286034 shutdown_connections 2023-11-28T18:33:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 --2- 172.21.15.7:0/2406286034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50159e40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 -- 172.21.15.7:0/2406286034 >> 172.21.15.7:0/2406286034 conn(0x7fac500662e0 msgr2=0x7fac500666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 -- 172.21.15.7:0/2406286034 shutdown_connections 2023-11-28T18:33:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 -- 172.21.15.7:0/2406286034 wait complete. 2023-11-28T18:33:24.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 Processor -- start 2023-11-28T18:33:24.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.437+0000 7fac55679640 1 -- start start 2023-11-28T18:33:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5015dc50 0x7fac50147970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5015e090 0x7fac500c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac50148580 con 0x7fac50154300 2023-11-28T18:33:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac501496d0 con 0x7fac5015dc50 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac50146fa0 con 0x7fac5015e090 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50147430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42530/0 (socket says 172.21.15.7:42530) 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 -- 172.21.15.7:0/1460071666 learned_addr learned my addr 172.21.15.7:0/1460071666 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4e7fc640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5015dc50 0x7fac50147970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4f7fe640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5015e090 0x7fac500c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 -- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5015e090 msgr2=0x7fac500c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5015e090 0x7fac500c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 -- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5015dc50 msgr2=0x7fac50147970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5015dc50 0x7fac50147970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 -- 172.21.15.7:0/1460071666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac400094d0 con 0x7fac50154300 2023-11-28T18:33:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4f7fe640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5015e090 0x7fac500c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:33:24.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4e7fc640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5015dc50 0x7fac50147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:33:24.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac4effd640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50147430 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fac40019120 tx=0x7fac4001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:24.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac377fe640 1 -- 172.21.15.7:0/1460071666 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac400b1800 con 0x7fac50154300 2023-11-28T18:33:24.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac377fe640 1 -- 172.21.15.7:0/1460071666 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac400b1980 con 0x7fac50154300 2023-11-28T18:33:24.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac377fe640 1 -- 172.21.15.7:0/1460071666 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac400b1d30 con 0x7fac50154300 2023-11-28T18:33:24.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac50146cc0 con 0x7fac50154300 2023-11-28T18:33:24.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac501498d0 con 0x7fac50154300 2023-11-28T18:33:24.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.441+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5004e890 con 0x7fac50154300 2023-11-28T18:33:24.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.445+0000 7fac377fe640 1 -- 172.21.15.7:0/1460071666 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fac400bb3b0 con 0x7fac50154300 2023-11-28T18:33:24.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.445+0000 7fac377fe640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:33:24.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.445+0000 7fac377fe640 1 -- 172.21.15.7:0/1460071666 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fac4011ed70 con 0x7fac50154300 2023-11-28T18:33:24.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.445+0000 7fac4e7fc640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:33:24.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.445+0000 7fac4e7fc640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fac50153b90 tx=0x7fac44007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:33:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.449+0000 7fac377fe640 1 -- 172.21.15.7:0/1460071666 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fac400ebd90 con 0x7fac50154300 2023-11-28T18:33:24.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.661+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac500cbeb0 con 0x7fac50154300 2023-11-28T18:33:24.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac377fe640 1 -- 172.21.15.7:0/1460071666 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7fac400ef3b0 con 0x7fac50154300 2023-11-28T18:33:24.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:33:24.665 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:33:24.665 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:33:24.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 msgr2=0x7fac3c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:24.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fac50153b90 tx=0x7fac44007d50 comp rx=0 tx=0).stop 2023-11-28T18:33:24.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 msgr2=0x7fac50147430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:33:24.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50147430 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fac40019120 tx=0x7fac4001ad50 comp rx=0 tx=0).stop 2023-11-28T18:33:24.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 shutdown_connections 2023-11-28T18:33:24.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fac3c066900 0x7fac3c068dc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:24.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fac5015e090 0x7fac500c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:24.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac5015dc50 0x7fac50147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:24.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 --2- 172.21.15.7:0/1460071666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac50154300 0x7fac50147430 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:33:24.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 >> 172.21.15.7:0/1460071666 conn(0x7fac500662e0 msgr2=0x7fac500cd910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:33:24.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 shutdown_connections 2023-11-28T18:33:24.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:33:24.665+0000 7fac55679640 1 -- 172.21.15.7:0/1460071666 wait complete. 2023-11-28T18:33:24.688 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-11-28T18:33:24.688 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2023-11-28T18:33:30.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:33:30.588 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:34:00.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:34:00.861 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:34:30.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:34:30.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:34:49.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:34:49.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c14cea640 1 Processor -- start 2023-11-28T18:34:49.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c14cea640 1 -- start start 2023-11-28T18:34:49.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c14cea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c1015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:49.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c14cea640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8c10072450 0x7f8c100728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:49.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c14cea640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c100ca920 con 0x7f8c100cbb70 2023-11-28T18:34:49.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c14cea640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c10071d50 con 0x7f8c10072450 2023-11-28T18:34:49.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c14cea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c10072030 con 0x7f8c101542e0 2023-11-28T18:34:49.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c0ed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8c10072450 0x7f8c100728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:49.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c07fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8c100cbb70 0x7f8c100cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48798/0 (socket says 172.21.15.7:48798) 2023-11-28T18:34:49.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c07fff640 1 -- 172.21.15.7:0/374187902 learned_addr learned my addr 172.21.15.7:0/374187902 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:34:49.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c0e59c640 1 --2- 172.21.15.7:0/374187902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c1015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:49.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c0e59c640 1 -- 172.21.15.7:0/374187902 >> v2:172.21.15.7:3300/0 conn(0x7f8c10072450 msgr2=0x7f8c100728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:34:49.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c0e59c640 1 --2- 172.21.15.7:0/374187902 >> v2:172.21.15.7:3300/0 conn(0x7f8c10072450 0x7f8c100728e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:49.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c0e59c640 1 -- 172.21.15.7:0/374187902 >> v1:172.21.15.7:6789/0 conn(0x7f8c100cbb70 legacy=0x7f8c100cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:49.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.848+0000 7f8c0e59c640 1 -- 172.21.15.7:0/374187902 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1015e550 con 0x7f8c101542e0 2023-11-28T18:34:49.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c0e59c640 1 --2- 172.21.15.7:0/374187902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c1015e010 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8c000097c0 tx=0x7f8c0001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2374094ca04d7d47 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:49.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c0dd9b640 1 -- 172.21.15.7:0/374187902 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000ad640 con 0x7f8c101542e0 2023-11-28T18:34:49.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c0dd9b640 1 -- 172.21.15.7:0/374187902 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c000ad7a0 con 0x7f8c101542e0 2023-11-28T18:34:49.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c0dd9b640 1 -- 172.21.15.7:0/374187902 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000adb30 con 0x7f8c101542e0 2023-11-28T18:34:49.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c14cea640 1 -- 172.21.15.7:0/374187902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 msgr2=0x7f8c1015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:49.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c14cea640 1 --2- 172.21.15.7:0/374187902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c1015e010 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8c000097c0 tx=0x7f8c0001c480 comp rx=0 tx=0).stop 2023-11-28T18:34:49.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c14cea640 1 -- 172.21.15.7:0/374187902 shutdown_connections 2023-11-28T18:34:49.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c14cea640 1 --2- 172.21.15.7:0/374187902 >> v2:172.21.15.7:3300/0 conn(0x7f8c10072450 0x7f8c100728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:49.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c14cea640 1 --2- 172.21.15.7:0/374187902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c1015e010 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:49.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c14cea640 1 -- 172.21.15.7:0/374187902 >> 172.21.15.7:0/374187902 conn(0x7f8c100717e0 msgr2=0x7f8c100662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:49.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c14cea640 1 -- 172.21.15.7:0/374187902 shutdown_connections 2023-11-28T18:34:49.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.852+0000 7f8c14cea640 1 -- 172.21.15.7:0/374187902 wait complete. 2023-11-28T18:34:49.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.856+0000 7f8c14cea640 1 Processor -- start 2023-11-28T18:34:49.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.856+0000 7f8c14cea640 1 -- start start 2023-11-28T18:34:49.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c10072450 0x7f8c101f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:49.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c100cbb70 0x7f8c1015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c100c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c100c7950 con 0x7f8c10072450 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c101f6130 con 0x7f8c100cbb70 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c100d0180 con 0x7f8c101542e0 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c10072450 0x7f8c101f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c10072450 0x7f8c101f6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55622/0 (socket says 172.21.15.7:55622) 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 -- 172.21.15.7:0/3738732265 learned_addr learned my addr 172.21.15.7:0/3738732265 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0ed9d640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c100c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:49.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c07fff640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c100cbb70 0x7f8c1015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:49.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 -- 172.21.15.7:0/3738732265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 msgr2=0x7f8c100c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:49.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c100c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:49.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 -- 172.21.15.7:0/3738732265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c100cbb70 msgr2=0x7f8c1015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:49.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c100cbb70 0x7f8c1015c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:49.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 -- 172.21.15.7:0/3738732265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c00009420 con 0x7f8c10072450 2023-11-28T18:34:49.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c0e59c640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c10072450 0x7f8c101f6550 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8c0001ab30 tx=0x7f8c00019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:49.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c06ffd640 1 -- 172.21.15.7:0/3738732265 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000b8620 con 0x7f8c10072450 2023-11-28T18:34:49.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c06ffd640 1 -- 172.21.15.7:0/3738732265 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c000a6040 con 0x7f8c10072450 2023-11-28T18:34:49.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c100d06b0 con 0x7f8c10072450 2023-11-28T18:34:49.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c06ffd640 1 -- 172.21.15.7:0/3738732265 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000b89c0 con 0x7f8c10072450 2023-11-28T18:34:49.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c101f5a70 con 0x7f8c10072450 2023-11-28T18:34:49.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.860+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1006c670 con 0x7f8c10072450 2023-11-28T18:34:49.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.864+0000 7f8c06ffd640 1 -- 172.21.15.7:0/3738732265 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8c000b4040 con 0x7f8c10072450 2023-11-28T18:34:49.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.864+0000 7f8c06ffd640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bf00668d0 0x7f8bf0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:49.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.864+0000 7f8c06ffd640 1 -- 172.21.15.7:0/3738732265 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f8c0011ea30 con 0x7f8c10072450 2023-11-28T18:34:49.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.864+0000 7f8c07fff640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bf00668d0 0x7f8bf0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:49.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.864+0000 7f8c07fff640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bf00668d0 0x7f8bf0068d90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8c10149e40 tx=0x7f8bf8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:49.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:49.868+0000 7f8c06ffd640 1 -- 172.21.15.7:0/3738732265 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8c000eb9d0 con 0x7f8c10072450 2023-11-28T18:34:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.100+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c1004e890 con 0x7f8c10072450 2023-11-28T18:34:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.100+0000 7f8c06ffd640 1 -- 172.21.15.7:0/3738732265 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f8c000b2070 con 0x7f8c10072450 2023-11-28T18:34:50.104 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:34:50.104 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:34:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:34:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bf00668d0 msgr2=0x7f8bf0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bf00668d0 0x7f8bf0068d90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8c10149e40 tx=0x7f8bf8009040 comp rx=0 tx=0).stop 2023-11-28T18:34:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c10072450 msgr2=0x7f8c101f6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c10072450 0x7f8c101f6550 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8c0001ab30 tx=0x7f8c00019220 comp rx=0 tx=0).stop 2023-11-28T18:34:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 shutdown_connections 2023-11-28T18:34:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8bf00668d0 0x7f8bf0068d90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8c101542e0 0x7f8c100c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c100cbb70 0x7f8c1015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 --2- 172.21.15.7:0/3738732265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c10072450 0x7f8c101f6550 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 >> 172.21.15.7:0/3738732265 conn(0x7f8c100717e0 msgr2=0x7f8c10147f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 shutdown_connections 2023-11-28T18:34:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.104+0000 7f8c14cea640 1 -- 172.21.15.7:0/3738732265 wait complete. 2023-11-28T18:34:50.127 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-11-28T18:34:50.128 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-11-28T18:34:50.128 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:34:50.143 INFO:tasks.ceph.mds.e:Stopped 2023-11-28T18:34:50.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:34:50.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.336+0000 7efe5a8e3640 1 Processor -- start 2023-11-28T18:34:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.336+0000 7efe5a8e3640 1 -- start start 2023-11-28T18:34:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.336+0000 7efe5a8e3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efe541542e0 0x7efe5415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:50.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.336+0000 7efe5a8e3640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7efe540cbb70 0x7efe540cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:50.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.336+0000 7efe5a8e3640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe540ca920 con 0x7efe540cc810 2023-11-28T18:34:50.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.336+0000 7efe5a8e3640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe54071d50 con 0x7efe541542e0 2023-11-28T18:34:50.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.336+0000 7efe5a8e3640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe54072030 con 0x7efe540cbb70 2023-11-28T18:34:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.340+0000 7efe53fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efe541542e0 0x7efe5415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.336+0000 7efe58e80640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efe540cc810 0x7efe54072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48802/0 (socket says 172.21.15.7:48802) 2023-11-28T18:34:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.340+0000 7efe58e80640 1 -- 172.21.15.7:0/4078163990 learned_addr learned my addr 172.21.15.7:0/4078163990 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:34:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.340+0000 7efe537fe640 1 --2- 172.21.15.7:0/4078163990 >> v2:172.21.15.193:3300/0 conn(0x7efe540cbb70 0x7efe540cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.340+0000 7efe53fff640 1 -- 172.21.15.7:0/4078163990 >> v2:172.21.15.193:3300/0 conn(0x7efe540cbb70 msgr2=0x7efe540cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:34:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.340+0000 7efe53fff640 1 --2- 172.21.15.7:0/4078163990 >> v2:172.21.15.193:3300/0 conn(0x7efe540cbb70 0x7efe540cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.340+0000 7efe53fff640 1 -- 172.21.15.7:0/4078163990 >> v1:172.21.15.7:6789/0 conn(0x7efe540cc810 legacy=0x7efe54072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.340+0000 7efe53fff640 1 -- 172.21.15.7:0/4078163990 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5415e550 con 0x7efe541542e0 2023-11-28T18:34:50.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe53fff640 1 --2- 172.21.15.7:0/4078163990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe5415e010 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efe540ca8b0 tx=0x7efe4401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9538743f85d07d76 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:50.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe52ffd640 1 -- 172.21.15.7:0/4078163990 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe440ab600 con 0x7efe541542e0 2023-11-28T18:34:50.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe52ffd640 1 -- 172.21.15.7:0/4078163990 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe440a7050 con 0x7efe541542e0 2023-11-28T18:34:50.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe52ffd640 1 -- 172.21.15.7:0/4078163990 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe440ab980 con 0x7efe541542e0 2023-11-28T18:34:50.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe5a8e3640 1 -- 172.21.15.7:0/4078163990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 msgr2=0x7efe5415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:50.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/4078163990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe5415e010 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efe540ca8b0 tx=0x7efe4401c460 comp rx=0 tx=0).stop 2023-11-28T18:34:50.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe5a8e3640 1 -- 172.21.15.7:0/4078163990 shutdown_connections 2023-11-28T18:34:50.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/4078163990 >> v2:172.21.15.193:3300/0 conn(0x7efe540cbb70 0x7efe540cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/4078163990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe5415e010 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe5a8e3640 1 -- 172.21.15.7:0/4078163990 >> 172.21.15.7:0/4078163990 conn(0x7efe540717e0 msgr2=0x7efe540662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe5a8e3640 1 -- 172.21.15.7:0/4078163990 shutdown_connections 2023-11-28T18:34:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.344+0000 7efe5a8e3640 1 -- 172.21.15.7:0/4078163990 wait complete. 2023-11-28T18:34:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 Processor -- start 2023-11-28T18:34:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 -- start start 2023-11-28T18:34:50.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe540cbb70 0x7efe541f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe540cc810 0x7efe54142110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe54142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe54159a00 con 0x7efe541542e0 2023-11-28T18:34:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe540ca540 con 0x7efe540cbb70 2023-11-28T18:34:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5415df10 con 0x7efe540cc810 2023-11-28T18:34:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe54142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe53fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe540cbb70 0x7efe541f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe537fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe540cc810 0x7efe54142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe54142650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55646/0 (socket says 172.21.15.7:55646) 2023-11-28T18:34:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 -- 172.21.15.7:0/1204455726 learned_addr learned my addr 172.21.15.7:0/1204455726 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:34:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe53fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe540cbb70 0x7efe541f5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:47284/0 (socket says 172.21.15.7:47284) 2023-11-28T18:34:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 -- 172.21.15.7:0/1204455726 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe540cc810 msgr2=0x7efe54142110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe540cc810 0x7efe54142110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 -- 172.21.15.7:0/1204455726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe540cbb70 msgr2=0x7efe541f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe540cbb70 0x7efe541f5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 -- 172.21.15.7:0/1204455726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe44009420 con 0x7efe541542e0 2023-11-28T18:34:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe58e80640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe54142650 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00cbc0 tx=0x7efe4c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe50ff9640 1 -- 172.21.15.7:0/1204455726 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4c00ad40 con 0x7efe541542e0 2023-11-28T18:34:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe50ff9640 1 -- 172.21.15.7:0/1204455726 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4c012400 con 0x7efe541542e0 2023-11-28T18:34:50.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe50ff9640 1 -- 172.21.15.7:0/1204455726 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4c00aea0 con 0x7efe541542e0 2023-11-28T18:34:50.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.348+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5415e1e0 con 0x7efe541542e0 2023-11-28T18:34:50.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.352+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe541f6700 con 0x7efe541542e0 2023-11-28T18:34:50.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.352+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe54159bb0 con 0x7efe541542e0 2023-11-28T18:34:50.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.352+0000 7efe50ff9640 1 -- 172.21.15.7:0/1204455726 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7efe4c007070 con 0x7efe541542e0 2023-11-28T18:34:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.352+0000 7efe50ff9640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efe40066900 0x7efe40068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.352+0000 7efe50ff9640 1 -- 172.21.15.7:0/1204455726 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7efe4c084570 con 0x7efe541542e0 2023-11-28T18:34:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.352+0000 7efe53fff640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efe40066900 0x7efe40068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.356+0000 7efe53fff640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efe40066900 0x7efe40068dc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efe440ad000 tx=0x7efe44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.356+0000 7efe50ff9640 1 -- 172.21.15.7:0/1204455726 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efe4c051510 con 0x7efe541542e0 2023-11-28T18:34:50.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe5406c670 con 0x7efe541542e0 2023-11-28T18:34:50.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe50ff9640 1 -- 172.21.15.7:0/1204455726 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7efe4c017090 con 0x7efe541542e0 2023-11-28T18:34:50.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:34:50.590 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:34:50.590 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:34:50.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efe40066900 msgr2=0x7efe40068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efe40066900 0x7efe40068dc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efe440ad000 tx=0x7efe44008040 comp rx=0 tx=0).stop 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 msgr2=0x7efe54142650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe54142650 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00cbc0 tx=0x7efe4c00a430 comp rx=0 tx=0).stop 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 shutdown_connections 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efe40066900 0x7efe40068dc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe541542e0 0x7efe54142650 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe540cc810 0x7efe54142110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 --2- 172.21.15.7:0/1204455726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe540cbb70 0x7efe541f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 >> 172.21.15.7:0/1204455726 conn(0x7efe540717e0 msgr2=0x7efe54151b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 shutdown_connections 2023-11-28T18:34:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:50.588+0000 7efe5a8e3640 1 -- 172.21.15.7:0/1204455726 wait complete. 2023-11-28T18:34:50.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-11-28T18:34:52.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:34:52.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.776+0000 7fc1c55d1640 1 Processor -- start 2023-11-28T18:34:52.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.776+0000 7fc1c55d1640 1 -- start start 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.776+0000 7fc1c55d1640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc1c00b3900 0x7fc1c00cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.776+0000 7fc1c55d1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1c006c210 0x7fc1c00cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1c55d1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c006a5f0 con 0x7fc1c0154080 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1c55d1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c006d8a0 con 0x7fc1c006c210 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1c55d1640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c0159e80 con 0x7fc1c00b3900 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1bffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1c006c210 0x7fc1c00cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1be7fc640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc1c00b3900 0x7fc1c00cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1beffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc1c0154080 0x7fc1c015a5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:37542/0 (socket says 172.21.15.7:37542) 2023-11-28T18:34:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1beffd640 1 -- 172.21.15.7:0/3659005165 learned_addr learned my addr 172.21.15.7:0/3659005165 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:34:52.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1bffff640 1 -- 172.21.15.7:0/3659005165 >> v2:172.21.15.193:3300/0 conn(0x7fc1c00b3900 msgr2=0x7fc1c00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:34:52.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1bffff640 1 --2- 172.21.15.7:0/3659005165 >> v2:172.21.15.193:3300/0 conn(0x7fc1c00b3900 0x7fc1c00cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:52.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1bffff640 1 -- 172.21.15.7:0/3659005165 >> v1:172.21.15.193:6789/0 conn(0x7fc1c0154080 legacy=0x7fc1c015a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:52.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1bffff640 1 -- 172.21.15.7:0/3659005165 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c015a160 con 0x7fc1c006c210 2023-11-28T18:34:52.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1bdffb640 1 -- 172.21.15.7:0/3659005165 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875144833 0 0) 0x7fc1c006a5f0 con 0x7fc1c0154080 2023-11-28T18:34:52.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.780+0000 7fc1bffff640 1 --2- 172.21.15.7:0/3659005165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c00cdce0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc1c006ddc0 tx=0x7fc1b801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc942e63d6956388 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1bdffb640 1 -- 172.21.15.7:0/3659005165 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80a58c0 con 0x7fc1c006c210 2023-11-28T18:34:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1bdffb640 1 -- 172.21.15.7:0/3659005165 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b80a5a20 con 0x7fc1c006c210 2023-11-28T18:34:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1bdffb640 1 -- 172.21.15.7:0/3659005165 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80a5db0 con 0x7fc1c006c210 2023-11-28T18:34:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- 172.21.15.7:0/3659005165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 msgr2=0x7fc1c00cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/3659005165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c00cdce0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc1c006ddc0 tx=0x7fc1b801c910 comp rx=0 tx=0).stop 2023-11-28T18:34:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- 172.21.15.7:0/3659005165 shutdown_connections 2023-11-28T18:34:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/3659005165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c00cdce0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/3659005165 >> v2:172.21.15.193:3300/0 conn(0x7fc1c00b3900 0x7fc1c00cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- 172.21.15.7:0/3659005165 >> 172.21.15.7:0/3659005165 conn(0x7fc1c008d3f0 msgr2=0x7fc1c00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- 172.21.15.7:0/3659005165 shutdown_connections 2023-11-28T18:34:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- 172.21.15.7:0/3659005165 wait complete. 2023-11-28T18:34:52.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 Processor -- start 2023-11-28T18:34:52.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- start start 2023-11-28T18:34:52.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c01f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1c00b3900 0x7fc1c013f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1c0154080 0x7fc1c0140c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0158200 con 0x7fc1c006c210 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0146640 con 0x7fc1c00b3900 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1c55d1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01f66d0 con 0x7fc1c0154080 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.784+0000 7fc1beffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c01f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1beffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c01f57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55674/0 (socket says 172.21.15.7:55674) 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1beffd640 1 -- 172.21.15.7:0/1268506627 learned_addr learned my addr 172.21.15.7:0/1268506627 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:34:52.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1be7fc640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1c00b3900 0x7fc1c013f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:52.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1bffff640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1c0154080 0x7fc1c0140c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:52.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1beffd640 1 -- 172.21.15.7:0/1268506627 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1c0154080 msgr2=0x7fc1c0140c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:52.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1beffd640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1c0154080 0x7fc1c0140c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:52.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1beffd640 1 -- 172.21.15.7:0/1268506627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1c00b3900 msgr2=0x7fc1c013f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:52.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1beffd640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1c00b3900 0x7fc1c013f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:52.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1beffd640 1 -- 172.21.15.7:0/1268506627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8009a70 con 0x7fc1c006c210 2023-11-28T18:34:52.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1beffd640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c01f57f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c480 tx=0x7fc1b000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:52.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1a77fe640 1 -- 172.21.15.7:0/1268506627 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b00074e0 con 0x7fc1c006c210 2023-11-28T18:34:52.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1a77fe640 1 -- 172.21.15.7:0/1268506627 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b0007660 con 0x7fc1c006c210 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1a77fe640 1 -- 172.21.15.7:0/1268506627 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0004a30 con 0x7fc1c006c210 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c01f5e30 con 0x7fc1c006c210 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c01f63f0 con 0x7fc1c006c210 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c0158340 con 0x7fc1c006c210 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.788+0000 7fc1a77fe640 1 -- 172.21.15.7:0/1268506627 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc1b0004b90 con 0x7fc1c006c210 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.792+0000 7fc1a77fe640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1ac0668d0 0x7fc1ac068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.792+0000 7fc1a77fe640 1 -- 172.21.15.7:0/1268506627 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fc1b00837d0 con 0x7fc1c006c210 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.792+0000 7fc1be7fc640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1ac0668d0 0x7fc1ac068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:52.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.792+0000 7fc1be7fc640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1ac0668d0 0x7fc1ac068d90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009b60 tx=0x7fc1b4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:52.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:52.796+0000 7fc1a77fe640 1 -- 172.21.15.7:0/1268506627 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc1c0158340 con 0x7fc1c006c210 2023-11-28T18:34:53.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1c0158340 con 0x7fc1c006c210 2023-11-28T18:34:53.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1a77fe640 1 -- 172.21.15.7:0/1268506627 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7fc1c0158340 con 0x7fc1c006c210 2023-11-28T18:34:53.046 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:34:53.047 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:34:53.047 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:34:53.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1ac0668d0 msgr2=0x7fc1ac068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:53.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1ac0668d0 0x7fc1ac068d90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009b60 tx=0x7fc1b4009190 comp rx=0 tx=0).stop 2023-11-28T18:34:53.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 msgr2=0x7fc1c01f57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c01f57f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c480 tx=0x7fc1b000c940 comp rx=0 tx=0).stop 2023-11-28T18:34:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 shutdown_connections 2023-11-28T18:34:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1ac0668d0 0x7fc1ac068d90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc1c0154080 0x7fc1c0140c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1c00b3900 0x7fc1c013f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 --2- 172.21.15.7:0/1268506627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c006c210 0x7fc1c01f57f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.044+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 >> 172.21.15.7:0/1268506627 conn(0x7fc1c008d3f0 msgr2=0x7fc1c00cf740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.048+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 shutdown_connections 2023-11-28T18:34:53.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:53.048+0000 7fc1c55d1640 1 -- 172.21.15.7:0/1268506627 wait complete. 2023-11-28T18:34:53.073 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-11-28T18:34:55.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:34:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.216+0000 7f52febf5640 1 Processor -- start 2023-11-28T18:34:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52febf5640 1 -- start start 2023-11-28T18:34:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52febf5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f8154300 0x7f52f8159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52febf5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f52f815e090 0x7f52f80ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52febf5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f815d490 con 0x7f52f815dc50 2023-11-28T18:34:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52febf5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f815a480 con 0x7f52f815e090 2023-11-28T18:34:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52febf5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f815a760 con 0x7f52f8154300 2023-11-28T18:34:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52f7fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f52f815dc50 0x7f52f80cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48986/0 (socket says 172.21.15.7:48986) 2023-11-28T18:34:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52f7fff640 1 -- 172.21.15.7:0/1171170253 learned_addr learned my addr 172.21.15.7:0/1171170253 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:34:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52fc991640 1 --2- 172.21.15.7:0/1171170253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f8154300 0x7f52f8159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52fd192640 1 --2- 172.21.15.7:0/1171170253 >> v2:172.21.15.193:3300/0 conn(0x7f52f815e090 0x7f52f80ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52f77fe640 1 -- 172.21.15.7:0/1171170253 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455856964 0 0) 0x7f52f815d490 con 0x7f52f815dc50 2023-11-28T18:34:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52f77fe640 1 -- 172.21.15.7:0/1171170253 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e8003530 con 0x7f52f815dc50 2023-11-28T18:34:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52fc991640 1 -- 172.21.15.7:0/1171170253 >> v2:172.21.15.193:3300/0 conn(0x7f52f815e090 msgr2=0x7f52f80ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:34:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52fc991640 1 --2- 172.21.15.7:0/1171170253 >> v2:172.21.15.193:3300/0 conn(0x7f52f815e090 0x7f52f80ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52fc991640 1 -- 172.21.15.7:0/1171170253 >> v1:172.21.15.7:6789/0 conn(0x7f52f815dc50 legacy=0x7f52f80cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.220+0000 7f52fc991640 1 -- 172.21.15.7:0/1171170253 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f815aa40 con 0x7f52f8154300 2023-11-28T18:34:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52fc991640 1 --2- 172.21.15.7:0/1171170253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f8154300 0x7f52f8159e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0097c0 tx=0x7f52ec01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=296e17994fe6cb05 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52f77fe640 1 -- 172.21.15.7:0/1171170253 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0ad660 con 0x7f52f8154300 2023-11-28T18:34:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52f77fe640 1 -- 172.21.15.7:0/1171170253 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec0ad7c0 con 0x7f52f8154300 2023-11-28T18:34:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52f77fe640 1 -- 172.21.15.7:0/1171170253 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0adb30 con 0x7f52f8154300 2023-11-28T18:34:55.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52febf5640 1 -- 172.21.15.7:0/1171170253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f8154300 msgr2=0x7f52f8159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:55.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52febf5640 1 --2- 172.21.15.7:0/1171170253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f8154300 0x7f52f8159e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0097c0 tx=0x7f52ec01c450 comp rx=0 tx=0).stop 2023-11-28T18:34:55.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52febf5640 1 -- 172.21.15.7:0/1171170253 shutdown_connections 2023-11-28T18:34:55.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52febf5640 1 --2- 172.21.15.7:0/1171170253 >> v2:172.21.15.193:3300/0 conn(0x7f52f815e090 0x7f52f80ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52febf5640 1 --2- 172.21.15.7:0/1171170253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f8154300 0x7f52f8159e40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52febf5640 1 -- 172.21.15.7:0/1171170253 >> 172.21.15.7:0/1171170253 conn(0x7f52f80662e0 msgr2=0x7f52f80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:55.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52febf5640 1 -- 172.21.15.7:0/1171170253 shutdown_connections 2023-11-28T18:34:55.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.224+0000 7f52febf5640 1 -- 172.21.15.7:0/1171170253 wait complete. 2023-11-28T18:34:55.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 Processor -- start 2023-11-28T18:34:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 -- start start 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f8154300 0x7f52f81f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f815dc50 0x7f52f81f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f815e090 0x7f52f814a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f81f60c0 con 0x7f52f8154300 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f81f29a0 con 0x7f52f815e090 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f815d1a0 con 0x7f52f815dc50 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f8154300 0x7f52f81f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f8154300 0x7f52f81f6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60128/0 (socket says 172.21.15.7:60128) 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 -- 172.21.15.7:0/3308057675 learned_addr learned my addr 172.21.15.7:0/3308057675 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:34:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52f7fff640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f815dc50 0x7f52f81f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fd192640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f815e090 0x7f52f814a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 -- 172.21.15.7:0/3308057675 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f815dc50 msgr2=0x7f52f81f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f815dc50 0x7f52f81f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 -- 172.21.15.7:0/3308057675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f815e090 msgr2=0x7f52f814a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f815e090 0x7f52f814a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 -- 172.21.15.7:0/3308057675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec009420 con 0x7f52f8154300 2023-11-28T18:34:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52fc991640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f8154300 0x7f52f81f6550 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f52ec01a080 tx=0x7f52ec0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:55.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52f57fa640 1 -- 172.21.15.7:0/3308057675 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0adb30 con 0x7f52f8154300 2023-11-28T18:34:55.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52f57fa640 1 -- 172.21.15.7:0/3308057675 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec0adc90 con 0x7f52f8154300 2023-11-28T18:34:55.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52f57fa640 1 -- 172.21.15.7:0/3308057675 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec006e20 con 0x7f52f8154300 2023-11-28T18:34:55.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f8153800 con 0x7f52f8154300 2023-11-28T18:34:55.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.228+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f81f5a70 con 0x7f52f8154300 2023-11-28T18:34:55.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.232+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f804e890 con 0x7f52f8154300 2023-11-28T18:34:55.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.232+0000 7f52f57fa640 1 -- 172.21.15.7:0/3308057675 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f52ec0b4050 con 0x7f52f8154300 2023-11-28T18:34:55.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.232+0000 7f52f57fa640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52dc0668d0 0x7f52dc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:55.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.232+0000 7f52f57fa640 1 -- 172.21.15.7:0/3308057675 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f52ec11eb20 con 0x7f52f8154300 2023-11-28T18:34:55.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.232+0000 7f52f7fff640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52dc0668d0 0x7f52dc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:55.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.232+0000 7f52f7fff640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52dc0668d0 0x7f52dc068d90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f52f8151a90 tx=0x7f52e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:55.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.236+0000 7f52f57fa640 1 -- 172.21.15.7:0/3308057675 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f52ec0ebbc0 con 0x7f52f8154300 2023-11-28T18:34:55.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.476+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52f80caf50 con 0x7f52f8154300 2023-11-28T18:34:55.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.476+0000 7f52f57fa640 1 -- 172.21.15.7:0/3308057675 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f52ec0b2090 con 0x7f52f8154300 2023-11-28T18:34:55.478 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:34:55.479 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:34:55.479 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52dc0668d0 msgr2=0x7f52dc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52dc0668d0 0x7f52dc068d90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f52f8151a90 tx=0x7f52e4009040 comp rx=0 tx=0).stop 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f8154300 msgr2=0x7f52f81f6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f8154300 0x7f52f81f6550 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f52ec01a080 tx=0x7f52ec0191f0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 shutdown_connections 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52dc0668d0 0x7f52dc068d90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f815e090 0x7f52f814a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f815dc50 0x7f52f81f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 --2- 172.21.15.7:0/3308057675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f8154300 0x7f52f81f6550 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 >> 172.21.15.7:0/3308057675 conn(0x7f52f80662e0 msgr2=0x7f52f80c56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:55.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 shutdown_connections 2023-11-28T18:34:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:55.480+0000 7f52febf5640 1 -- 172.21.15.7:0/3308057675 wait complete. 2023-11-28T18:34:55.501 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-11-28T18:34:57.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:34:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e8aac9640 1 Processor -- start 2023-11-28T18:34:57.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e8aac9640 1 -- start start 2023-11-28T18:34:57.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e8aac9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e84143130 0x7f8e840c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:57.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e8aac9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8406f380 0x7f8e840662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:57.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e8aac9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8415e050 con 0x7f8e84066820 2023-11-28T18:34:57.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e8aac9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e84159fc0 con 0x7f8e84143130 2023-11-28T18:34:57.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e8aac9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8415a2a0 con 0x7f8e8406f380 2023-11-28T18:34:57.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e88865640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e84143130 0x7f8e840c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:57.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e89066640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8e84066820 0x7f8e84071140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49002/0 (socket says 172.21.15.7:49002) 2023-11-28T18:34:57.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e89066640 1 -- 172.21.15.7:0/471059120 learned_addr learned my addr 172.21.15.7:0/471059120 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:34:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e83fff640 1 --2- 172.21.15.7:0/471059120 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8406f380 0x7f8e840662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e88865640 1 -- 172.21.15.7:0/471059120 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8406f380 msgr2=0x7f8e840662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:34:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e88865640 1 --2- 172.21.15.7:0/471059120 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8406f380 0x7f8e840662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e88865640 1 -- 172.21.15.7:0/471059120 >> v1:172.21.15.7:6789/0 conn(0x7f8e84066820 legacy=0x7f8e84071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e88865640 1 -- 172.21.15.7:0/471059120 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8415a580 con 0x7f8e84143130 2023-11-28T18:34:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.656+0000 7f8e83fff640 1 --2- 172.21.15.7:0/471059120 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8406f380 0x7f8e840662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:34:57.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e88865640 1 --2- 172.21.15.7:0/471059120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e84143130 0x7f8e840c7e10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009d90 tx=0x7f8e7c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a8c20b1c156bc7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e837fe640 1 -- 172.21.15.7:0/471059120 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0af600 con 0x7f8e84143130 2023-11-28T18:34:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e837fe640 1 -- 172.21.15.7:0/471059120 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c0ab050 con 0x7f8e84143130 2023-11-28T18:34:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e837fe640 1 -- 172.21.15.7:0/471059120 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0af980 con 0x7f8e84143130 2023-11-28T18:34:57.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e8aac9640 1 -- 172.21.15.7:0/471059120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e84143130 msgr2=0x7f8e840c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:57.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/471059120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e84143130 0x7f8e840c7e10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009d90 tx=0x7f8e7c01f130 comp rx=0 tx=0).stop 2023-11-28T18:34:57.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e8aac9640 1 -- 172.21.15.7:0/471059120 shutdown_connections 2023-11-28T18:34:57.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/471059120 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8406f380 0x7f8e840662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/471059120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e84143130 0x7f8e840c7e10 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e8aac9640 1 -- 172.21.15.7:0/471059120 >> 172.21.15.7:0/471059120 conn(0x7f8e840cc490 msgr2=0x7f8e840cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:57.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e8aac9640 1 -- 172.21.15.7:0/471059120 shutdown_connections 2023-11-28T18:34:57.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.660+0000 7f8e8aac9640 1 -- 172.21.15.7:0/471059120 wait complete. 2023-11-28T18:34:57.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e8aac9640 1 Processor -- start 2023-11-28T18:34:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e8aac9640 1 -- start start 2023-11-28T18:34:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e8aac9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e84066820 0x7f8e84155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e8aac9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8406f380 0x7f8e841ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e8aac9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e84143130 0x7f8e8413e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e8aac9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e841e3930 con 0x7f8e8406f380 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e8aac9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e84154b20 con 0x7f8e84066820 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e8aac9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e84141900 con 0x7f8e84143130 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e83fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8406f380 0x7f8e841ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e83fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8406f380 0x7f8e841ef600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60140/0 (socket says 172.21.15.7:60140) 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e83fff640 1 -- 172.21.15.7:0/3498891193 learned_addr learned my addr 172.21.15.7:0/3498891193 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e88865640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e84066820 0x7f8e84155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e89066640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e84143130 0x7f8e8413e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e83fff640 1 -- 172.21.15.7:0/3498891193 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e84143130 msgr2=0x7f8e8413e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e83fff640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e84143130 0x7f8e8413e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e83fff640 1 -- 172.21.15.7:0/3498891193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e84066820 msgr2=0x7f8e84155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e83fff640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e84066820 0x7f8e84155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.664+0000 7f8e83fff640 1 -- 172.21.15.7:0/3498891193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c009a70 con 0x7f8e8406f380 2023-11-28T18:34:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e83fff640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8406f380 0x7f8e841ef600 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400e810 tx=0x7f8e7400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e817fa640 1 -- 172.21.15.7:0/3498891193 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7400c8d0 con 0x7f8e8406f380 2023-11-28T18:34:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e817fa640 1 -- 172.21.15.7:0/3498891193 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7400ca30 con 0x7f8e8406f380 2023-11-28T18:34:57.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e817fa640 1 -- 172.21.15.7:0/3498891193 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7400cdc0 con 0x7f8e8406f380 2023-11-28T18:34:57.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e841558a0 con 0x7f8e8406f380 2023-11-28T18:34:57.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e84155530 con 0x7f8e8406f380 2023-11-28T18:34:57.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8404e890 con 0x7f8e8406f380 2023-11-28T18:34:57.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e817fa640 1 -- 172.21.15.7:0/3498891193 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8e74013410 con 0x7f8e8406f380 2023-11-28T18:34:57.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e817fa640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e640668d0 0x7f8e64068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:34:57.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.668+0000 7f8e817fa640 1 -- 172.21.15.7:0/3498891193 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f8e74084400 con 0x7f8e8406f380 2023-11-28T18:34:57.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.672+0000 7f8e88865640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e640668d0 0x7f8e64068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:34:57.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.672+0000 7f8e88865640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e640668d0 0x7f8e64068d90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0b1000 tx=0x7f8e7c0aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:34:57.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.676+0000 7f8e817fa640 1 -- 172.21.15.7:0/3498891193 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8e740514c0 con 0x7f8e8406f380 2023-11-28T18:34:57.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.884+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e84066600 con 0x7f8e8406f380 2023-11-28T18:34:57.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e817fa640 1 -- 172.21.15.7:0/3498891193 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f8e74054ae0 con 0x7f8e8406f380 2023-11-28T18:34:57.890 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:34:57.890 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:34:57.891 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:34:57.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e640668d0 msgr2=0x7f8e64068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:57.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e640668d0 0x7f8e64068d90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0b1000 tx=0x7f8e7c0aa000 comp rx=0 tx=0).stop 2023-11-28T18:34:57.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8406f380 msgr2=0x7f8e841ef600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8406f380 0x7f8e841ef600 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400e810 tx=0x7f8e7400ecd0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 shutdown_connections 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e640668d0 0x7f8e64068d90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e84143130 0x7f8e8413e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8406f380 0x7f8e841ef600 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 --2- 172.21.15.7:0/3498891193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e84066820 0x7f8e84155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 >> 172.21.15.7:0/3498891193 conn(0x7f8e840cc490 msgr2=0x7f8e8415b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 shutdown_connections 2023-11-28T18:34:57.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:34:57.888+0000 7f8e8aac9640 1 -- 172.21.15.7:0/3498891193 wait complete. 2023-11-28T18:34:57.913 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-11-28T18:34:59.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.060+0000 7fe5e6384640 1 Processor -- start 2023-11-28T18:35:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.060+0000 7fe5e6384640 1 -- start start 2023-11-28T18:35:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e6384640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe5e0159c90 0x7fe5e015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e6384640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5e015def0 0x7fe5e00ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:00.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e6384640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e015d240 con 0x7fe5e0153ee0 2023-11-28T18:35:00.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e6384640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e015d520 con 0x7fe5e015def0 2023-11-28T18:35:00.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e6384640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e00cbd10 con 0x7fe5e0159c90 2023-11-28T18:35:00.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e5b83640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5e015def0 0x7fe5e00ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e5382640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe5e0153ee0 0x7fe5e00cb600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49016/0 (socket says 172.21.15.7:49016) 2023-11-28T18:35:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e5382640 1 -- 172.21.15.7:0/969131525 learned_addr learned my addr 172.21.15.7:0/969131525 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:35:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e4b81640 1 --2- 172.21.15.7:0/969131525 >> v2:172.21.15.193:3300/0 conn(0x7fe5e0159c90 0x7fe5e015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e5b83640 1 -- 172.21.15.7:0/969131525 >> v2:172.21.15.193:3300/0 conn(0x7fe5e0159c90 msgr2=0x7fe5e015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:00.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e5b83640 1 --2- 172.21.15.7:0/969131525 >> v2:172.21.15.193:3300/0 conn(0x7fe5e0159c90 0x7fe5e015d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e5b83640 1 -- 172.21.15.7:0/969131525 >> v1:172.21.15.7:6789/0 conn(0x7fe5e0153ee0 legacy=0x7fe5e00cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:00.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.064+0000 7fe5e5b83640 1 -- 172.21.15.7:0/969131525 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00c4bc0 con 0x7fe5e015def0 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e5b83640 1 --2- 172.21.15.7:0/969131525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e00ca330 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00b58a0 tx=0x7fe5d801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581f3d27755444e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5d7fff640 1 -- 172.21.15.7:0/969131525 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80b0440 con 0x7fe5e015def0 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5d7fff640 1 -- 172.21.15.7:0/969131525 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d80b05a0 con 0x7fe5e015def0 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5d7fff640 1 -- 172.21.15.7:0/969131525 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80b0930 con 0x7fe5e015def0 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e6384640 1 -- 172.21.15.7:0/969131525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 msgr2=0x7fe5e00ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e6384640 1 --2- 172.21.15.7:0/969131525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e00ca330 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00b58a0 tx=0x7fe5d801f0f0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e6384640 1 -- 172.21.15.7:0/969131525 shutdown_connections 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e6384640 1 --2- 172.21.15.7:0/969131525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e00ca330 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e6384640 1 --2- 172.21.15.7:0/969131525 >> v2:172.21.15.193:3300/0 conn(0x7fe5e0159c90 0x7fe5e015d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e6384640 1 -- 172.21.15.7:0/969131525 >> 172.21.15.7:0/969131525 conn(0x7fe5e0071e20 msgr2=0x7fe5e0072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:00.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e6384640 1 -- 172.21.15.7:0/969131525 shutdown_connections 2023-11-28T18:35:00.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.068+0000 7fe5e6384640 1 -- 172.21.15.7:0/969131525 wait complete. 2023-11-28T18:35:00.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 Processor -- start 2023-11-28T18:35:00.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 -- start start 2023-11-28T18:35:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe5e0153ee0 0x7fe5e0149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e0159c90 0x7fe5e00c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e0144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00c7340 con 0x7fe5e015def0 2023-11-28T18:35:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e014fb30 con 0x7fe5e0159c90 2023-11-28T18:35:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00c6ad0 con 0x7fe5e0153ee0 2023-11-28T18:35:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e0144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e0144060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60164/0 (socket says 172.21.15.7:60164) 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5382640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe5e0153ee0 0x7fe5e0149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 -- 172.21.15.7:0/1997531172 learned_addr learned my addr 172.21.15.7:0/1997531172 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e4b81640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e0159c90 0x7fe5e00c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 -- 172.21.15.7:0/1997531172 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe5e0153ee0 msgr2=0x7fe5e0149570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe5e0153ee0 0x7fe5e0149570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 -- 172.21.15.7:0/1997531172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e0159c90 msgr2=0x7fe5e00c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e0159c90 0x7fe5e00c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 -- 172.21.15.7:0/1997531172 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8009a70 con 0x7fe5e015def0 2023-11-28T18:35:00.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5382640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe5e0153ee0 0x7fe5e0149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:35:00.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e5b83640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e0144060 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe5d801bd00 tx=0x7fe5d801be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:00.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/1997531172 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80b0440 con 0x7fe5e015def0 2023-11-28T18:35:00.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/1997531172 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d80ba820 con 0x7fe5e015def0 2023-11-28T18:35:00.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/1997531172 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8007000 con 0x7fe5e015def0 2023-11-28T18:35:00.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.072+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e00c6db0 con 0x7fe5e015def0 2023-11-28T18:35:00.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.076+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e00c9540 con 0x7fe5e015def0 2023-11-28T18:35:00.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.076+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e00c8490 con 0x7fe5e015def0 2023-11-28T18:35:00.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.076+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/1997531172 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe5d80b7050 con 0x7fe5e015def0 2023-11-28T18:35:00.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.076+0000 7fe5d5ffb640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe5c40668d0 0x7fe5c4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:00.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.076+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/1997531172 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fe5d8122c30 con 0x7fe5e015def0 2023-11-28T18:35:00.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.076+0000 7fe5e5382640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe5c40668d0 0x7fe5c4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:00.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.080+0000 7fe5e5382640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe5c40668d0 0x7fe5c4068d90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc000f70 tx=0x7fe5dc007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:00.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.080+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/1997531172 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe5d80efc70 con 0x7fe5e015def0 2023-11-28T18:35:00.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.320+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5e004e890 con 0x7fe5e015def0 2023-11-28T18:35:00.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/1997531172 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7fe5d80b5070 con 0x7fe5e015def0 2023-11-28T18:35:00.324 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:00.326 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:00.327 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:35:00.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe5c40668d0 msgr2=0x7fe5c4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe5c40668d0 0x7fe5c4068d90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc000f70 tx=0x7fe5dc007e90 comp rx=0 tx=0).stop 2023-11-28T18:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 msgr2=0x7fe5e0144060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e0144060 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe5d801bd00 tx=0x7fe5d801be90 comp rx=0 tx=0).stop 2023-11-28T18:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 shutdown_connections 2023-11-28T18:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe5c40668d0 0x7fe5c4068d90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e015def0 0x7fe5e0144060 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e0159c90 0x7fe5e00c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 --2- 172.21.15.7:0/1997531172 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe5e0153ee0 0x7fe5e0149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 >> 172.21.15.7:0/1997531172 conn(0x7fe5e0071e20 msgr2=0x7fe5e00cb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 shutdown_connections 2023-11-28T18:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:00.324+0000 7fe5e6384640 1 -- 172.21.15.7:0/1997531172 wait complete. 2023-11-28T18:35:00.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-11-28T18:35:00.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:35:00.984 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:35:02.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:02.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.488+0000 7f369ac32640 1 Processor -- start 2023-11-28T18:35:02.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f369ac32640 1 -- start start 2023-11-28T18:35:02.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f369ac32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3694150b30 0x7f36940c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:02.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f369ac32640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f36940c7980 0x7f36940c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:02.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f369ac32640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36940c6140 con 0x7f36941575c0 2023-11-28T18:35:02.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f369ac32640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36940c72b0 con 0x7f36940c7980 2023-11-28T18:35:02.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f369ac32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3694070c00 con 0x7f3694150b30 2023-11-28T18:35:02.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36991cf640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f36940c7980 0x7f36940c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:02.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36989ce640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3694150b30 0x7f36940c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:02.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36991cf640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f36940c7980 0x7f36940c7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:60460/0 (socket says 172.21.15.7:60460) 2023-11-28T18:35:02.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36991cf640 1 -- 172.21.15.7:0/129170446 learned_addr learned my addr 172.21.15.7:0/129170446 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:02.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36989ce640 1 -- 172.21.15.7:0/129170446 >> v2:172.21.15.193:3300/0 conn(0x7f36940c7980 msgr2=0x7f36940c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:02.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36989ce640 1 --2- 172.21.15.7:0/129170446 >> v2:172.21.15.193:3300/0 conn(0x7f36940c7980 0x7f36940c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36989ce640 1 -- 172.21.15.7:0/129170446 >> v1:172.21.15.193:6789/0 conn(0x7f36941575c0 legacy=0x7f36940c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:02.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36989ce640 1 -- 172.21.15.7:0/129170446 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36940c8350 con 0x7f3694150b30 2023-11-28T18:35:02.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.492+0000 7f36991cf640 1 --2- 172.21.15.7:0/129170446 >> v2:172.21.15.193:3300/0 conn(0x7f36940c7980 0x7f36940c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:35:02.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f36989ce640 1 --2- 172.21.15.7:0/129170446 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3694150b30 0x7f36940c24f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f36880097c0 tx=0x7f368801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e1d45c38f52ce9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:02.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f36937fe640 1 -- 172.21.15.7:0/129170446 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36880a8970 con 0x7f3694150b30 2023-11-28T18:35:02.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f36937fe640 1 -- 172.21.15.7:0/129170446 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36880a8ad0 con 0x7f3694150b30 2023-11-28T18:35:02.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f36937fe640 1 -- 172.21.15.7:0/129170446 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36880a8e60 con 0x7f3694150b30 2023-11-28T18:35:02.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 -- 172.21.15.7:0/129170446 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3694150b30 msgr2=0x7f36940c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:02.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 --2- 172.21.15.7:0/129170446 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3694150b30 0x7f36940c24f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f36880097c0 tx=0x7f368801ec70 comp rx=0 tx=0).stop 2023-11-28T18:35:02.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 -- 172.21.15.7:0/129170446 shutdown_connections 2023-11-28T18:35:02.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 --2- 172.21.15.7:0/129170446 >> v2:172.21.15.193:3300/0 conn(0x7f36940c7980 0x7f36940c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 --2- 172.21.15.7:0/129170446 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3694150b30 0x7f36940c24f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 -- 172.21.15.7:0/129170446 >> 172.21.15.7:0/129170446 conn(0x7f36940662e0 msgr2=0x7f36940666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:02.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 -- 172.21.15.7:0/129170446 shutdown_connections 2023-11-28T18:35:02.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 -- 172.21.15.7:0/129170446 wait complete. 2023-11-28T18:35:02.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.496+0000 7f369ac32640 1 Processor -- start 2023-11-28T18:35:02.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 -- start start 2023-11-28T18:35:02.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940c7980 0x7f36941f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:02.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3694150b30 0x7f36941f0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:02.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36941575c0 0x7f369414d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:02.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3694155480 con 0x7f36940c7980 2023-11-28T18:35:02.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36941ef630 con 0x7f3694150b30 2023-11-28T18:35:02.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36941f3af0 con 0x7f36941575c0 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940c7980 0x7f36941f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940c7980 0x7f36941f35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60168/0 (socket says 172.21.15.7:60168) 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 -- 172.21.15.7:0/2006338589 learned_addr learned my addr 172.21.15.7:0/2006338589 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f3693fff640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3694150b30 0x7f36941f0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36991cf640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36941575c0 0x7f369414d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 -- 172.21.15.7:0/2006338589 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36941575c0 msgr2=0x7f369414d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36941575c0 0x7f369414d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 -- 172.21.15.7:0/2006338589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3694150b30 msgr2=0x7f36941f0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3694150b30 0x7f36941f0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 -- 172.21.15.7:0/2006338589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c009b30 con 0x7f36940c7980 2023-11-28T18:35:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36989ce640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940c7980 0x7f36941f35b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f36880b3040 tx=0x7f368801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36917fa640 1 -- 172.21.15.7:0/2006338589 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36880b9630 con 0x7f36940c7980 2023-11-28T18:35:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36917fa640 1 -- 172.21.15.7:0/2006338589 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36880b97d0 con 0x7f36940c7980 2023-11-28T18:35:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3688009420 con 0x7f36940c7980 2023-11-28T18:35:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f36917fa640 1 -- 172.21.15.7:0/2006338589 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36880b9b80 con 0x7f36940c7980 2023-11-28T18:35:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36941efbb0 con 0x7f36940c7980 2023-11-28T18:35:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.500+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369404e890 con 0x7f36940c7980 2023-11-28T18:35:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.504+0000 7f36917fa640 1 -- 172.21.15.7:0/2006338589 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f36880b7040 con 0x7f36940c7980 2023-11-28T18:35:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.504+0000 7f36917fa640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36780668d0 0x7f3678068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.504+0000 7f36917fa640 1 -- 172.21.15.7:0/2006338589 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f3688122520 con 0x7f36940c7980 2023-11-28T18:35:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.504+0000 7f3693fff640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36780668d0 0x7f3678068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.504+0000 7f3693fff640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36780668d0 0x7f3678068d90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f369414ac40 tx=0x7f3680009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:02.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.508+0000 7f36917fa640 1 -- 172.21.15.7:0/2006338589 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f36880ef670 con 0x7f36940c7980 2023-11-28T18:35:02.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.760+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3694154be0 con 0x7f36940c7980 2023-11-28T18:35:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f36917fa640 1 -- 172.21.15.7:0/2006338589 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f36880b5090 con 0x7f36940c7980 2023-11-28T18:35:02.764 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:02.764 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36780668d0 msgr2=0x7f3678068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36780668d0 0x7f3678068d90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f369414ac40 tx=0x7f3680009040 comp rx=0 tx=0).stop 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940c7980 msgr2=0x7f36941f35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940c7980 0x7f36941f35b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f36880b3040 tx=0x7f368801ba10 comp rx=0 tx=0).stop 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 shutdown_connections 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36780668d0 0x7f3678068d90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36941575c0 0x7f369414d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3694150b30 0x7f36941f0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 --2- 172.21.15.7:0/2006338589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940c7980 0x7f36941f35b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:02.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 >> 172.21.15.7:0/2006338589 conn(0x7f36940662e0 msgr2=0x7f36940666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:02.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 shutdown_connections 2023-11-28T18:35:02.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:02.764+0000 7f369ac32640 1 -- 172.21.15.7:0/2006338589 wait complete. 2023-11-28T18:35:02.790 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-11-28T18:35:04.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b50b56640 1 Processor -- start 2023-11-28T18:35:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b50b56640 1 -- start start 2023-11-28T18:35:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b50b56640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c1540a0 0x7f1b4c0c8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b50b56640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b4c0c9980 0x7f1b4c15d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b50b56640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c0c94f0 con 0x7f1b4c0c7ea0 2023-11-28T18:35:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b50b56640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4c154480 con 0x7f1b4c0c9980 2023-11-28T18:35:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b50b56640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c0ca850 con 0x7f1b4c1540a0 2023-11-28T18:35:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b4c0c9980 0x7f1b4c15d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1b4c0c7ea0 0x7f1b4c072890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38128/0 (socket says 172.21.15.7:38128) 2023-11-28T18:35:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b4c0c9980 0x7f1b4c15d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47190/0 (socket says 172.21.15.7:47190) 2023-11-28T18:35:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4bfff640 1 -- 172.21.15.7:0/3385125878 learned_addr learned my addr 172.21.15.7:0/3385125878 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:35:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4b7fe640 1 --2- 172.21.15.7:0/3385125878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c1540a0 0x7f1b4c0c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4affd640 1 -- 172.21.15.7:0/3385125878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c1540a0 msgr2=0x7f1b4c0c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4affd640 1 --2- 172.21.15.7:0/3385125878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c1540a0 0x7f1b4c0c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4affd640 1 -- 172.21.15.7:0/3385125878 >> v1:172.21.15.7:6789/0 conn(0x7f1b4c0c7ea0 legacy=0x7f1b4c072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4affd640 1 -- 172.21.15.7:0/3385125878 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c06c5b0 con 0x7f1b4c0c9980 2023-11-28T18:35:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.952+0000 7f1b4a7fc640 1 -- 172.21.15.7:0/3385125878 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366706637 0 0) 0x7f1b4c0c94f0 con 0x7f1b4c0c7ea0 2023-11-28T18:35:04.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b4affd640 1 --2- 172.21.15.7:0/3385125878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c0c9980 0x7f1b4c15d2f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f1b400097c0 tx=0x7f1b4001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d611cd7bfd4f45ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b4a7fc640 1 -- 172.21.15.7:0/3385125878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b400ab5f0 con 0x7f1b4c0c9980 2023-11-28T18:35:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b4a7fc640 1 -- 172.21.15.7:0/3385125878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b400ab750 con 0x7f1b4c0c9980 2023-11-28T18:35:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b4a7fc640 1 -- 172.21.15.7:0/3385125878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b400abae0 con 0x7f1b4c0c9980 2023-11-28T18:35:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b50b56640 1 -- 172.21.15.7:0/3385125878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c0c9980 msgr2=0x7f1b4c15d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3385125878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c0c9980 0x7f1b4c15d2f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f1b400097c0 tx=0x7f1b4001c3f0 comp rx=0 tx=0).stop 2023-11-28T18:35:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b50b56640 1 -- 172.21.15.7:0/3385125878 shutdown_connections 2023-11-28T18:35:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3385125878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c0c9980 0x7f1b4c15d2f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3385125878 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c1540a0 0x7f1b4c0c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b50b56640 1 -- 172.21.15.7:0/3385125878 >> 172.21.15.7:0/3385125878 conn(0x7f1b4c073ea0 msgr2=0x7f1b4c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b50b56640 1 -- 172.21.15.7:0/3385125878 shutdown_connections 2023-11-28T18:35:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.956+0000 7f1b50b56640 1 -- 172.21.15.7:0/3385125878 wait complete. 2023-11-28T18:35:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b50b56640 1 Processor -- start 2023-11-28T18:35:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b50b56640 1 -- start start 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b50b56640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c0c7ea0 0x7f1b4c154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b50b56640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b4c0c9980 0x7f1b4c1f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b50b56640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c1540a0 0x7f1b4c146d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b50b56640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c149d20 con 0x7f1b4c1540a0 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b50b56640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c1f21e0 con 0x7f1b4c0c9980 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b50b56640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c149a40 con 0x7f1b4c0c7ea0 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b4bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c1540a0 0x7f1b4c146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b4bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c1540a0 0x7f1b4c146d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47204/0 (socket says 172.21.15.7:47204) 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b4bfff640 1 -- 172.21.15.7:0/3638939939 learned_addr learned my addr 172.21.15.7:0/3638939939 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b4affd640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b4c0c9980 0x7f1b4c1f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.960+0000 7f1b4b7fe640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c0c7ea0 0x7f1b4c154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b4bfff640 1 -- 172.21.15.7:0/3638939939 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c0c7ea0 msgr2=0x7f1b4c154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b4bfff640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c0c7ea0 0x7f1b4c154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b4bfff640 1 -- 172.21.15.7:0/3638939939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b4c0c9980 msgr2=0x7f1b4c1f2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b4bfff640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b4c0c9980 0x7f1b4c1f2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b4bfff640 1 -- 172.21.15.7:0/3638939939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b40009420 con 0x7f1b4c1540a0 2023-11-28T18:35:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b4bfff640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c1540a0 0x7f1b4c146d20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c780 tx=0x7f1b4400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b33fff640 1 -- 172.21.15.7:0/3638939939 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4400ce90 con 0x7f1b4c1540a0 2023-11-28T18:35:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b33fff640 1 -- 172.21.15.7:0/3638939939 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4400a640 con 0x7f1b4c1540a0 2023-11-28T18:35:04.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4c1f58f0 con 0x7f1b4c1540a0 2023-11-28T18:35:04.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b33fff640 1 -- 172.21.15.7:0/3638939939 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4400a9d0 con 0x7f1b4c1540a0 2023-11-28T18:35:04.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c1f2780 con 0x7f1b4c1540a0 2023-11-28T18:35:04.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4c1f2970 con 0x7f1b4c1540a0 2023-11-28T18:35:04.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.964+0000 7f1b33fff640 1 -- 172.21.15.7:0/3638939939 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1b4400ab30 con 0x7f1b4c1540a0 2023-11-28T18:35:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.968+0000 7f1b33fff640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1b380668d0 0x7f1b38068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.968+0000 7f1b33fff640 1 -- 172.21.15.7:0/3638939939 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f1b44083d80 con 0x7f1b4c1540a0 2023-11-28T18:35:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.968+0000 7f1b4b7fe640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1b380668d0 0x7f1b38068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.968+0000 7f1b4b7fe640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1b380668d0 0x7f1b38068d90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009b60 tx=0x7f1b3c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:04.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:04.972+0000 7f1b33fff640 1 -- 172.21.15.7:0/3638939939 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1b44050e50 con 0x7f1b4c1540a0 2023-11-28T18:35:05.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.200+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b4c04e890 con 0x7f1b4c1540a0 2023-11-28T18:35:05.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.200+0000 7f1b33fff640 1 -- 172.21.15.7:0/3638939939 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f1b44018020 con 0x7f1b4c1540a0 2023-11-28T18:35:05.203 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:05.203 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:33:14.275917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6839/1819203827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":1819203827},{"type":"v1","addr":"172.21.15.7:6839","nonce":1819203827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:05.204 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 53 2023-11-28T18:35:05.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1b380668d0 msgr2=0x7f1b38068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:05.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1b380668d0 0x7f1b38068d90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009b60 tx=0x7f1b3c009190 comp rx=0 tx=0).stop 2023-11-28T18:35:05.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c1540a0 msgr2=0x7f1b4c146d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:05.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c1540a0 0x7f1b4c146d20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c780 tx=0x7f1b4400cc40 comp rx=0 tx=0).stop 2023-11-28T18:35:05.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 shutdown_connections 2023-11-28T18:35:05.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1b380668d0 0x7f1b38068d90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:05.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b4c1540a0 0x7f1b4c146d20 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:05.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b4c0c9980 0x7f1b4c1f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:05.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 --2- 172.21.15.7:0/3638939939 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b4c0c7ea0 0x7f1b4c154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:05.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 >> 172.21.15.7:0/3638939939 conn(0x7f1b4c073ea0 msgr2=0x7f1b4c0c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:05.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 shutdown_connections 2023-11-28T18:35:05.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:05.204+0000 7f1b50b56640 1 -- 172.21.15.7:0/3638939939 wait complete. 2023-11-28T18:35:05.228 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-11-28T18:35:07.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fccd28fe640 1 Processor -- start 2023-11-28T18:35:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fccd28fe640 1 -- start start 2023-11-28T18:35:07.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fccd28fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc1542e0 0x7fcccc15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fccd28fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcccc0cbb70 0x7fcccc0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fccd28fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc0ca920 con 0x7fcccc0cc810 2023-11-28T18:35:07.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fccd28fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc071d50 con 0x7fcccc0cbb70 2023-11-28T18:35:07.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fccd28fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc072030 con 0x7fcccc1542e0 2023-11-28T18:35:07.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccb7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcccc0cbb70 0x7fcccc0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccb7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcccc0cbb70 0x7fcccc0cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47212/0 (socket says 172.21.15.7:47212) 2023-11-28T18:35:07.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccb7fe640 1 -- 172.21.15.7:0/1589674774 learned_addr learned my addr 172.21.15.7:0/1589674774 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:07.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccbfff640 1 --2- 172.21.15.7:0/1589674774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc1542e0 0x7fcccc15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccaffd640 1 -- 172.21.15.7:0/1589674774 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227299377 0 0) 0x7fcccc0ca920 con 0x7fcccc0cc810 2023-11-28T18:35:07.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccb7fe640 1 -- 172.21.15.7:0/1589674774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc1542e0 msgr2=0x7fcccc15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:07.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccb7fe640 1 --2- 172.21.15.7:0/1589674774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc1542e0 0x7fcccc15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccb7fe640 1 -- 172.21.15.7:0/1589674774 >> v1:172.21.15.193:6789/0 conn(0x7fcccc0cc810 legacy=0x7fcccc072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.392+0000 7fcccb7fe640 1 -- 172.21.15.7:0/1589674774 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc15e550 con 0x7fcccc0cbb70 2023-11-28T18:35:07.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fcccb7fe640 1 --2- 172.21.15.7:0/1589674774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cc2d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009870 tx=0x7fccc001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5608b22951e4ad5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:07.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fcccaffd640 1 -- 172.21.15.7:0/1589674774 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc00a9400 con 0x7fcccc0cbb70 2023-11-28T18:35:07.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fcccaffd640 1 -- 172.21.15.7:0/1589674774 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc00a9560 con 0x7fcccc0cbb70 2023-11-28T18:35:07.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fcccaffd640 1 -- 172.21.15.7:0/1589674774 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc00a98f0 con 0x7fcccc0cbb70 2023-11-28T18:35:07.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fccd28fe640 1 -- 172.21.15.7:0/1589674774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 msgr2=0x7fcccc0cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fccd28fe640 1 --2- 172.21.15.7:0/1589674774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cc2d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009870 tx=0x7fccc001c540 comp rx=0 tx=0).stop 2023-11-28T18:35:07.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fccd28fe640 1 -- 172.21.15.7:0/1589674774 shutdown_connections 2023-11-28T18:35:07.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fccd28fe640 1 --2- 172.21.15.7:0/1589674774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cc2d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fccd28fe640 1 --2- 172.21.15.7:0/1589674774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc1542e0 0x7fcccc15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fccd28fe640 1 -- 172.21.15.7:0/1589674774 >> 172.21.15.7:0/1589674774 conn(0x7fcccc0717e0 msgr2=0x7fcccc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:07.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fccd28fe640 1 -- 172.21.15.7:0/1589674774 shutdown_connections 2023-11-28T18:35:07.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.396+0000 7fccd28fe640 1 -- 172.21.15.7:0/1589674774 wait complete. 2023-11-28T18:35:07.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fccd28fe640 1 Processor -- start 2023-11-28T18:35:07.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fccd28fe640 1 -- start start 2023-11-28T18:35:07.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fccd28fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fccd28fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc0cc810 0x7fcccc13fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fccd28fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc1542e0 0x7fcccc1402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fccd28fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc13f8e0 con 0x7fcccc0cbb70 2023-11-28T18:35:07.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fccd28fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0ce8e0 con 0x7fcccc1542e0 2023-11-28T18:35:07.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fccd28fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0cebc0 con 0x7fcccc0cc810 2023-11-28T18:35:07.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fcccbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fcccbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47220/0 (socket says 172.21.15.7:47220) 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.400+0000 7fcccbfff640 1 -- 172.21.15.7:0/2657582766 learned_addr learned my addr 172.21.15.7:0/2657582766 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fcccb7fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc0cc810 0x7fcccc13fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fccd0e9b640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc1542e0 0x7fcccc1402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fcccbfff640 1 -- 172.21.15.7:0/2657582766 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc0cc810 msgr2=0x7fcccc13fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fcccbfff640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc0cc810 0x7fcccc13fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fcccbfff640 1 -- 172.21.15.7:0/2657582766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc1542e0 msgr2=0x7fcccc1402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fcccbfff640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc1542e0 0x7fcccc1402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fcccbfff640 1 -- 172.21.15.7:0/2657582766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00094d0 con 0x7fcccc0cbb70 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fcccbfff640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cfe70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00e870 tx=0x7fccbc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:07.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fccc8ff9640 1 -- 172.21.15.7:0/2657582766 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc009510 con 0x7fcccc0cbb70 2023-11-28T18:35:07.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fccc8ff9640 1 -- 172.21.15.7:0/2657582766 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc00eec0 con 0x7fcccc0cbb70 2023-11-28T18:35:07.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fccc8ff9640 1 -- 172.21.15.7:0/2657582766 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc00cb40 con 0x7fcccc0cbb70 2023-11-28T18:35:07.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc0ceea0 con 0x7fcccc0cbb70 2023-11-28T18:35:07.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc0cf460 con 0x7fcccc0cbb70 2023-11-28T18:35:07.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.404+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc06c670 con 0x7fcccc0cbb70 2023-11-28T18:35:07.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.408+0000 7fccc8ff9640 1 -- 172.21.15.7:0/2657582766 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fccbc00cca0 con 0x7fcccc0cbb70 2023-11-28T18:35:07.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.408+0000 7fccc8ff9640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb80668d0 0x7fccb8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.408+0000 7fccc8ff9640 1 -- 172.21.15.7:0/2657582766 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fccbc084c20 con 0x7fcccc0cbb70 2023-11-28T18:35:07.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.408+0000 7fcccb7fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb80668d0 0x7fccb8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.408+0000 7fcccb7fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb80668d0 0x7fccb8068d90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fccc00ae000 tx=0x7fccc00b3040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:07.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.412+0000 7fccc8ff9640 1 -- 172.21.15.7:0/2657582766 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fccbc051b60 con 0x7fcccc0cbb70 2023-11-28T18:35:07.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.656+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcccc04e890 con 0x7fcccc0cbb70 2023-11-28T18:35:07.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.656+0000 7fccc8ff9640 1 -- 172.21.15.7:0/2657582766 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5767 (secure 0 0 0) 0x7fccbc021090 con 0x7fcccc0cbb70 2023-11-28T18:35:07.658 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:07.658 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:07.279922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:07.659 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 54 2023-11-28T18:35:07.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb80668d0 msgr2=0x7fccb8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb80668d0 0x7fccb8068d90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fccc00ae000 tx=0x7fccc00b3040 comp rx=0 tx=0).stop 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 msgr2=0x7fcccc0cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cfe70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00e870 tx=0x7fccbc00ec70 comp rx=0 tx=0).stop 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 shutdown_connections 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb80668d0 0x7fccb8068d90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc1542e0 0x7fcccc1402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcccc0cc810 0x7fcccc13fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 --2- 172.21.15.7:0/2657582766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc0cbb70 0x7fcccc0cfe70 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 >> 172.21.15.7:0/2657582766 conn(0x7fcccc0717e0 msgr2=0x7fcccc0b5070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 shutdown_connections 2023-11-28T18:35:07.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.660+0000 7fccd28fe640 1 -- 172.21.15.7:0/2657582766 wait complete. 2023-11-28T18:35:07.685 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-11-28T18:35:07.685 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:35:07.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.840+0000 7fd8086ad640 1 Processor -- start 2023-11-28T18:35:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.840+0000 7fd8086ad640 1 -- start start 2023-11-28T18:35:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.840+0000 7fd8086ad640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fd800143130 0x7fd8000c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.840+0000 7fd8086ad640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8000669f0 0x7fd800164830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.840+0000 7fd8086ad640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80015e050 con 0x7fd80006f380 2023-11-28T18:35:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.840+0000 7fd8086ad640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd800159fc0 con 0x7fd800143130 2023-11-28T18:35:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.840+0000 7fd8086ad640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80015a2a0 con 0x7fd8000669f0 2023-11-28T18:35:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd806449640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fd800143130 0x7fd8000c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd806c4a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8000669f0 0x7fd800164830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd806449640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fd800143130 0x7fd8000c7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:44432/0 (socket says 172.21.15.7:44432) 2023-11-28T18:35:07.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd806449640 1 -- 172.21.15.7:0/3669227583 learned_addr learned my addr 172.21.15.7:0/3669227583 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:07.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd805c48640 1 --1- 172.21.15.7:0/3669227583 >> v1:172.21.15.193:6789/0 conn(0x7fd80006f380 0x7fd8000662e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:43706/0 (socket says 172.21.15.7:43706) 2023-11-28T18:35:07.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd806c4a640 1 -- 172.21.15.7:0/3669227583 >> v2:172.21.15.193:3300/0 conn(0x7fd800143130 msgr2=0x7fd8000c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:07.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd806c4a640 1 --2- 172.21.15.7:0/3669227583 >> v2:172.21.15.193:3300/0 conn(0x7fd800143130 0x7fd8000c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd806c4a640 1 -- 172.21.15.7:0/3669227583 >> v1:172.21.15.193:6789/0 conn(0x7fd80006f380 legacy=0x7fd8000662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.844+0000 7fd806c4a640 1 -- 172.21.15.7:0/3669227583 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80015a580 con 0x7fd8000669f0 2023-11-28T18:35:07.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd806c4a640 1 --2- 172.21.15.7:0/3669227583 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8000669f0 0x7fd800164830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009d90 tx=0x7fd7f801c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766a68fb2e7ccce9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:07.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd805447640 1 -- 172.21.15.7:0/3669227583 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80a59a0 con 0x7fd8000669f0 2023-11-28T18:35:07.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd805447640 1 -- 172.21.15.7:0/3669227583 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f80a5b00 con 0x7fd8000669f0 2023-11-28T18:35:07.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd805447640 1 -- 172.21.15.7:0/3669227583 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80a5e90 con 0x7fd8000669f0 2023-11-28T18:35:07.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 -- 172.21.15.7:0/3669227583 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8000669f0 msgr2=0x7fd800164830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3669227583 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8000669f0 0x7fd800164830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009d90 tx=0x7fd7f801c950 comp rx=0 tx=0).stop 2023-11-28T18:35:07.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 -- 172.21.15.7:0/3669227583 shutdown_connections 2023-11-28T18:35:07.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3669227583 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8000669f0 0x7fd800164830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3669227583 >> v2:172.21.15.193:3300/0 conn(0x7fd800143130 0x7fd8000c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 -- 172.21.15.7:0/3669227583 >> 172.21.15.7:0/3669227583 conn(0x7fd8000cc490 msgr2=0x7fd8000cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:07.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 -- 172.21.15.7:0/3669227583 shutdown_connections 2023-11-28T18:35:07.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 -- 172.21.15.7:0/3669227583 wait complete. 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 Processor -- start 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.848+0000 7fd8086ad640 1 -- start start 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8000669f0 0x7fd80015c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80006f380 0x7fd80015cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd800143130 0x7fd8001f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8001551b0 con 0x7fd8000669f0 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8001553c0 con 0x7fd80006f380 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8001556a0 con 0x7fd800143130 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8000669f0 0x7fd80015c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8000669f0 0x7fd80015c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47226/0 (socket says 172.21.15.7:47226) 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 -- 172.21.15.7:0/3277030187 learned_addr learned my addr 172.21.15.7:0/3277030187 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd805c48640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80006f380 0x7fd80015cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806c4a640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd800143130 0x7fd8001f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 -- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd800143130 msgr2=0x7fd8001f4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd800143130 0x7fd8001f4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 -- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80006f380 msgr2=0x7fd80015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80006f380 0x7fd80015cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 -- 172.21.15.7:0/3277030187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f8009a70 con 0x7fd8000669f0 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806c4a640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd800143130 0x7fd8001f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd806449640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8000669f0 0x7fd80015c840 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00e890 tx=0x7fd7fc00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd7f6ffd640 1 -- 172.21.15.7:0/3277030187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc009620 con 0x7fd8000669f0 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd7f6ffd640 1 -- 172.21.15.7:0/3277030187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc00c940 con 0x7fd8000669f0 2023-11-28T18:35:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd7f6ffd640 1 -- 172.21.15.7:0/3277030187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00ccf0 con 0x7fd8000669f0 2023-11-28T18:35:07.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd800155980 con 0x7fd8000669f0 2023-11-28T18:35:07.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd800141410 con 0x7fd8000669f0 2023-11-28T18:35:07.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.852+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80004e890 con 0x7fd8000669f0 2023-11-28T18:35:07.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.856+0000 7fd7f6ffd640 1 -- 172.21.15.7:0/3277030187 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd7fc013470 con 0x7fd8000669f0 2023-11-28T18:35:07.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.856+0000 7fd7f6ffd640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7e4066900 0x7fd7e4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:07.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.856+0000 7fd7f6ffd640 1 -- 172.21.15.7:0/3277030187 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fd7fc084740 con 0x7fd8000669f0 2023-11-28T18:35:07.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.856+0000 7fd805c48640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7e4066900 0x7fd7e4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:07.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.856+0000 7fd805c48640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7e4066900 0x7fd7e4068dc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd80006b480 tx=0x7fd7f0007e50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:07.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:07.860+0000 7fd7f6ffd640 1 -- 172.21.15.7:0/3277030187 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd7fc0517b0 con 0x7fd8000669f0 2023-11-28T18:35:08.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.084+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd800141ba0 con 0x7fd8000669f0 2023-11-28T18:35:08.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.084+0000 7fd7f6ffd640 1 -- 172.21.15.7:0/3277030187 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5767 (secure 0 0 0) 0x7fd7fc054dd0 con 0x7fd8000669f0 2023-11-28T18:35:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:08.088 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:07.279922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 54 2023-11-28T18:35:08.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7e4066900 msgr2=0x7fd7e4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:08.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7e4066900 0x7fd7e4068dc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd80006b480 tx=0x7fd7f0007e50 comp rx=0 tx=0).stop 2023-11-28T18:35:08.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8000669f0 msgr2=0x7fd80015c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:08.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8000669f0 0x7fd80015c840 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00e890 tx=0x7fd7fc00ed50 comp rx=0 tx=0).stop 2023-11-28T18:35:08.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 shutdown_connections 2023-11-28T18:35:08.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd7e4066900 0x7fd7e4068dc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd800143130 0x7fd8001f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80006f380 0x7fd80015cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 --2- 172.21.15.7:0/3277030187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8000669f0 0x7fd80015c840 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 >> 172.21.15.7:0/3277030187 conn(0x7fd8000cc490 msgr2=0x7fd800158620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 shutdown_connections 2023-11-28T18:35:08.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:08.088+0000 7fd8086ad640 1 -- 172.21.15.7:0/3277030187 wait complete. 2023-11-28T18:35:08.113 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 54, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 33 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:35:07.279922+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5359, "mds_2": 5307, "mds_3": 5086, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:35:10.114 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.272+0000 7fbb38d01640 1 Processor -- start 2023-11-28T18:35:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.272+0000 7fbb38d01640 1 -- start start 2023-11-28T18:35:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb38d01640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb341540e0 0x7fbb340ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb38d01640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbb3415a8a0 0x7fbb3415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb38d01640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3415d7e0 con 0x7fbb340cf6c0 2023-11-28T18:35:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb38d01640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb340cab80 con 0x7fbb341540e0 2023-11-28T18:35:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb38d01640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb340cb850 con 0x7fbb3415a8a0 2023-11-28T18:35:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb3259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb341540e0 0x7fbb340ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb3259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb341540e0 0x7fbb340ca4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47236/0 (socket says 172.21.15.7:47236) 2023-11-28T18:35:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb3259c640 1 -- 172.21.15.7:0/1590878815 learned_addr learned my addr 172.21.15.7:0/1590878815 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:10.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb2bfff640 1 --2- 172.21.15.7:0/1590878815 >> v2:172.21.15.193:3300/0 conn(0x7fbb3415a8a0 0x7fbb3415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:10.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb3259c640 1 -- 172.21.15.7:0/1590878815 >> v2:172.21.15.193:3300/0 conn(0x7fbb3415a8a0 msgr2=0x7fbb3415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:10.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb3259c640 1 --2- 172.21.15.7:0/1590878815 >> v2:172.21.15.193:3300/0 conn(0x7fbb3415a8a0 0x7fbb3415dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb3259c640 1 -- 172.21.15.7:0/1590878815 >> v1:172.21.15.193:6789/0 conn(0x7fbb340cf6c0 legacy=0x7fbb340cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:10.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.276+0000 7fbb3259c640 1 -- 172.21.15.7:0/1590878815 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb340cbb30 con 0x7fbb341540e0 2023-11-28T18:35:10.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb3259c640 1 --2- 172.21.15.7:0/1590878815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb341540e0 0x7fbb340ca4d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbb3406b6a0 tx=0x7fbb2401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633f0b867320d66d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:10.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb31d9b640 1 -- 172.21.15.7:0/1590878815 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb240ad640 con 0x7fbb341540e0 2023-11-28T18:35:10.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb31d9b640 1 -- 172.21.15.7:0/1590878815 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb240ad7a0 con 0x7fbb341540e0 2023-11-28T18:35:10.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb31d9b640 1 -- 172.21.15.7:0/1590878815 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb240adb30 con 0x7fbb341540e0 2023-11-28T18:35:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 -- 172.21.15.7:0/1590878815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb341540e0 msgr2=0x7fbb340ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 --2- 172.21.15.7:0/1590878815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb341540e0 0x7fbb340ca4d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbb3406b6a0 tx=0x7fbb2401c4c0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 -- 172.21.15.7:0/1590878815 shutdown_connections 2023-11-28T18:35:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 --2- 172.21.15.7:0/1590878815 >> v2:172.21.15.193:3300/0 conn(0x7fbb3415a8a0 0x7fbb3415dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 --2- 172.21.15.7:0/1590878815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb341540e0 0x7fbb340ca4d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 -- 172.21.15.7:0/1590878815 >> 172.21.15.7:0/1590878815 conn(0x7fbb340662e0 msgr2=0x7fbb340666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 -- 172.21.15.7:0/1590878815 shutdown_connections 2023-11-28T18:35:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 -- 172.21.15.7:0/1590878815 wait complete. 2023-11-28T18:35:10.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 Processor -- start 2023-11-28T18:35:10.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.280+0000 7fbb38d01640 1 -- start start 2023-11-28T18:35:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340cf6c0 0x7fbb341453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb341540e0 0x7fbb3415cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb3415a8a0 0x7fbb34147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb34155a40 con 0x7fbb3415a8a0 2023-11-28T18:35:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb341577f0 con 0x7fbb340cf6c0 2023-11-28T18:35:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb341565e0 con 0x7fbb341540e0 2023-11-28T18:35:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb3415a8a0 0x7fbb34147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb3415a8a0 0x7fbb34147ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47238/0 (socket says 172.21.15.7:47238) 2023-11-28T18:35:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 -- 172.21.15.7:0/2298058800 learned_addr learned my addr 172.21.15.7:0/2298058800 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:10.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb3259c640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340cf6c0 0x7fbb341453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:10.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb2bfff640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb341540e0 0x7fbb3415cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:10.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 -- 172.21.15.7:0/2298058800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb341540e0 msgr2=0x7fbb3415cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:10.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb341540e0 0x7fbb3415cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 -- 172.21.15.7:0/2298058800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340cf6c0 msgr2=0x7fbb341453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:10.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340cf6c0 0x7fbb341453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 -- 172.21.15.7:0/2298058800 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb240094d0 con 0x7fbb3415a8a0 2023-11-28T18:35:10.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb32d9d640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb3415a8a0 0x7fbb34147ee0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00c7b0 tx=0x7fbb2c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:10.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb2affd640 1 -- 172.21.15.7:0/2298058800 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c004b70 con 0x7fbb3415a8a0 2023-11-28T18:35:10.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb2affd640 1 -- 172.21.15.7:0/2298058800 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2c004cd0 con 0x7fbb3415a8a0 2023-11-28T18:35:10.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb2affd640 1 -- 172.21.15.7:0/2298058800 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c00aa90 con 0x7fbb3415a8a0 2023-11-28T18:35:10.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb34145910 con 0x7fbb3415a8a0 2023-11-28T18:35:10.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb34159c70 con 0x7fbb3415a8a0 2023-11-28T18:35:10.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.284+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb340cbdd0 con 0x7fbb3415a8a0 2023-11-28T18:35:10.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.288+0000 7fbb2affd640 1 -- 172.21.15.7:0/2298058800 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbb2c00abf0 con 0x7fbb3415a8a0 2023-11-28T18:35:10.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.288+0000 7fbb2affd640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb14066900 0x7fbb14068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:10.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.288+0000 7fbb2affd640 1 -- 172.21.15.7:0/2298058800 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fbb2c0846b0 con 0x7fbb3415a8a0 2023-11-28T18:35:10.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.288+0000 7fbb3259c640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb14066900 0x7fbb14068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:10.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.288+0000 7fbb3259c640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb14066900 0x7fbb14068dc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb240af000 tx=0x7fbb24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:10.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.292+0000 7fbb2affd640 1 -- 172.21.15.7:0/2298058800 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbb2c051710 con 0x7fbb3415a8a0 2023-11-28T18:35:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.496+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb341f2f10 con 0x7fbb3415a8a0 2023-11-28T18:35:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.496+0000 7fbb2affd640 1 -- 172.21.15.7:0/2298058800 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5767 (secure 0 0 0) 0x7fbb2c054d30 con 0x7fbb3415a8a0 2023-11-28T18:35:10.499 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:10.499 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:07.279922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 54 2023-11-28T18:35:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb14066900 msgr2=0x7fbb14068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb14066900 0x7fbb14068dc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb240af000 tx=0x7fbb24008040 comp rx=0 tx=0).stop 2023-11-28T18:35:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb3415a8a0 msgr2=0x7fbb34147ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb3415a8a0 0x7fbb34147ee0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00c7b0 tx=0x7fbb2c00cc70 comp rx=0 tx=0).stop 2023-11-28T18:35:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 shutdown_connections 2023-11-28T18:35:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb14066900 0x7fbb14068dc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb3415a8a0 0x7fbb34147ee0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb341540e0 0x7fbb3415cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 --2- 172.21.15.7:0/2298058800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340cf6c0 0x7fbb341453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 >> 172.21.15.7:0/2298058800 conn(0x7fbb340662e0 msgr2=0x7fbb340666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 shutdown_connections 2023-11-28T18:35:10.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:10.500+0000 7fbb38d01640 1 -- 172.21.15.7:0/2298058800 wait complete. 2023-11-28T18:35:10.523 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:35:12.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.440+0000 7fb645c38640 1 Processor -- start 2023-11-28T18:35:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.440+0000 7fb645c38640 1 -- start start 2023-11-28T18:35:13.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb645c38640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb64015ce10 0x7fb6400ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:13.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb645c38640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fb6400cafe0 0x7fb6400ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:13.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb645c38640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64015a3a0 con 0x7fb64015ca30 2023-11-28T18:35:13.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb645c38640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64015a620 con 0x7fb64015ce10 2023-11-28T18:35:13.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb645c38640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64015a900 con 0x7fb6400cafe0 2023-11-28T18:35:13.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb63ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb64015ce10 0x7fb6400ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:13.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb63ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb64015ce10 0x7fb6400ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47248/0 (socket says 172.21.15.7:47248) 2023-11-28T18:35:13.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb63ffff640 1 -- 172.21.15.7:0/2011945341 learned_addr learned my addr 172.21.15.7:0/2011945341 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:13.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb645437640 1 --2- 172.21.15.7:0/2011945341 >> v2:172.21.15.193:3300/0 conn(0x7fb6400cafe0 0x7fb6400ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:13.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb63ffff640 1 -- 172.21.15.7:0/2011945341 >> v2:172.21.15.193:3300/0 conn(0x7fb6400cafe0 msgr2=0x7fb6400ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:13.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb63ffff640 1 --2- 172.21.15.7:0/2011945341 >> v2:172.21.15.193:3300/0 conn(0x7fb6400cafe0 0x7fb6400ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb63ffff640 1 -- 172.21.15.7:0/2011945341 >> v1:172.21.15.193:6789/0 conn(0x7fb64015ca30 legacy=0x7fb640159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:13.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.444+0000 7fb63ffff640 1 -- 172.21.15.7:0/2011945341 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64015abe0 con 0x7fb64015ce10 2023-11-28T18:35:13.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb63ffff640 1 --2- 172.21.15.7:0/2011945341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64015ce10 0x7fb6400ca1d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb6400671d0 tx=0x7fb62c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428dfa3ee4199c6e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb63f7fe640 1 -- 172.21.15.7:0/2011945341 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0aa430 con 0x7fb64015ce10 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb63f7fe640 1 -- 172.21.15.7:0/2011945341 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c0aa590 con 0x7fb64015ce10 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb63f7fe640 1 -- 172.21.15.7:0/2011945341 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0aa920 con 0x7fb64015ce10 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb645c38640 1 -- 172.21.15.7:0/2011945341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64015ce10 msgr2=0x7fb6400ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb645c38640 1 --2- 172.21.15.7:0/2011945341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64015ce10 0x7fb6400ca1d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb6400671d0 tx=0x7fb62c01c4d0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb645c38640 1 -- 172.21.15.7:0/2011945341 shutdown_connections 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb645c38640 1 --2- 172.21.15.7:0/2011945341 >> v2:172.21.15.193:3300/0 conn(0x7fb6400cafe0 0x7fb6400ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb645c38640 1 --2- 172.21.15.7:0/2011945341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64015ce10 0x7fb6400ca1d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb645c38640 1 -- 172.21.15.7:0/2011945341 >> 172.21.15.7:0/2011945341 conn(0x7fb64006c060 msgr2=0x7fb6400662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:13.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb645c38640 1 -- 172.21.15.7:0/2011945341 shutdown_connections 2023-11-28T18:35:13.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.448+0000 7fb645c38640 1 -- 172.21.15.7:0/2011945341 wait complete. 2023-11-28T18:35:13.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645c38640 1 Processor -- start 2023-11-28T18:35:13.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645c38640 1 -- start start 2023-11-28T18:35:13.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645c38640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6400cafe0 0x7fb6401f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:13.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645c38640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb64015ca30 0x7fb640157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:13.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645c38640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb64015ce10 0x7fb640155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:13.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645c38640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6401480b0 con 0x7fb6400cafe0 2023-11-28T18:35:13.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645c38640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6401f3110 con 0x7fb64015ce10 2023-11-28T18:35:13.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645c38640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb640159f50 con 0x7fb64015ca30 2023-11-28T18:35:13.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb644c36640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6400cafe0 0x7fb6401f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:13.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb644c36640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6400cafe0 0x7fb6401f35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47264/0 (socket says 172.21.15.7:47264) 2023-11-28T18:35:13.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb644c36640 1 -- 172.21.15.7:0/196428316 learned_addr learned my addr 172.21.15.7:0/196428316 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:13.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb63ffff640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb64015ca30 0x7fb640157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:13.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb645437640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb64015ce10 0x7fb640155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:13.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb644c36640 1 -- 172.21.15.7:0/196428316 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb64015ca30 msgr2=0x7fb640157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:13.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb644c36640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb64015ca30 0x7fb640157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb644c36640 1 -- 172.21.15.7:0/196428316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb64015ce10 msgr2=0x7fb640155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:13.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb644c36640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb64015ce10 0x7fb640155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.452+0000 7fb644c36640 1 -- 172.21.15.7:0/196428316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0094d0 con 0x7fb6400cafe0 2023-11-28T18:35:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.456+0000 7fb644c36640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6400cafe0 0x7fb6401f35a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb63400c500 tx=0x7fb63400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.456+0000 7fb63d7fa640 1 -- 172.21.15.7:0/196428316 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634005220 con 0x7fb6400cafe0 2023-11-28T18:35:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.456+0000 7fb63d7fa640 1 -- 172.21.15.7:0/196428316 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb634005380 con 0x7fb6400cafe0 2023-11-28T18:35:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.456+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb640159990 con 0x7fb6400cafe0 2023-11-28T18:35:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.456+0000 7fb63d7fa640 1 -- 172.21.15.7:0/196428316 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634007270 con 0x7fb6400cafe0 2023-11-28T18:35:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.456+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6401f2ac0 con 0x7fb6400cafe0 2023-11-28T18:35:13.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.456+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64004e890 con 0x7fb6400cafe0 2023-11-28T18:35:13.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.456+0000 7fb63d7fa640 1 -- 172.21.15.7:0/196428316 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb6340073d0 con 0x7fb6400cafe0 2023-11-28T18:35:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.460+0000 7fb63d7fa640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb624066900 0x7fb624068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.460+0000 7fb63d7fa640 1 -- 172.21.15.7:0/196428316 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fb634083ec0 con 0x7fb6400cafe0 2023-11-28T18:35:13.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.460+0000 7fb63ffff640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb624066900 0x7fb624068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:13.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.460+0000 7fb63ffff640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb624066900 0x7fb624068dc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0ae000 tx=0x7fb62c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.464+0000 7fb63d7fa640 1 -- 172.21.15.7:0/196428316 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb634050ea0 con 0x7fb6400cafe0 2023-11-28T18:35:13.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.696+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb64014aab0 con 0x7fb6400cafe0 2023-11-28T18:35:13.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.696+0000 7fb63d7fa640 1 -- 172.21.15.7:0/196428316 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fb6340544c0 con 0x7fb6400cafe0 2023-11-28T18:35:13.700 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:13.700 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:12.676974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:resolve","state_seq":29,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:13.700 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 55 2023-11-28T18:35:13.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb624066900 msgr2=0x7fb624068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:13.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb624066900 0x7fb624068dc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0ae000 tx=0x7fb62c0a7000 comp rx=0 tx=0).stop 2023-11-28T18:35:13.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6400cafe0 msgr2=0x7fb6401f35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:13.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6400cafe0 0x7fb6401f35a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb63400c500 tx=0x7fb63400c9c0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 shutdown_connections 2023-11-28T18:35:13.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb624066900 0x7fb624068dc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb64015ce10 0x7fb640155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb64015ca30 0x7fb640157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 --2- 172.21.15.7:0/196428316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6400cafe0 0x7fb6401f35a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:13.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 >> 172.21.15.7:0/196428316 conn(0x7fb64006c060 msgr2=0x7fb6400c8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:13.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 shutdown_connections 2023-11-28T18:35:13.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:13.700+0000 7fb645c38640 1 -- 172.21.15.7:0/196428316 wait complete. 2023-11-28T18:35:13.723 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:35:15.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa6712dd640 1 Processor -- start 2023-11-28T18:35:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa6712dd640 1 -- start start 2023-11-28T18:35:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa6712dd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa66c1542e0 0x7fa66c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa6712dd640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c0ca920 con 0x7fa66c0cbb70 2023-11-28T18:35:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa6712dd640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c15e550 con 0x7fa66c072450 2023-11-28T18:35:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa6712dd640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c071d50 con 0x7fa66c1542e0 2023-11-28T18:35:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa66a7fc640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa66c0cbb70 0x7fa66c0cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39124/0 (socket says 172.21.15.7:39124) 2023-11-28T18:35:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa66a7fc640 1 -- 172.21.15.7:0/2897661060 learned_addr learned my addr 172.21.15.7:0/2897661060 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:35:15.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa66affd640 1 --2- 172.21.15.7:0/2897661060 >> v2:172.21.15.193:3300/0 conn(0x7fa66c1542e0 0x7fa66c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:15.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa669ffb640 1 -- 172.21.15.7:0/2897661060 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920737374 0 0) 0x7fa66c0ca920 con 0x7fa66c0cbb70 2023-11-28T18:35:15.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.868+0000 7fa669ffb640 1 -- 172.21.15.7:0/2897661060 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa654003530 con 0x7fa66c0cbb70 2023-11-28T18:35:15.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.872+0000 7fa669ffb640 1 -- 172.21.15.7:0/2897661060 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3497558975 0 0) 0x7fa654003530 con 0x7fa66c0cbb70 2023-11-28T18:35:15.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.872+0000 7fa66affd640 1 -- 172.21.15.7:0/2897661060 >> v1:172.21.15.193:6789/0 conn(0x7fa66c072450 legacy=0x7fa66c164990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:15.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.872+0000 7fa66affd640 1 -- 172.21.15.7:0/2897661060 >> v1:172.21.15.7:6789/0 conn(0x7fa66c0cbb70 legacy=0x7fa66c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:15.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.872+0000 7fa66affd640 1 -- 172.21.15.7:0/2897661060 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c0c4cd0 con 0x7fa66c1542e0 2023-11-28T18:35:15.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa66affd640 1 --2- 172.21.15.7:0/2897661060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c1542e0 0x7fa66c15e010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0097c0 tx=0x7fa65c01c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15c882cee6e614d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:15.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa669ffb640 1 -- 172.21.15.7:0/2897661060 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660001680 con 0x7fa66c1542e0 2023-11-28T18:35:15.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa669ffb640 1 -- 172.21.15.7:0/2897661060 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c0a7050 con 0x7fa66c1542e0 2023-11-28T18:35:15.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa669ffb640 1 -- 172.21.15.7:0/2897661060 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0ab620 con 0x7fa66c1542e0 2023-11-28T18:35:15.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 -- 172.21.15.7:0/2897661060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c1542e0 msgr2=0x7fa66c15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:15.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 --2- 172.21.15.7:0/2897661060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c1542e0 0x7fa66c15e010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0097c0 tx=0x7fa65c01c3a0 comp rx=0 tx=0).stop 2023-11-28T18:35:15.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 -- 172.21.15.7:0/2897661060 shutdown_connections 2023-11-28T18:35:15.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 --2- 172.21.15.7:0/2897661060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c1542e0 0x7fa66c15e010 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:15.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 -- 172.21.15.7:0/2897661060 >> 172.21.15.7:0/2897661060 conn(0x7fa66c0717e0 msgr2=0x7fa66c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:15.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 -- 172.21.15.7:0/2897661060 shutdown_connections 2023-11-28T18:35:15.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 -- 172.21.15.7:0/2897661060 wait complete. 2023-11-28T18:35:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 Processor -- start 2023-11-28T18:35:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.876+0000 7fa6712dd640 1 -- start start 2023-11-28T18:35:15.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c072450 0x7fa66c149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0cbb70 0x7fa66c1f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa66c1542e0 0x7fa66c1f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c072030 con 0x7fa66c0cbb70 2023-11-28T18:35:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c1f6570 con 0x7fa66c072450 2023-11-28T18:35:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c15dc40 con 0x7fa66c1542e0 2023-11-28T18:35:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0cbb70 0x7fa66c1f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0cbb70 0x7fa66c1f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56402/0 (socket says 172.21.15.7:56402) 2023-11-28T18:35:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 -- 172.21.15.7:0/3883635991 learned_addr learned my addr 172.21.15.7:0/3883635991 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:15.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66b7fe640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa66c1542e0 0x7fa66c1f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:15.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66affd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c072450 0x7fa66c149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 -- 172.21.15.7:0/3883635991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa66c1542e0 msgr2=0x7fa66c1f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa66c1542e0 0x7fa66c1f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 -- 172.21.15.7:0/3883635991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c072450 msgr2=0x7fa66c149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c072450 0x7fa66c149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 -- 172.21.15.7:0/3883635991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c009420 con 0x7fa66c0cbb70 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa66a7fc640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0cbb70 0x7fa66c1f6a00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa66000c730 tx=0x7fa66000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6537fe640 1 -- 172.21.15.7:0/3883635991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660005ae0 con 0x7fa66c0cbb70 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6537fe640 1 -- 172.21.15.7:0/3883635991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa660005c40 con 0x7fa66c0cbb70 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6537fe640 1 -- 172.21.15.7:0/3883635991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6600072b0 con 0x7fa66c0cbb70 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c15df00 con 0x7fa66c0cbb70 2023-11-28T18:35:15.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c1f6200 con 0x7fa66c0cbb70 2023-11-28T18:35:15.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.880+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66c06c670 con 0x7fa66c0cbb70 2023-11-28T18:35:15.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.884+0000 7fa6537fe640 1 -- 172.21.15.7:0/3883635991 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa660007410 con 0x7fa66c0cbb70 2023-11-28T18:35:15.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.884+0000 7fa6537fe640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa658066900 0x7fa658068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:15.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.884+0000 7fa6537fe640 1 -- 172.21.15.7:0/3883635991 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fa660083fc0 con 0x7fa66c0cbb70 2023-11-28T18:35:15.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.884+0000 7fa66affd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa658066900 0x7fa658068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:15.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.884+0000 7fa66affd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa658066900 0x7fa658068dc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0ad000 tx=0x7fa65c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:15.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:15.888+0000 7fa6537fe640 1 -- 172.21.15.7:0/3883635991 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa660050e90 con 0x7fa66c0cbb70 2023-11-28T18:35:16.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.136+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa66c1f66e0 con 0x7fa66c0cbb70 2023-11-28T18:35:16.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6537fe640 1 -- 172.21.15.7:0/3883635991 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fa660017070 con 0x7fa66c0cbb70 2023-11-28T18:35:16.141 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:16.141 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:12.676974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:resolve","state_seq":29,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:16.141 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 55 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa658066900 msgr2=0x7fa658068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa658066900 0x7fa658068dc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0ad000 tx=0x7fa65c008040 comp rx=0 tx=0).stop 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0cbb70 msgr2=0x7fa66c1f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0cbb70 0x7fa66c1f6a00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa66000c730 tx=0x7fa66000cbf0 comp rx=0 tx=0).stop 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 shutdown_connections 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa658066900 0x7fa658068dc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa66c1542e0 0x7fa66c1f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0cbb70 0x7fa66c1f6a00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:16.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 --2- 172.21.15.7:0/3883635991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66c072450 0x7fa66c149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:16.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 >> 172.21.15.7:0/3883635991 conn(0x7fa66c0717e0 msgr2=0x7fa66c0cdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:16.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 shutdown_connections 2023-11-28T18:35:16.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:16.140+0000 7fa6712dd640 1 -- 172.21.15.7:0/3883635991 wait complete. 2023-11-28T18:35:16.167 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:35:18.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:18.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039e103640 1 Processor -- start 2023-11-28T18:35:18.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039e103640 1 -- start start 2023-11-28T18:35:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039e103640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f03980c4c80 0x7f0398070820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039e103640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03981515d0 0x7f0398151a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039e103640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0398158a90 con 0x7f03980c5060 2023-11-28T18:35:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039e103640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0398151f40 con 0x7f03981515d0 2023-11-28T18:35:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039e103640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0398157ba0 con 0x7f03980c4c80 2023-11-28T18:35:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039d902640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03981515d0 0x7f0398151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039d902640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03981515d0 0x7f0398151a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56408/0 (socket says 172.21.15.7:56408) 2023-11-28T18:35:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.328+0000 7f039d902640 1 -- 172.21.15.7:0/1142598661 learned_addr learned my addr 172.21.15.7:0/1142598661 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:18.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f039d101640 1 --2- 172.21.15.7:0/1142598661 >> v2:172.21.15.193:3300/0 conn(0x7f03980c4c80 0x7f0398070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:18.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f039d902640 1 -- 172.21.15.7:0/1142598661 >> v2:172.21.15.193:3300/0 conn(0x7f03980c4c80 msgr2=0x7f0398070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:18.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f039d902640 1 --2- 172.21.15.7:0/1142598661 >> v2:172.21.15.193:3300/0 conn(0x7f03980c4c80 0x7f0398070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f039d902640 1 -- 172.21.15.7:0/1142598661 >> v1:172.21.15.193:6789/0 conn(0x7f03980c5060 legacy=0x7f039806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:18.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f039d902640 1 -- 172.21.15.7:0/1142598661 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0398157e80 con 0x7f03981515d0 2023-11-28T18:35:18.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f039d902640 1 --2- 172.21.15.7:0/1142598661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398151a00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f03980c8e80 tx=0x7f039001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf477c60df852ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:18.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f038ffff640 1 -- 172.21.15.7:0/1142598661 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900af400 con 0x7f03981515d0 2023-11-28T18:35:18.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f038ffff640 1 -- 172.21.15.7:0/1142598661 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03900ab050 con 0x7f03981515d0 2023-11-28T18:35:18.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.332+0000 7f038ffff640 1 -- 172.21.15.7:0/1142598661 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900af780 con 0x7f03981515d0 2023-11-28T18:35:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- 172.21.15.7:0/1142598661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 msgr2=0x7f0398151a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 --2- 172.21.15.7:0/1142598661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398151a00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f03980c8e80 tx=0x7f039001f1c0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- 172.21.15.7:0/1142598661 shutdown_connections 2023-11-28T18:35:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 --2- 172.21.15.7:0/1142598661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398151a00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 --2- 172.21.15.7:0/1142598661 >> v2:172.21.15.193:3300/0 conn(0x7f03980c4c80 0x7f0398070820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- 172.21.15.7:0/1142598661 >> 172.21.15.7:0/1142598661 conn(0x7f039806c370 msgr2=0x7f03980662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- 172.21.15.7:0/1142598661 shutdown_connections 2023-11-28T18:35:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- 172.21.15.7:0/1142598661 wait complete. 2023-11-28T18:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 Processor -- start 2023-11-28T18:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- start start 2023-11-28T18:35:18.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03980c4c80 0x7f03981521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03980c5060 0x7f03981541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039814abb0 con 0x7f03981515d0 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03980c9e20 con 0x7f03980c4c80 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039e103640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0398152880 con 0x7f03980c5060 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039d902640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039d902640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398154700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56420/0 (socket says 172.21.15.7:56420) 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039d902640 1 -- 172.21.15.7:0/204615020 learned_addr learned my addr 172.21.15.7:0/204615020 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039d101640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03980c4c80 0x7f03981521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:18.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.336+0000 7f039c900640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03980c5060 0x7f03981541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:18.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039d902640 1 -- 172.21.15.7:0/204615020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03980c5060 msgr2=0x7f03981541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:18.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039d902640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03980c5060 0x7f03981541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039d902640 1 -- 172.21.15.7:0/204615020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03980c4c80 msgr2=0x7f03981521f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039d902640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03980c4c80 0x7f03981521f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039d902640 1 -- 172.21.15.7:0/204615020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0390009b30 con 0x7f03981515d0 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039d902640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398154700 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f03900b2040 tx=0x7f039001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f038dffb640 1 -- 172.21.15.7:0/204615020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900afdc0 con 0x7f03981515d0 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f038dffb640 1 -- 172.21.15.7:0/204615020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0390009510 con 0x7f03981515d0 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03980c92a0 con 0x7f03981515d0 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f038dffb640 1 -- 172.21.15.7:0/204615020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900ba660 con 0x7f03981515d0 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03980c9860 con 0x7f03981515d0 2023-11-28T18:35:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03980ca350 con 0x7f03981515d0 2023-11-28T18:35:18.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.340+0000 7f038dffb640 1 -- 172.21.15.7:0/204615020 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f03900b6040 con 0x7f03981515d0 2023-11-28T18:35:18.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.344+0000 7f038dffb640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f037c0668d0 0x7f037c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.344+0000 7f038dffb640 1 -- 172.21.15.7:0/204615020 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f0390122750 con 0x7f03981515d0 2023-11-28T18:35:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.344+0000 7f039d101640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f037c0668d0 0x7f037c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.344+0000 7f039d101640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f037c0668d0 0x7f037c068d90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0394006aa0 tx=0x7f0394008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:18.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.348+0000 7f038dffb640 1 -- 172.21.15.7:0/204615020 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03980ca350 con 0x7f03981515d0 2023-11-28T18:35:18.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03980ca350 con 0x7f03981515d0 2023-11-28T18:35:18.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f038dffb640 1 -- 172.21.15.7:0/204615020 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f03980ca350 con 0x7f03981515d0 2023-11-28T18:35:18.597 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:18.597 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:12.676974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:resolve","state_seq":29,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:18.598 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 55 2023-11-28T18:35:18.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f037c0668d0 msgr2=0x7f037c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:18.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f037c0668d0 0x7f037c068d90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0394006aa0 tx=0x7f0394008040 comp rx=0 tx=0).stop 2023-11-28T18:35:18.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 msgr2=0x7f0398154700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398154700 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f03900b2040 tx=0x7f039001bf60 comp rx=0 tx=0).stop 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 shutdown_connections 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f037c0668d0 0x7f037c068d90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03981515d0 0x7f0398154700 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03980c5060 0x7f03981541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 --2- 172.21.15.7:0/204615020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03980c4c80 0x7f03981521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 >> 172.21.15.7:0/204615020 conn(0x7f039806c370 msgr2=0x7f039814c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 shutdown_connections 2023-11-28T18:35:18.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:18.596+0000 7f039e103640 1 -- 172.21.15.7:0/204615020 wait complete. 2023-11-28T18:35:18.621 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:35:20.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:20.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146d9b5640 1 Processor -- start 2023-11-28T18:35:20.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146d9b5640 1 -- start start 2023-11-28T18:35:20.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146d9b5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f146806ac30 0x7f14680cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146d9b5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1468154110 0x7f1468159c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146d9b5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14680cc5f0 con 0x7f146806a850 2023-11-28T18:35:20.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146d9b5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14680c82b0 con 0x7f1468154110 2023-11-28T18:35:20.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146d9b5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14680c6690 con 0x7f146806ac30 2023-11-28T18:35:20.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146c9b3640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f146806a850 0x7f1468153a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39136/0 (socket says 172.21.15.7:39136) 2023-11-28T18:35:20.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146c9b3640 1 -- 172.21.15.7:0/775522046 learned_addr learned my addr 172.21.15.7:0/775522046 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:35:20.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f1467fff640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f146806ac30 0x7f14680cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f146d1b4640 1 --2- 172.21.15.7:0/775522046 >> v2:172.21.15.193:3300/0 conn(0x7f1468154110 0x7f1468159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550940654 0 0) 0x7f14680cc5f0 con 0x7f146806a850 2023-11-28T18:35:20.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.788+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1458003530 con 0x7f146806a850 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3040311807 0 0) 0x7f1458003530 con 0x7f146806a850 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f146806ac30 msgr2=0x7f14680cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f146806ac30 0x7f14680cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 >> v2:172.21.15.193:3300/0 conn(0x7f1468154110 msgr2=0x7f1468159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 --2- 172.21.15.7:0/775522046 >> v2:172.21.15.193:3300/0 conn(0x7f1468154110 0x7f1468159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14680c79c0 con 0x7f146806a850 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f145c001680 con 0x7f146806a850 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 >> v1:172.21.15.7:6789/0 conn(0x7f146806a850 legacy=0x7f1468153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1458003980 0x7f1458003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f146806ac30 0x7f14580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1458004d10 0x7f1458005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680c82b0 con 0x7f1458004d10 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680c6690 con 0x7f1458003980 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145800b990 con 0x7f146806ac30 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f146d1b4640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1458004d10 0x7f1458005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f1467fff640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f146806ac30 0x7f14580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f146c9b3640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1458003980 0x7f1458003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f146d1b4640 1 -- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f146806ac30 msgr2=0x7f14580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f146d1b4640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f146806ac30 0x7f14580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f146d1b4640 1 -- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1458003980 msgr2=0x7f1458003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f146d1b4640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1458003980 0x7f1458003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:20.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.792+0000 7f146d1b4640 1 -- 172.21.15.7:0/775522046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458003530 con 0x7f1458004d10 2023-11-28T18:35:20.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d1b4640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1458004d10 0x7f1458005190 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1460009f90 tx=0x7f146001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fb340eb242ca9d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:20.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14600a5c70 con 0x7f1458004d10 2023-11-28T18:35:20.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146000f040 con 0x7f1458004d10 2023-11-28T18:35:20.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f14677fe640 1 -- 172.21.15.7:0/775522046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14600ad440 con 0x7f1458004d10 2023-11-28T18:35:20.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d9b5640 1 -- 172.21.15.7:0/775522046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1458004d10 msgr2=0x7f1458005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:20.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d9b5640 1 --2- 172.21.15.7:0/775522046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1458004d10 0x7f1458005190 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1460009f90 tx=0x7f146001cc60 comp rx=0 tx=0).stop 2023-11-28T18:35:20.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d1b4640 1 -- 172.21.15.7:0/775522046 reap_dead start 2023-11-28T18:35:20.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d9b5640 1 -- 172.21.15.7:0/775522046 shutdown_connections 2023-11-28T18:35:20.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d9b5640 1 -- 172.21.15.7:0/775522046 >> 172.21.15.7:0/775522046 conn(0x7f146806e900 msgr2=0x7f14680cbc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:20.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d9b5640 1 -- 172.21.15.7:0/775522046 shutdown_connections 2023-11-28T18:35:20.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d9b5640 1 -- 172.21.15.7:0/775522046 wait complete. 2023-11-28T18:35:20.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d9b5640 1 Processor -- start 2023-11-28T18:35:20.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.796+0000 7f146d9b5640 1 -- start start 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1458004d10 0x7f14681e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1468071cc0 0x7f14681e22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14680b0390 0x7f146814f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1468071170 con 0x7f14680b0390 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1468072da0 con 0x7f1468071cc0 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14680b0c90 con 0x7f1458004d10 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14680b0390 0x7f146814f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146c9b3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1458004d10 0x7f14681e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14680b0390 0x7f146814f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56450/0 (socket says 172.21.15.7:56450) 2023-11-28T18:35:20.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 -- 172.21.15.7:0/3514858393 learned_addr learned my addr 172.21.15.7:0/3514858393 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:20.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f1467fff640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1468071cc0 0x7f14681e22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 -- 172.21.15.7:0/3514858393 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1458004d10 msgr2=0x7f14681e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1458004d10 0x7f14681e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 -- 172.21.15.7:0/3514858393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1468071cc0 msgr2=0x7f14681e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1468071cc0 0x7f14681e22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 -- 172.21.15.7:0/3514858393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14680b0f50 con 0x7f14680b0390 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d1b4640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14680b0390 0x7f146814f570 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1460002240 tx=0x7f1460009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f14657fa640 1 -- 172.21.15.7:0/3514858393 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14600039a0 con 0x7f14680b0390 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f14657fa640 1 -- 172.21.15.7:0/3514858393 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1460003b00 con 0x7f14680b0390 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f14657fa640 1 -- 172.21.15.7:0/3514858393 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460003e90 con 0x7f14680b0390 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1468072750 con 0x7f14680b0390 2023-11-28T18:35:20.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14681e2810 con 0x7f14680b0390 2023-11-28T18:35:20.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.800+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14680725b0 con 0x7f14680b0390 2023-11-28T18:35:20.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.804+0000 7f14657fa640 1 -- 172.21.15.7:0/3514858393 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f14600b4040 con 0x7f14680b0390 2023-11-28T18:35:20.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.804+0000 7f14657fa640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f144c066900 0x7f144c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:20.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.804+0000 7f14657fa640 1 -- 172.21.15.7:0/3514858393 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f146011e580 con 0x7f14680b0390 2023-11-28T18:35:20.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.804+0000 7f146c9b3640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f144c066900 0x7f144c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:20.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.804+0000 7f146c9b3640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f144c066900 0x7f144c068dc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f145c000c10 tx=0x7f145c001e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:20.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:20.808+0000 7f14657fa640 1 -- 172.21.15.7:0/3514858393 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f14680725b0 con 0x7f14680b0390 2023-11-28T18:35:21.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.024+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14680725b0 con 0x7f14680b0390 2023-11-28T18:35:21.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f14657fa640 1 -- 172.21.15.7:0/3514858393 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f14680725b0 con 0x7f14680b0390 2023-11-28T18:35:21.028 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:21.028 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:19.022923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:21.029 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 56 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f144c066900 msgr2=0x7f144c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f144c066900 0x7f144c068dc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f145c000c10 tx=0x7f145c001e00 comp rx=0 tx=0).stop 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14680b0390 msgr2=0x7f146814f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14680b0390 0x7f146814f570 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1460002240 tx=0x7f1460009750 comp rx=0 tx=0).stop 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 shutdown_connections 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f144c066900 0x7f144c068dc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14680b0390 0x7f146814f570 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1468071cc0 0x7f14681e22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 --2- 172.21.15.7:0/3514858393 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1458004d10 0x7f14681e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 >> 172.21.15.7:0/3514858393 conn(0x7f146806e900 msgr2=0x7f14680c0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 shutdown_connections 2023-11-28T18:35:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:21.028+0000 7f146d9b5640 1 -- 172.21.15.7:0/3514858393 wait complete. 2023-11-28T18:35:21.055 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4978": { "addr": "172.21.15.193:6836/1685294311", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1685294311, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1685294311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4978, "incarnation": 28, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 33 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:35:19.022923+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5359, "mds_2": 5307, "mds_3": 5086, "mds_4": 4978 } } } ], "standbys": [] } 2023-11-28T18:35:23.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:23.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.220+0000 7fdd7dff2640 1 Processor -- start 2023-11-28T18:35:23.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.220+0000 7fdd7dff2640 1 -- start start 2023-11-28T18:35:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7dff2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdd7815ca30 0x7fdd78159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7dff2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd780cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7dff2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7815a620 con 0x7fdd7815ce10 2023-11-28T18:35:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7dff2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7815a900 con 0x7fdd7815ca30 2023-11-28T18:35:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7dff2640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7815abe0 con 0x7fdd780cafe0 2023-11-28T18:35:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7cff0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdd7815ca30 0x7fdd78159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7cff0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdd7815ca30 0x7fdd78159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56460/0 (socket says 172.21.15.7:56460) 2023-11-28T18:35:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7cff0640 1 -- 172.21.15.7:0/686314310 learned_addr learned my addr 172.21.15.7:0/686314310 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7cff0640 1 -- 172.21.15.7:0/686314310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 msgr2=0x7fdd780cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7d7f1640 1 --2- 172.21.15.7:0/686314310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd780cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7cff0640 1 --2- 172.21.15.7:0/686314310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd780cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7cff0640 1 -- 172.21.15.7:0/686314310 >> v1:172.21.15.193:6789/0 conn(0x7fdd7815ce10 legacy=0x7fdd780ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7cff0640 1 -- 172.21.15.7:0/686314310 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd780cbb10 con 0x7fdd7815ca30 2023-11-28T18:35:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.224+0000 7fdd7d7f1640 1 --2- 172.21.15.7:0/686314310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd780cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:35:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7cff0640 1 --2- 172.21.15.7:0/686314310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ca30 0x7fdd78159c90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009870 tx=0x7fdd6c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48cab372c4a6fe1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd777fe640 1 -- 172.21.15.7:0/686314310 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0a89b0 con 0x7fdd7815ca30 2023-11-28T18:35:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd777fe640 1 -- 172.21.15.7:0/686314310 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c0a8b10 con 0x7fdd7815ca30 2023-11-28T18:35:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd777fe640 1 -- 172.21.15.7:0/686314310 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0a8ea0 con 0x7fdd7815ca30 2023-11-28T18:35:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 -- 172.21.15.7:0/686314310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ca30 msgr2=0x7fdd78159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/686314310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ca30 0x7fdd78159c90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009870 tx=0x7fdd6c01ecf0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 -- 172.21.15.7:0/686314310 shutdown_connections 2023-11-28T18:35:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/686314310 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd780cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/686314310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ca30 0x7fdd78159c90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 -- 172.21.15.7:0/686314310 >> 172.21.15.7:0/686314310 conn(0x7fdd7806c060 msgr2=0x7fdd780662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 -- 172.21.15.7:0/686314310 shutdown_connections 2023-11-28T18:35:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 -- 172.21.15.7:0/686314310 wait complete. 2023-11-28T18:35:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 Processor -- start 2023-11-28T18:35:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.228+0000 7fdd7dff2640 1 -- start start 2023-11-28T18:35:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd781561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd7815ca30 0x7fdd78156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ce10 0x7fdd781f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd780c6ed0 con 0x7fdd7815ce10 2023-11-28T18:35:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd78155770 con 0x7fdd7815ca30 2023-11-28T18:35:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd780c68e0 con 0x7fdd780cafe0 2023-11-28T18:35:23.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7d7f1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ce10 0x7fdd781f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:23.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7cff0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd781561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:23.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7d7f1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ce10 0x7fdd781f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56468/0 (socket says 172.21.15.7:56468) 2023-11-28T18:35:23.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7d7f1640 1 -- 172.21.15.7:0/3867575075 learned_addr learned my addr 172.21.15.7:0/3867575075 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:23.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd77fff640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd7815ca30 0x7fdd78156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7cff0640 1 -- 172.21.15.7:0/3867575075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd7815ca30 msgr2=0x7fdd78156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7cff0640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd7815ca30 0x7fdd78156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7cff0640 1 -- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ce10 msgr2=0x7fdd781f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7cff0640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ce10 0x7fdd781f2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7cff0640 1 -- 172.21.15.7:0/3867575075 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c0094d0 con 0x7fdd780cafe0 2023-11-28T18:35:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7cff0640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd781561f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0af000 tx=0x7fdd6c01ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd757fa640 1 -- 172.21.15.7:0/3867575075 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0a89b0 con 0x7fdd780cafe0 2023-11-28T18:35:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd757fa640 1 -- 172.21.15.7:0/3867575075 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c0bb480 con 0x7fdd780cafe0 2023-11-28T18:35:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd781551b0 con 0x7fdd780cafe0 2023-11-28T18:35:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd757fa640 1 -- 172.21.15.7:0/3867575075 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0bb830 con 0x7fdd780cafe0 2023-11-28T18:35:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd780c6af0 con 0x7fdd780cafe0 2023-11-28T18:35:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.232+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7804e890 con 0x7fdd780cafe0 2023-11-28T18:35:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.236+0000 7fdd757fa640 1 -- 172.21.15.7:0/3867575075 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fdd6c0bbba0 con 0x7fdd780cafe0 2023-11-28T18:35:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.236+0000 7fdd757fa640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdd5c0668d0 0x7fdd5c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.236+0000 7fdd757fa640 1 -- 172.21.15.7:0/3867575075 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fdd6c123300 con 0x7fdd780cafe0 2023-11-28T18:35:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.236+0000 7fdd77fff640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdd5c0668d0 0x7fdd5c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.236+0000 7fdd77fff640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdd5c0668d0 0x7fdd5c068d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdd780c2d50 tx=0x7fdd64009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:23.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.240+0000 7fdd757fa640 1 -- 172.21.15.7:0/3867575075 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdd6c0f0500 con 0x7fdd780cafe0 2023-11-28T18:35:23.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.476+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd7804e890 con 0x7fdd780cafe0 2023-11-28T18:35:23.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd757fa640 1 -- 172.21.15.7:0/3867575075 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5771 (secure 0 0 0) 0x7fdd6c0f3b20 con 0x7fdd780cafe0 2023-11-28T18:35:23.480 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:23.481 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:19.022923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:23.481 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 56 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdd5c0668d0 msgr2=0x7fdd5c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdd5c0668d0 0x7fdd5c068d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdd780c2d50 tx=0x7fdd64009060 comp rx=0 tx=0).stop 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 msgr2=0x7fdd781561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd781561f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0af000 tx=0x7fdd6c01ba90 comp rx=0 tx=0).stop 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 shutdown_connections 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdd5c0668d0 0x7fdd5c068d90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7815ce10 0x7fdd781f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd7815ca30 0x7fdd78156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 --2- 172.21.15.7:0/3867575075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdd780cafe0 0x7fdd781561f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 >> 172.21.15.7:0/3867575075 conn(0x7fdd7806c060 msgr2=0x7fdd78158d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 shutdown_connections 2023-11-28T18:35:23.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:23.480+0000 7fdd7dff2640 1 -- 172.21.15.7:0/3867575075 wait complete. 2023-11-28T18:35:23.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:35:25.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fda6d6640 1 Processor -- start 2023-11-28T18:35:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fda6d6640 1 -- start start 2023-11-28T18:35:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fda6d6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd40c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fda6d6640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8fd41575c0 0x7f8fd40c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fda6d6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40c6140 con 0x7f8fd40c4020 2023-11-28T18:35:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fda6d6640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40c72b0 con 0x7f8fd41575c0 2023-11-28T18:35:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fda6d6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4070c00 con 0x7f8fd4150b30 2023-11-28T18:35:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd37fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8fd41575c0 0x7f8fd40c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd40c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd37fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8fd41575c0 0x7f8fd40c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:42976/0 (socket says 172.21.15.7:42976) 2023-11-28T18:35:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd37fe640 1 -- 172.21.15.7:0/439648709 learned_addr learned my addr 172.21.15.7:0/439648709 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd40c24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:60174/0 (socket says 172.21.15.7:60174) 2023-11-28T18:35:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd37fe640 1 -- 172.21.15.7:0/439648709 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 msgr2=0x7f8fd40c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd37fe640 1 --2- 172.21.15.7:0/439648709 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd40c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd37fe640 1 -- 172.21.15.7:0/439648709 >> v1:172.21.15.193:6789/0 conn(0x7f8fd40c4020 legacy=0x7f8fd40c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd37fe640 1 -- 172.21.15.7:0/439648709 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40c6370 con 0x7f8fd41575c0 2023-11-28T18:35:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.660+0000 7f8fd3fff640 1 --2- 172.21.15.7:0/439648709 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd40c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:35:25.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fd37fe640 1 --2- 172.21.15.7:0/439648709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd41575c0 0x7f8fd40c3ae0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009870 tx=0x7f8fc801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5539b1a9a8987583 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fd2ffd640 1 -- 172.21.15.7:0/439648709 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80a8a30 con 0x7f8fd41575c0 2023-11-28T18:35:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fd2ffd640 1 -- 172.21.15.7:0/439648709 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc80a8b90 con 0x7f8fd41575c0 2023-11-28T18:35:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fd2ffd640 1 -- 172.21.15.7:0/439648709 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80b0440 con 0x7f8fd41575c0 2023-11-28T18:35:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fda6d6640 1 -- 172.21.15.7:0/439648709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd41575c0 msgr2=0x7f8fd40c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/439648709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd41575c0 0x7f8fd40c3ae0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009870 tx=0x7f8fc801ed30 comp rx=0 tx=0).stop 2023-11-28T18:35:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fda6d6640 1 -- 172.21.15.7:0/439648709 shutdown_connections 2023-11-28T18:35:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/439648709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd41575c0 0x7f8fd40c3ae0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/439648709 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd40c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fda6d6640 1 -- 172.21.15.7:0/439648709 >> 172.21.15.7:0/439648709 conn(0x7f8fd40662e0 msgr2=0x7f8fd40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fda6d6640 1 -- 172.21.15.7:0/439648709 shutdown_connections 2023-11-28T18:35:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.664+0000 7f8fda6d6640 1 -- 172.21.15.7:0/439648709 wait complete. 2023-11-28T18:35:25.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 Processor -- start 2023-11-28T18:35:25.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 -- start start 2023-11-28T18:35:25.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd40c4020 0x7f8fd4145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:25.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd41eef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fd41575c0 0x7f8fd4154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41534e0 con 0x7f8fd41575c0 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41f3a10 con 0x7f8fd40c4020 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41464a0 con 0x7f8fd4150b30 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fd41575c0 0x7f8fd4154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fd41575c0 0x7f8fd4154010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60034/0 (socket says 172.21.15.7:60034) 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 -- 172.21.15.7:0/1371273101 learned_addr learned my addr 172.21.15.7:0/1371273101 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd3fff640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd40c4020 0x7f8fd4145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:25.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd37fe640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd41eef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:25.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 -- 172.21.15.7:0/1371273101 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 msgr2=0x7f8fd41eef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:25.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd41eef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 -- 172.21.15.7:0/1371273101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd40c4020 msgr2=0x7f8fd4145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:25.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd40c4020 0x7f8fd4145f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 -- 172.21.15.7:0/1371273101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4009420 con 0x7f8fd41575c0 2023-11-28T18:35:25.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd8c73640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fd41575c0 0x7f8fd4154010 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c770 tx=0x7f8fcc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:25.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd0ff9640 1 -- 172.21.15.7:0/1371273101 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc003b80 con 0x7f8fd41575c0 2023-11-28T18:35:25.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd0ff9640 1 -- 172.21.15.7:0/1371273101 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc003ce0 con 0x7f8fd41575c0 2023-11-28T18:35:25.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc80094d0 con 0x7f8fd41575c0 2023-11-28T18:35:25.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fd0ff9640 1 -- 172.21.15.7:0/1371273101 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc00aa90 con 0x7f8fd41575c0 2023-11-28T18:35:25.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.668+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd41eeae0 con 0x7f8fd41575c0 2023-11-28T18:35:25.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.672+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4154550 con 0x7f8fd41575c0 2023-11-28T18:35:25.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.672+0000 7f8fd0ff9640 1 -- 172.21.15.7:0/1371273101 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8fcc00abf0 con 0x7f8fd41575c0 2023-11-28T18:35:25.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.672+0000 7f8fd0ff9640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8fc0066900 0x7f8fc0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:25.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.672+0000 7f8fd0ff9640 1 -- 172.21.15.7:0/1371273101 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f8fcc084380 con 0x7f8fd41575c0 2023-11-28T18:35:25.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.672+0000 7f8fd3fff640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8fc0066900 0x7f8fc0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:25.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.672+0000 7f8fd3fff640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8fc0066900 0x7f8fc0068dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009cf0 tx=0x7f8fc4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:25.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.676+0000 7f8fd0ff9640 1 -- 172.21.15.7:0/1371273101 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8fcc0513e0 con 0x7f8fd41575c0 2023-11-28T18:35:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fd41eec70 con 0x7f8fd41575c0 2023-11-28T18:35:25.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fd0ff9640 1 -- 172.21.15.7:0/1371273101 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5768 (secure 0 0 0) 0x7f8fcc054a00 con 0x7f8fd41575c0 2023-11-28T18:35:25.870 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:25.870 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:24.439054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:25.870 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 57 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8fc0066900 msgr2=0x7f8fc0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8fc0066900 0x7f8fc0068dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009cf0 tx=0x7f8fc4009190 comp rx=0 tx=0).stop 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fd41575c0 msgr2=0x7f8fd4154010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fd41575c0 0x7f8fd4154010 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c770 tx=0x7f8fcc00cc30 comp rx=0 tx=0).stop 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 shutdown_connections 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8fc0066900 0x7f8fc0068dc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fd41575c0 0x7f8fd4154010 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8fd4150b30 0x7f8fd41eef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 --2- 172.21.15.7:0/1371273101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fd40c4020 0x7f8fd4145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 >> 172.21.15.7:0/1371273101 conn(0x7f8fd40662e0 msgr2=0x7f8fd414e010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 shutdown_connections 2023-11-28T18:35:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:25.868+0000 7f8fda6d6640 1 -- 172.21.15.7:0/1371273101 wait complete. 2023-11-28T18:35:25.891 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:35:27.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:28.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.044+0000 7fe40646b640 1 Processor -- start 2023-11-28T18:35:28.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.044+0000 7fe40646b640 1 -- start start 2023-11-28T18:35:28.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe40646b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe400072950 0x7fe400074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:28.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe40646b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe40006c920 0x7fe40006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:28.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe40646b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4000c8490 con 0x7fe400073070 2023-11-28T18:35:28.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe40646b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40006d2e0 con 0x7fe40006c920 2023-11-28T18:35:28.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe40646b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4000c81f0 con 0x7fe400072950 2023-11-28T18:35:28.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe404a08640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe40006c920 0x7fe40006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:28.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe404a08640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe40006c920 0x7fe40006cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60040/0 (socket says 172.21.15.7:60040) 2023-11-28T18:35:28.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe404a08640 1 -- 172.21.15.7:0/3731102189 learned_addr learned my addr 172.21.15.7:0/3731102189 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:28.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe3ff7fe640 1 --1- 172.21.15.7:0/3731102189 >> v1:172.21.15.7:6789/0 conn(0x7fe400073070 0x7fe4000c19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48246/0 (socket says 172.21.15.7:48246) 2023-11-28T18:35:28.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe3fffff640 1 --2- 172.21.15.7:0/3731102189 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe400072950 0x7fe400074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:28.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe404a08640 1 -- 172.21.15.7:0/3731102189 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe400072950 msgr2=0x7fe400074030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:28.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe404a08640 1 --2- 172.21.15.7:0/3731102189 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe400072950 0x7fe400074030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe404a08640 1 -- 172.21.15.7:0/3731102189 >> v1:172.21.15.7:6789/0 conn(0x7fe400073070 legacy=0x7fe4000c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:28.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe3feffd640 1 -- 172.21.15.7:0/3731102189 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542785832 0 0) 0x7fe4000c8490 con 0x7fe400073070 2023-11-28T18:35:28.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.048+0000 7fe404a08640 1 -- 172.21.15.7:0/3731102189 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40015ab00 con 0x7fe40006c920 2023-11-28T18:35:28.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe404a08640 1 --2- 172.21.15.7:0/3731102189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe40006c920 0x7fe40006cda0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e50 tx=0x7fe3f801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3029cc414fd680f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:28.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe3feffd640 1 -- 172.21.15.7:0/3731102189 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f80a5590 con 0x7fe40006c920 2023-11-28T18:35:28.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe3feffd640 1 -- 172.21.15.7:0/3731102189 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f80a56f0 con 0x7fe40006c920 2023-11-28T18:35:28.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe3feffd640 1 -- 172.21.15.7:0/3731102189 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f80a5a80 con 0x7fe40006c920 2023-11-28T18:35:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 -- 172.21.15.7:0/3731102189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe40006c920 msgr2=0x7fe40006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 --2- 172.21.15.7:0/3731102189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe40006c920 0x7fe40006cda0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e50 tx=0x7fe3f801c8f0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 -- 172.21.15.7:0/3731102189 shutdown_connections 2023-11-28T18:35:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 --2- 172.21.15.7:0/3731102189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe40006c920 0x7fe40006cda0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 --2- 172.21.15.7:0/3731102189 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe400072950 0x7fe400074030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 -- 172.21.15.7:0/3731102189 >> 172.21.15.7:0/3731102189 conn(0x7fe4000cbf80 msgr2=0x7fe4000cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 -- 172.21.15.7:0/3731102189 shutdown_connections 2023-11-28T18:35:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 -- 172.21.15.7:0/3731102189 wait complete. 2023-11-28T18:35:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 Processor -- start 2023-11-28T18:35:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.052+0000 7fe40646b640 1 -- start start 2023-11-28T18:35:28.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe40006c920 0x7fe40015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:28.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400072950 0x7fe400146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe400073070 0x7fe40014fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40015d3e0 con 0x7fe400072950 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40014dc20 con 0x7fe400073070 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40014e430 con 0x7fe40006c920 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400072950 0x7fe400146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400072950 0x7fe400146450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60052/0 (socket says 172.21.15.7:60052) 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 -- 172.21.15.7:0/2851555339 learned_addr learned my addr 172.21.15.7:0/2851555339 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3fffff640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe40006c920 0x7fe40015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe404a08640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe400073070 0x7fe40014fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:28.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 -- 172.21.15.7:0/2851555339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe40006c920 msgr2=0x7fe40015bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:28.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe40006c920 0x7fe40015bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 -- 172.21.15.7:0/2851555339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe400073070 msgr2=0x7fe40014fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:28.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe400073070 0x7fe40014fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 -- 172.21.15.7:0/2851555339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8009b30 con 0x7fe400072950 2023-11-28T18:35:28.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3ff7fe640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400072950 0x7fe400146450 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c450 tx=0x7fe3f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:28.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3fcff9640 1 -- 172.21.15.7:0/2851555339 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40074e0 con 0x7fe400072950 2023-11-28T18:35:28.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3fcff9640 1 -- 172.21.15.7:0/2851555339 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f4007640 con 0x7fe400072950 2023-11-28T18:35:28.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe3fcff9640 1 -- 172.21.15.7:0/2851555339 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40049f0 con 0x7fe400072950 2023-11-28T18:35:28.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40014f110 con 0x7fe400072950 2023-11-28T18:35:28.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40014e150 con 0x7fe400072950 2023-11-28T18:35:28.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.056+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40004e890 con 0x7fe400072950 2023-11-28T18:35:28.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.060+0000 7fe3fcff9640 1 -- 172.21.15.7:0/2851555339 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe3f4004b50 con 0x7fe400072950 2023-11-28T18:35:28.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.060+0000 7fe3fcff9640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3ec0668d0 0x7fe3ec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:28.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.060+0000 7fe3fcff9640 1 -- 172.21.15.7:0/2851555339 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fe3f4083810 con 0x7fe400072950 2023-11-28T18:35:28.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.060+0000 7fe3fffff640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3ec0668d0 0x7fe3ec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:28.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.060+0000 7fe3fffff640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3ec0668d0 0x7fe3ec068d90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0007a60 tx=0x7fe3f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:28.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.064+0000 7fe3fcff9640 1 -- 172.21.15.7:0/2851555339 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe3f4050870 con 0x7fe400072950 2023-11-28T18:35:28.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.252+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe40014e7b0 con 0x7fe400072950 2023-11-28T18:35:28.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe3fcff9640 1 -- 172.21.15.7:0/2851555339 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5768 (secure 0 0 0) 0x7fe3f4018090 con 0x7fe400072950 2023-11-28T18:35:28.256 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:28.256 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:24.439054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:28.257 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 57 2023-11-28T18:35:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3ec0668d0 msgr2=0x7fe3ec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3ec0668d0 0x7fe3ec068d90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0007a60 tx=0x7fe3f0008040 comp rx=0 tx=0).stop 2023-11-28T18:35:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400072950 msgr2=0x7fe400146450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400072950 0x7fe400146450 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c450 tx=0x7fe3f400c910 comp rx=0 tx=0).stop 2023-11-28T18:35:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 shutdown_connections 2023-11-28T18:35:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3ec0668d0 0x7fe3ec068d90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe400073070 0x7fe40014fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400072950 0x7fe400146450 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 --2- 172.21.15.7:0/2851555339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe40006c920 0x7fe40015bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 >> 172.21.15.7:0/2851555339 conn(0x7fe4000cbf80 msgr2=0x7fe4000caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 shutdown_connections 2023-11-28T18:35:28.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:28.256+0000 7fe40646b640 1 -- 172.21.15.7:0/2851555339 wait complete. 2023-11-28T18:35:28.278 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:35:30.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ef05a640 1 Processor -- start 2023-11-28T18:35:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ef05a640 1 -- start start 2023-11-28T18:35:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ef05a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f21e8150b30 0x7f21e80c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ef05a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f21e81575c0 0x7f21e80c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ef05a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e80c4020 0x7f21e80c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ef05a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e80c72b0 con 0x7f21e8150b30 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ef05a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e8070c00 con 0x7f21e81575c0 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ef05a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e80c6140 con 0x7f21e80c4020 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ecdf6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f21e8150b30 0x7f21e80c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ecdf6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f21e8150b30 0x7f21e80c24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60062/0 (socket says 172.21.15.7:60062) 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ecdf6640 1 -- 172.21.15.7:0/2748723895 learned_addr learned my addr 172.21.15.7:0/2748723895 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ecdf6640 1 -- 172.21.15.7:0/2748723895 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e80c4020 msgr2=0x7f21e80c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ecdf6640 1 --2- 172.21.15.7:0/2748723895 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e80c4020 0x7f21e80c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ecdf6640 1 -- 172.21.15.7:0/2748723895 >> v2:172.21.15.193:3300/0 conn(0x7f21e81575c0 msgr2=0x7f21e80c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ecdf6640 1 --2- 172.21.15.7:0/2748723895 >> v2:172.21.15.193:3300/0 conn(0x7f21e81575c0 0x7f21e80c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.432+0000 7f21ecdf6640 1 -- 172.21.15.7:0/2748723895 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e80c6370 con 0x7f21e8150b30 2023-11-28T18:35:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.436+0000 7f21ecdf6640 1 --2- 172.21.15.7:0/2748723895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e8150b30 0x7f21e80c24f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f21e806fee0 tx=0x7f21dc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87f1cf22091e429 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.436+0000 7f21e77fe640 1 -- 172.21.15.7:0/2748723895 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0a89b0 con 0x7f21e8150b30 2023-11-28T18:35:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.436+0000 7f21e77fe640 1 -- 172.21.15.7:0/2748723895 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0a8b10 con 0x7f21e8150b30 2023-11-28T18:35:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.436+0000 7f21e77fe640 1 -- 172.21.15.7:0/2748723895 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0a8ea0 con 0x7f21e8150b30 2023-11-28T18:35:30.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.436+0000 7f21ef05a640 1 -- 172.21.15.7:0/2748723895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e8150b30 msgr2=0x7f21e80c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:30.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.436+0000 7f21ef05a640 1 --2- 172.21.15.7:0/2748723895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e8150b30 0x7f21e80c24f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f21e806fee0 tx=0x7f21dc01ecf0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 -- 172.21.15.7:0/2748723895 shutdown_connections 2023-11-28T18:35:30.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 --2- 172.21.15.7:0/2748723895 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e80c4020 0x7f21e80c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 --2- 172.21.15.7:0/2748723895 >> v2:172.21.15.193:3300/0 conn(0x7f21e81575c0 0x7f21e80c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 --2- 172.21.15.7:0/2748723895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e8150b30 0x7f21e80c24f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 -- 172.21.15.7:0/2748723895 >> 172.21.15.7:0/2748723895 conn(0x7f21e80662e0 msgr2=0x7f21e80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 -- 172.21.15.7:0/2748723895 shutdown_connections 2023-11-28T18:35:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 -- 172.21.15.7:0/2748723895 wait complete. 2023-11-28T18:35:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 Processor -- start 2023-11-28T18:35:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 -- start start 2023-11-28T18:35:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e80c4020 0x7f21e81f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8150b30 0x7f21e81f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e81575c0 0x7f21e81f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e814cc40 con 0x7f21e80c4020 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e8156db0 con 0x7f21e8150b30 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ef05a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e8157090 con 0x7f21e81575c0 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ecdf6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e80c4020 0x7f21e81f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ecdf6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e80c4020 0x7f21e81f0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60066/0 (socket says 172.21.15.7:60066) 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ecdf6640 1 -- 172.21.15.7:0/37418680 learned_addr learned my addr 172.21.15.7:0/37418680 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21e7fff640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8150b30 0x7f21e81f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ed5f7640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e81575c0 0x7f21e81f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ecdf6640 1 -- 172.21.15.7:0/37418680 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e81575c0 msgr2=0x7f21e81f33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ecdf6640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e81575c0 0x7f21e81f33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ecdf6640 1 -- 172.21.15.7:0/37418680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8150b30 msgr2=0x7f21e81f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ecdf6640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8150b30 0x7f21e81f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.440+0000 7f21ecdf6640 1 -- 172.21.15.7:0/37418680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc0094d0 con 0x7f21e80c4020 2023-11-28T18:35:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21ecdf6640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e80c4020 0x7f21e81f0b40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f21dc01c920 tx=0x7f21dc01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21e57fa640 1 -- 172.21.15.7:0/37418680 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b0e20 con 0x7f21e80c4020 2023-11-28T18:35:30.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21e57fa640 1 -- 172.21.15.7:0/37418680 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ba440 con 0x7f21e80c4020 2023-11-28T18:35:30.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21e57fa640 1 -- 172.21.15.7:0/37418680 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba7f0 con 0x7f21e80c4020 2023-11-28T18:35:30.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e814ce90 con 0x7f21e80c4020 2023-11-28T18:35:30.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e814d450 con 0x7f21e80c4020 2023-11-28T18:35:30.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e814d640 con 0x7f21e80c4020 2023-11-28T18:35:30.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21e57fa640 1 -- 172.21.15.7:0/37418680 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f21dc0ba950 con 0x7f21e80c4020 2023-11-28T18:35:30.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21e57fa640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21cc0668d0 0x7f21cc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:30.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21e57fa640 1 -- 172.21.15.7:0/37418680 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f21dc122690 con 0x7f21e80c4020 2023-11-28T18:35:30.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.444+0000 7f21e7fff640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21cc0668d0 0x7f21cc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.448+0000 7f21e7fff640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21cc0668d0 0x7f21cc068d90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f21d4006fd0 tx=0x7f21d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.448+0000 7f21e57fa640 1 -- 172.21.15.7:0/37418680 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f21dc0ef6e0 con 0x7f21e80c4020 2023-11-28T18:35:30.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.640+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21e804e890 con 0x7f21e80c4020 2023-11-28T18:35:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.640+0000 7f21e57fa640 1 -- 172.21.15.7:0/37418680 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5768 (secure 0 0 0) 0x7f21dc0b5090 con 0x7f21e80c4020 2023-11-28T18:35:30.643 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:30.643 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 58 2023-11-28T18:35:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21cc0668d0 msgr2=0x7f21cc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21cc0668d0 0x7f21cc068d90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f21d4006fd0 tx=0x7f21d4008040 comp rx=0 tx=0).stop 2023-11-28T18:35:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e80c4020 msgr2=0x7f21e81f0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e80c4020 0x7f21e81f0b40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f21dc01c920 tx=0x7f21dc01ba90 comp rx=0 tx=0).stop 2023-11-28T18:35:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 shutdown_connections 2023-11-28T18:35:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21cc0668d0 0x7f21cc068d90 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21e81575c0 0x7f21e81f33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8150b30 0x7f21e81f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 --2- 172.21.15.7:0/37418680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21e80c4020 0x7f21e81f0b40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 >> 172.21.15.7:0/37418680 conn(0x7f21e80662e0 msgr2=0x7f21e80c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 shutdown_connections 2023-11-28T18:35:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:30.644+0000 7f21ef05a640 1 -- 172.21.15.7:0/37418680 wait complete. 2023-11-28T18:35:30.665 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=4291 2023-11-28T18:35:30.665 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.e 2023-11-28T18:35:31.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:35:31.038 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:35:31.666 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-11-28T18:35:31.666 INFO:tasks.ceph.mds.e:Restarting daemon 2023-11-28T18:35:31.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-11-28T18:35:31.671 INFO:tasks.ceph.mds.e:Started 2023-11-28T18:35:31.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:31.768 INFO:tasks.ceph.mds.e.smithi007.stdout:starting mds.e at 2023-11-28T18:35:31.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.824+0000 7fabc9d2a640 1 Processor -- start 2023-11-28T18:35:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.824+0000 7fabc9d2a640 1 -- start start 2023-11-28T18:35:31.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc9d2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc41541a0 0x7fabc40c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:31.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc9d2a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc415e190 con 0x7fabc40c7db0 2023-11-28T18:35:31.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc9d2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc415d330 con 0x7fabc40c68e0 2023-11-28T18:35:31.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc9d2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40c6d70 con 0x7fabc41541a0 2023-11-28T18:35:31.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc2ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fabc40c7db0 0x7fabc40c7200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48264/0 (socket says 172.21.15.7:48264) 2023-11-28T18:35:31.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc2ffd640 1 -- 172.21.15.7:0/2064932856 learned_addr learned my addr 172.21.15.7:0/2064932856 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:35:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc37fe640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc41541a0 0x7fabc40c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329423672 0 0) 0x7fabc415e190 con 0x7fabc40c7db0 2023-11-28T18:35:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac003530 con 0x7fabc40c7db0 2023-11-28T18:35:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3747360755 0 0) 0x7fabac003530 con 0x7fabc40c7db0 2023-11-28T18:35:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc41541a0 msgr2=0x7fabc40c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc41541a0 0x7fabc40c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 >> v1:172.21.15.193:6789/0 conn(0x7fabc40c68e0 legacy=0x7fabc4159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc415d580 con 0x7fabc40c7db0 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fabb8001680 con 0x7fabc40c7db0 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 >> v1:172.21.15.7:6789/0 conn(0x7fabc40c7db0 legacy=0x7fabc40c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc41541a0 0x7fabac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabac003f30 0x7fabac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40c6d70 con 0x7fabac004d10 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac00bb00 con 0x7fabac003f30 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac00bcc0 con 0x7fabc41541a0 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc3fff640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc37fe640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc41541a0 0x7fabac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:31.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc2ffd640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabac003f30 0x7fabac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:31.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc3fff640 1 -- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc41541a0 msgr2=0x7fabac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:31.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc3fff640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc41541a0 0x7fabac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:31.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc3fff640 1 -- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabac003f30 msgr2=0x7fabac0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:31.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc3fff640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabac003f30 0x7fabac0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:31.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.828+0000 7fabc3fff640 1 -- 172.21.15.7:0/2064932856 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac003530 con 0x7fabac004d10 2023-11-28T18:35:31.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc3fff640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabac005190 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fabbc001e70 tx=0x7fabbc01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5645bfbe21490603 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc003b80 con 0x7fabac004d10 2023-11-28T18:35:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc003ce0 con 0x7fabac004d10 2023-11-28T18:35:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc27fc640 1 -- 172.21.15.7:0/2064932856 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc0052c0 con 0x7fabac004d10 2023-11-28T18:35:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc9d2a640 1 -- 172.21.15.7:0/2064932856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 msgr2=0x7fabac005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc9d2a640 1 --2- 172.21.15.7:0/2064932856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabac005190 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fabbc001e70 tx=0x7fabbc01c760 comp rx=0 tx=0).stop 2023-11-28T18:35:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc3fff640 1 -- 172.21.15.7:0/2064932856 reap_dead start 2023-11-28T18:35:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc9d2a640 1 -- 172.21.15.7:0/2064932856 shutdown_connections 2023-11-28T18:35:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc9d2a640 1 -- 172.21.15.7:0/2064932856 >> 172.21.15.7:0/2064932856 conn(0x7fabc40662e0 msgr2=0x7fabc40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc9d2a640 1 -- 172.21.15.7:0/2064932856 shutdown_connections 2023-11-28T18:35:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.832+0000 7fabc9d2a640 1 -- 172.21.15.7:0/2064932856 wait complete. 2023-11-28T18:35:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 Processor -- start 2023-11-28T18:35:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 -- start start 2023-11-28T18:35:31.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabc4145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:31.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc4144c50 0x7fabc40c3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:31.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc40c4260 0x7fabc41436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:31.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc41f3220 con 0x7fabac004d10 2023-11-28T18:35:31.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc41456c0 con 0x7fabc4144c50 2023-11-28T18:35:31.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc41f36b0 con 0x7fabc40c4260 2023-11-28T18:35:31.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabc4145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:31.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabc4145e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60100/0 (socket says 172.21.15.7:60100) 2023-11-28T18:35:31.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 -- 172.21.15.7:0/303293581 learned_addr learned my addr 172.21.15.7:0/303293581 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:31.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc2ffd640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc4144c50 0x7fabc40c3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:31.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc3fff640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc40c4260 0x7fabc41436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:31.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 -- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc40c4260 msgr2=0x7fabc41436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:31.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc40c4260 0x7fabc41436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:31.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 -- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc4144c50 msgr2=0x7fabc40c3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc4144c50 0x7fabc40c3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 -- 172.21.15.7:0/303293581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc4145420 con 0x7fabac004d10 2023-11-28T18:35:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc2ffd640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc4144c50 0x7fabc40c3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:35:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc37fe640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabc4145e40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fabb40119d0 tx=0x7fabb4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fababfff640 1 -- 172.21.15.7:0/303293581 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb400fb50 con 0x7fabac004d10 2023-11-28T18:35:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fababfff640 1 -- 172.21.15.7:0/303293581 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb400fcb0 con 0x7fabac004d10 2023-11-28T18:35:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fababfff640 1 -- 172.21.15.7:0/303293581 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4014440 con 0x7fabac004d10 2023-11-28T18:35:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc41f28f0 con 0x7fabac004d10 2023-11-28T18:35:31.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.836+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc41f2eb0 con 0x7fabac004d10 2023-11-28T18:35:31.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.840+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc41f3070 con 0x7fabac004d10 2023-11-28T18:35:31.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.840+0000 7fababfff640 1 -- 172.21.15.7:0/303293581 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fabb40145a0 con 0x7fabac004d10 2023-11-28T18:35:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.840+0000 7fababfff640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabb00668d0 0x7fabb0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.840+0000 7fababfff640 1 -- 172.21.15.7:0/303293581 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fabb4084fe0 con 0x7fabac004d10 2023-11-28T18:35:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.840+0000 7fabc2ffd640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabb00668d0 0x7fabb0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.840+0000 7fabc2ffd640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabb00668d0 0x7fabb0068d90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fabc4153ab0 tx=0x7fabb8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:31.844+0000 7fababfff640 1 -- 172.21.15.7:0/303293581 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fabb4052050 con 0x7fabac004d10 2023-11-28T18:35:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.032+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabc415de00 con 0x7fabac004d10 2023-11-28T18:35:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fababfff640 1 -- 172.21.15.7:0/303293581 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5768 (secure 0 0 0) 0x7fabc415de00 con 0x7fabac004d10 2023-11-28T18:35:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:32.036 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:32.037 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 58 2023-11-28T18:35:32.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabb00668d0 msgr2=0x7fabb0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:32.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabb00668d0 0x7fabb0068d90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fabc4153ab0 tx=0x7fabb8009040 comp rx=0 tx=0).stop 2023-11-28T18:35:32.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 msgr2=0x7fabc4145e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:32.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabc4145e40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fabb40119d0 tx=0x7fabb4011e90 comp rx=0 tx=0).stop 2023-11-28T18:35:32.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 shutdown_connections 2023-11-28T18:35:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabb00668d0 0x7fabb0068d90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabc40c4260 0x7fabc41436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc4144c50 0x7fabc40c3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 --2- 172.21.15.7:0/303293581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabac004d10 0x7fabc4145e40 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 >> 172.21.15.7:0/303293581 conn(0x7fabc40662e0 msgr2=0x7fabc40c2190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 shutdown_connections 2023-11-28T18:35:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:32.036+0000 7fabc9d2a640 1 -- 172.21.15.7:0/303293581 wait complete. 2023-11-28T18:35:32.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-11-28T18:35:34.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:35:34.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856f980640 1 Processor -- start 2023-11-28T18:35:34.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856f980640 1 -- start start 2023-11-28T18:35:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856f980640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85680cbb70 0x7f85680cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856f980640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85680ca920 con 0x7f85681542e0 2023-11-28T18:35:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856f980640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85680c4cd0 con 0x7f85680cc810 2023-11-28T18:35:34.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856f980640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8568071d50 con 0x7f85680cbb70 2023-11-28T18:35:34.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856cf1b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85680cbb70 0x7f85680cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:34.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856cf1b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85680cbb70 0x7f85680cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34754/0 (socket says 172.21.15.7:34754) 2023-11-28T18:35:34.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856cf1b640 1 -- 172.21.15.7:0/2473709766 learned_addr learned my addr 172.21.15.7:0/2473709766 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:34.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856cf1b640 1 -- 172.21.15.7:0/2473709766 >> v1:172.21.15.193:6789/0 conn(0x7f85680cc810 legacy=0x7f8568072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:35:34.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856cf1b640 1 -- 172.21.15.7:0/2473709766 >> v1:172.21.15.7:6789/0 conn(0x7f85681542e0 legacy=0x7f856815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:34.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.212+0000 7f856cf1b640 1 -- 172.21.15.7:0/2473709766 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85680cab70 con 0x7f85680cbb70 2023-11-28T18:35:34.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f856cf1b640 1 --2- 172.21.15.7:0/2473709766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85680cbb70 0x7f85680cc2d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8558009870 tx=0x7f855801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af770ccba3f8967f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:34.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f855ffff640 1 -- 172.21.15.7:0/2473709766 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85580ab610 con 0x7f85680cbb70 2023-11-28T18:35:34.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f855ffff640 1 -- 172.21.15.7:0/2473709766 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85580ab770 con 0x7f85680cbb70 2023-11-28T18:35:34.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f855ffff640 1 -- 172.21.15.7:0/2473709766 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85580abb00 con 0x7f85680cbb70 2023-11-28T18:35:34.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f856f980640 1 -- 172.21.15.7:0/2473709766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85680cbb70 msgr2=0x7f85680cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:34.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f856f980640 1 --2- 172.21.15.7:0/2473709766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85680cbb70 0x7f85680cc2d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8558009870 tx=0x7f855801c500 comp rx=0 tx=0).stop 2023-11-28T18:35:34.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f856f980640 1 -- 172.21.15.7:0/2473709766 shutdown_connections 2023-11-28T18:35:34.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f856f980640 1 --2- 172.21.15.7:0/2473709766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85680cbb70 0x7f85680cc2d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:34.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f856f980640 1 -- 172.21.15.7:0/2473709766 >> 172.21.15.7:0/2473709766 conn(0x7f85680717e0 msgr2=0x7f85680662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:34.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f856f980640 1 -- 172.21.15.7:0/2473709766 shutdown_connections 2023-11-28T18:35:34.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.216+0000 7f856f980640 1 -- 172.21.15.7:0/2473709766 wait complete. 2023-11-28T18:35:34.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 Processor -- start 2023-11-28T18:35:34.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 -- start start 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85680cbb70 0x7f856815a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85680cc810 0x7f8568152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85681542e0 0x7f8568155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8568072030 con 0x7f85681542e0 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85681576d0 con 0x7f85680cc810 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856815a440 con 0x7f85680cbb70 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85681542e0 0x7f8568155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85681542e0 0x7f8568155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34766/0 (socket says 172.21.15.7:34766) 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 -- 172.21.15.7:0/3052326794 learned_addr learned my addr 172.21.15.7:0/3052326794 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:35:34.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856d71c640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85680cbb70 0x7f856815a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:34.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856cf1b640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85680cc810 0x7f8568152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:34.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 -- 172.21.15.7:0/3052326794 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85680cbb70 msgr2=0x7f856815a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:34.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85680cbb70 0x7f856815a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:34.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 -- 172.21.15.7:0/3052326794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85680cc810 msgr2=0x7f8568152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:34.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85680cc810 0x7f8568152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:34.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 -- 172.21.15.7:0/3052326794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580094d0 con 0x7f85681542e0 2023-11-28T18:35:34.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856df1d640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85681542e0 0x7f8568155a10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f856000c7b0 tx=0x7f856000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:34.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f855dffb640 1 -- 172.21.15.7:0/3052326794 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8560004b70 con 0x7f85681542e0 2023-11-28T18:35:34.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f855dffb640 1 -- 172.21.15.7:0/3052326794 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8560004cd0 con 0x7f85681542e0 2023-11-28T18:35:34.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8568157f60 con 0x7f85681542e0 2023-11-28T18:35:34.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f855dffb640 1 -- 172.21.15.7:0/3052326794 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856000aa90 con 0x7f85681542e0 2023-11-28T18:35:34.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.220+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856815a0d0 con 0x7f85681542e0 2023-11-28T18:35:34.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.224+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856806c670 con 0x7f85681542e0 2023-11-28T18:35:34.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.224+0000 7f855dffb640 1 -- 172.21.15.7:0/3052326794 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f856000abf0 con 0x7f85681542e0 2023-11-28T18:35:34.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.224+0000 7f855dffb640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f854c066900 0x7f854c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:35:34.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.224+0000 7f855dffb640 1 -- 172.21.15.7:0/3052326794 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f85600848e0 con 0x7f85681542e0 2023-11-28T18:35:34.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.224+0000 7f856d71c640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f854c066900 0x7f854c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:35:34.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.224+0000 7f856d71c640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f854c066900 0x7f854c068dc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8564000f70 tx=0x7f8564009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:35:34.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.228+0000 7f855dffb640 1 -- 172.21.15.7:0/3052326794 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8560051800 con 0x7f85681542e0 2023-11-28T18:35:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.416+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f856815a5b0 con 0x7f85681542e0 2023-11-28T18:35:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f855dffb640 1 -- 172.21.15.7:0/3052326794 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f8560054e20 con 0x7f85681542e0 2023-11-28T18:35:34.420 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:35:34.420 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5700,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:35:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 59 2023-11-28T18:35:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f854c066900 msgr2=0x7f854c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f854c066900 0x7f854c068dc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8564000f70 tx=0x7f8564009040 comp rx=0 tx=0).stop 2023-11-28T18:35:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85681542e0 msgr2=0x7f8568155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85681542e0 0x7f8568155a10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f856000c7b0 tx=0x7f856000cc70 comp rx=0 tx=0).stop 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 shutdown_connections 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f854c066900 0x7f854c068dc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85681542e0 0x7f8568155a10 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85680cc810 0x7f8568152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 --2- 172.21.15.7:0/3052326794 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f85680cbb70 0x7f856815a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 >> 172.21.15.7:0/3052326794 conn(0x7f85680717e0 msgr2=0x7f85680d01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 shutdown_connections 2023-11-28T18:35:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:35:34.420+0000 7f856f980640 1 -- 172.21.15.7:0/3052326794 wait complete. 2023-11-28T18:35:34.441 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-11-28T18:35:34.442 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-11-28T18:35:37.324 INFO:tasks.ceph.mds.c.smithi007.stderr:2023-11-28T18:35:37.324+0000 7f6db0485640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T18:35:37.324 INFO:tasks.ceph.mds.c.smithi007.stderr:2023-11-28T18:35:37.324+0000 7f6db0485640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-11-28T18:35:37.324 INFO:tasks.ceph.mds.c.smithi007.stderr:2023-11-28T18:35:37.324+0000 7f6db0485640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-11-28T18:35:37.325 INFO:tasks.ceph.mds.c.smithi007.stderr:2023-11-28T18:35:37.324+0000 7f6db0485640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-11-28T18:35:37.325 INFO:tasks.ceph.mds.c.smithi007.stderr:2023-11-28T18:35:37.324+0000 7f6db0485640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-11-28T18:36:01.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:36:01.121 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:36:31.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:36:31.191 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:36:40.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:40.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.591+0000 7f635590f640 1 Processor -- start 2023-11-28T18:36:40.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.591+0000 7f635590f640 1 -- start start 2023-11-28T18:36:40.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f635590f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6350151cd0 0x7f635015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f635590f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f635006e560 0x7f6350157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f635590f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6350071e60 con 0x7f63500701f0 2023-11-28T18:36:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f635590f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6350070930 con 0x7f635006e560 2023-11-28T18:36:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f635590f640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6350070bf0 con 0x7f6350151cd0 2023-11-28T18:36:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634e7fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f635006e560 0x7f6350157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f63500701f0 0x7f63500c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51294/0 (socket says 172.21.15.7:51294) 2023-11-28T18:36:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634f7fe640 1 -- 172.21.15.7:0/4125368181 learned_addr learned my addr 172.21.15.7:0/4125368181 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:36:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634effd640 1 --2- 172.21.15.7:0/4125368181 >> v2:172.21.15.193:3300/0 conn(0x7f6350151cd0 0x7f635015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:40.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634e7fc640 1 -- 172.21.15.7:0/4125368181 >> v2:172.21.15.193:3300/0 conn(0x7f6350151cd0 msgr2=0x7f635015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:40.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634e7fc640 1 --2- 172.21.15.7:0/4125368181 >> v2:172.21.15.193:3300/0 conn(0x7f6350151cd0 0x7f635015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634e7fc640 1 -- 172.21.15.7:0/4125368181 >> v1:172.21.15.7:6789/0 conn(0x7f63500701f0 legacy=0x7f63500c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:40.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634e7fc640 1 -- 172.21.15.7:0/4125368181 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350070ed0 con 0x7f635006e560 2023-11-28T18:36:40.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.595+0000 7f634dffb640 1 -- 172.21.15.7:0/4125368181 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927355784 0 0) 0x7f6350071e60 con 0x7f63500701f0 2023-11-28T18:36:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f634e7fc640 1 --2- 172.21.15.7:0/4125368181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635006e560 0x7f6350157b90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f635006fee0 tx=0x7f634401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e81ada4ed703d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f634dffb640 1 -- 172.21.15.7:0/4125368181 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63440ab600 con 0x7f635006e560 2023-11-28T18:36:40.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f634dffb640 1 -- 172.21.15.7:0/4125368181 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63440a7050 con 0x7f635006e560 2023-11-28T18:36:40.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f634dffb640 1 -- 172.21.15.7:0/4125368181 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63440ab980 con 0x7f635006e560 2023-11-28T18:36:40.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 -- 172.21.15.7:0/4125368181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635006e560 msgr2=0x7f6350157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:40.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 --2- 172.21.15.7:0/4125368181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635006e560 0x7f6350157b90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f635006fee0 tx=0x7f634401c3d0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 -- 172.21.15.7:0/4125368181 shutdown_connections 2023-11-28T18:36:40.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 --2- 172.21.15.7:0/4125368181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635006e560 0x7f6350157b90 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 --2- 172.21.15.7:0/4125368181 >> v2:172.21.15.193:3300/0 conn(0x7f6350151cd0 0x7f635015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 -- 172.21.15.7:0/4125368181 >> 172.21.15.7:0/4125368181 conn(0x7f63500662e0 msgr2=0x7f63500666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:40.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 -- 172.21.15.7:0/4125368181 shutdown_connections 2023-11-28T18:36:40.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 -- 172.21.15.7:0/4125368181 wait complete. 2023-11-28T18:36:40.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 Processor -- start 2023-11-28T18:36:40.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.599+0000 7f635590f640 1 -- start start 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635006e560 0x7f6350148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63500701f0 0x7f63501f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6350151cd0 0x7f6350147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6350155960 con 0x7f63500701f0 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635014b9d0 con 0x7f635006e560 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6350155b90 con 0x7f6350151cd0 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63500701f0 0x7f63501f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63500701f0 0x7f63501f5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36930/0 (socket says 172.21.15.7:36930) 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 -- 172.21.15.7:0/3243514819 learned_addr learned my addr 172.21.15.7:0/3243514819 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634effd640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635006e560 0x7f6350148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634f7fe640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6350151cd0 0x7f6350147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 -- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6350151cd0 msgr2=0x7f6350147fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6350151cd0 0x7f6350147fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 -- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635006e560 msgr2=0x7f6350148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635006e560 0x7f6350148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 -- 172.21.15.7:0/3243514819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6344009420 con 0x7f63500701f0 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634f7fe640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6350151cd0 0x7f6350147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f634e7fc640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63500701f0 0x7f63501f5d50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f63440ae040 tx=0x7f6344019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f6337fff640 1 -- 172.21.15.7:0/3243514819 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63440b4430 con 0x7f63500701f0 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f6337fff640 1 -- 172.21.15.7:0/3243514819 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63440b45b0 con 0x7f63500701f0 2023-11-28T18:36:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6350155e50 con 0x7f63500701f0 2023-11-28T18:36:40.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f6337fff640 1 -- 172.21.15.7:0/3243514819 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63440b4960 con 0x7f63500701f0 2023-11-28T18:36:40.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635014b1d0 con 0x7f63500701f0 2023-11-28T18:36:40.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.603+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635004e890 con 0x7f63500701f0 2023-11-28T18:36:40.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.607+0000 7f6337fff640 1 -- 172.21.15.7:0/3243514819 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f63440b2040 con 0x7f63500701f0 2023-11-28T18:36:40.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.607+0000 7f6337fff640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f633c0668d0 0x7f633c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:40.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.607+0000 7f6337fff640 1 -- 172.21.15.7:0/3243514819 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f634411e5e0 con 0x7f63500701f0 2023-11-28T18:36:40.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.607+0000 7f634effd640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f633c0668d0 0x7f633c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:40.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.607+0000 7f634effd640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f633c0668d0 0x7f633c068d90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f63400079c0 tx=0x7f6340008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.611+0000 7f6337fff640 1 -- 172.21.15.7:0/3243514819 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f63440eb5a0 con 0x7f63500701f0 2023-11-28T18:36:40.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.819+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63501555b0 con 0x7f63500701f0 2023-11-28T18:36:40.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.819+0000 7f6337fff640 1 -- 172.21.15.7:0/3243514819 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f63440eebc0 con 0x7f63500701f0 2023-11-28T18:36:40.824 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:40.824 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5700,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:40.824 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 59 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f633c0668d0 msgr2=0x7f633c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f633c0668d0 0x7f633c068d90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f63400079c0 tx=0x7f6340008040 comp rx=0 tx=0).stop 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63500701f0 msgr2=0x7f63501f5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63500701f0 0x7f63501f5d50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f63440ae040 tx=0x7f6344019170 comp rx=0 tx=0).stop 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 shutdown_connections 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f633c0668d0 0x7f633c068d90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6350151cd0 0x7f6350147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63500701f0 0x7f63501f5d50 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 --2- 172.21.15.7:0/3243514819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f635006e560 0x7f6350148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:40.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 >> 172.21.15.7:0/3243514819 conn(0x7f63500662e0 msgr2=0x7f63500c9640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:40.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 shutdown_connections 2023-11-28T18:36:40.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:40.823+0000 7f635590f640 1 -- 172.21.15.7:0/3243514819 wait complete. 2023-11-28T18:36:40.848 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2023-11-28T18:36:40.848 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T18:36:40.848 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:36:40.864 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T18:36:40.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.011+0000 7ff1655b5640 1 Processor -- start 2023-11-28T18:36:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.011+0000 7ff1655b5640 1 -- start start 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.011+0000 7ff1655b5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff160151d80 0x7ff1600c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.011+0000 7ff1655b5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff16006ea30 0x7ff1600c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.011+0000 7ff1655b5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff160157fd0 0x7ff160158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.011+0000 7ff1655b5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1600c4ce0 con 0x7ff160157fd0 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff1655b5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff16015be90 con 0x7ff160151d80 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff1655b5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1601589a0 con 0x7ff16006ea30 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff160157fd0 0x7ff160158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff160157fd0 0x7ff160158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36942/0 (socket says 172.21.15.7:36942) 2023-11-28T18:36:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 -- 172.21.15.7:0/4098864035 learned_addr learned my addr 172.21.15.7:0/4098864035 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff15f7fe640 1 --2- 172.21.15.7:0/4098864035 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff16006ea30 0x7ff1600c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff15ffff640 1 --2- 172.21.15.7:0/4098864035 >> v2:172.21.15.193:3300/0 conn(0x7ff160151d80 0x7ff1600c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 -- 172.21.15.7:0/4098864035 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff16006ea30 msgr2=0x7ff1600c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 --2- 172.21.15.7:0/4098864035 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff16006ea30 0x7ff1600c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 -- 172.21.15.7:0/4098864035 >> v2:172.21.15.193:3300/0 conn(0x7ff160151d80 msgr2=0x7ff1600c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 --2- 172.21.15.7:0/4098864035 >> v2:172.21.15.193:3300/0 conn(0x7ff160151d80 0x7ff1600c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 -- 172.21.15.7:0/4098864035 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff160152160 con 0x7ff160157fd0 2023-11-28T18:36:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.015+0000 7ff164db4640 1 --2- 172.21.15.7:0/4098864035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160157fd0 0x7ff160158460 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff158009eb0 tx=0x7ff15801f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5879c4e3e56bcf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff15effd640 1 -- 172.21.15.7:0/4098864035 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580af400 con 0x7ff160157fd0 2023-11-28T18:36:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff15effd640 1 -- 172.21.15.7:0/4098864035 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1580ab050 con 0x7ff160157fd0 2023-11-28T18:36:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff15effd640 1 -- 172.21.15.7:0/4098864035 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580af780 con 0x7ff160157fd0 2023-11-28T18:36:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- 172.21.15.7:0/4098864035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160157fd0 msgr2=0x7ff160158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 --2- 172.21.15.7:0/4098864035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160157fd0 0x7ff160158460 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff158009eb0 tx=0x7ff15801f180 comp rx=0 tx=0).stop 2023-11-28T18:36:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- 172.21.15.7:0/4098864035 shutdown_connections 2023-11-28T18:36:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 --2- 172.21.15.7:0/4098864035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160157fd0 0x7ff160158460 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 --2- 172.21.15.7:0/4098864035 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff16006ea30 0x7ff1600c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 --2- 172.21.15.7:0/4098864035 >> v2:172.21.15.193:3300/0 conn(0x7ff160151d80 0x7ff1600c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- 172.21.15.7:0/4098864035 >> 172.21.15.7:0/4098864035 conn(0x7ff16006a5f0 msgr2=0x7ff16006aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- 172.21.15.7:0/4098864035 shutdown_connections 2023-11-28T18:36:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- 172.21.15.7:0/4098864035 wait complete. 2023-11-28T18:36:41.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 Processor -- start 2023-11-28T18:36:41.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- start start 2023-11-28T18:36:41.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff16006ea30 0x7ff1601595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:41.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160151d80 0x7ff1600721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:41.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff160157fd0 0x7ff160148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:41.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16014a6d0 con 0x7ff160151d80 2023-11-28T18:36:41.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff160159b00 con 0x7ff16006ea30 2023-11-28T18:36:41.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.019+0000 7ff1655b5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff160158b20 con 0x7ff160157fd0 2023-11-28T18:36:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160151d80 0x7ff1600721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160151d80 0x7ff1600721f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36958/0 (socket says 172.21.15.7:36958) 2023-11-28T18:36:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 -- 172.21.15.7:0/995440994 learned_addr learned my addr 172.21.15.7:0/995440994 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff164db4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff160157fd0 0x7ff160148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15ffff640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff16006ea30 0x7ff1601595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 -- 172.21.15.7:0/995440994 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff160157fd0 msgr2=0x7ff160148ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff160157fd0 0x7ff160148ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 -- 172.21.15.7:0/995440994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff16006ea30 msgr2=0x7ff1601595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff16006ea30 0x7ff1601595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 -- 172.21.15.7:0/995440994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff158009b30 con 0x7ff160151d80 2023-11-28T18:36:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15f7fe640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160151d80 0x7ff1600721f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff15400d800 tx=0x7ff15400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15cff9640 1 -- 172.21.15.7:0/995440994 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1540096a0 con 0x7ff160151d80 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15cff9640 1 -- 172.21.15.7:0/995440994 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff154009800 con 0x7ff160151d80 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15cff9640 1 -- 172.21.15.7:0/995440994 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff154009b70 con 0x7ff160151d80 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16014a990 con 0x7ff160151d80 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff16014af50 con 0x7ff160151d80 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1601518d0 con 0x7ff160151d80 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.023+0000 7ff15cff9640 1 -- 172.21.15.7:0/995440994 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff154009cd0 con 0x7ff160151d80 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.027+0000 7ff15cff9640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff14c0668d0 0x7ff14c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.027+0000 7ff15cff9640 1 -- 172.21.15.7:0/995440994 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7ff154084e10 con 0x7ff160151d80 2023-11-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.027+0000 7ff15ffff640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff14c0668d0 0x7ff14c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.027+0000 7ff15ffff640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff14c0668d0 0x7ff14c068d90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff150007a60 tx=0x7ff150008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.031+0000 7ff15cff9640 1 -- 172.21.15.7:0/995440994 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff154088030 con 0x7ff160151d80 2023-11-28T18:36:41.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.287+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff16004e890 con 0x7ff160151d80 2023-11-28T18:36:41.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.287+0000 7ff15cff9640 1 -- 172.21.15.7:0/995440994 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6555 (secure 0 0 0) 0x7ff154051d50 con 0x7ff160151d80 2023-11-28T18:36:41.291 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:41.291 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5700,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:41.292 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 59 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff14c0668d0 msgr2=0x7ff14c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff14c0668d0 0x7ff14c068d90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff150007a60 tx=0x7ff150008040 comp rx=0 tx=0).stop 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160151d80 msgr2=0x7ff1600721f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160151d80 0x7ff1600721f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff15400d800 tx=0x7ff15400dcc0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 shutdown_connections 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff14c0668d0 0x7ff14c068d90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff160157fd0 0x7ff160148ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff160151d80 0x7ff1600721f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 --2- 172.21.15.7:0/995440994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff16006ea30 0x7ff1601595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 >> 172.21.15.7:0/995440994 conn(0x7ff16006a5f0 msgr2=0x7ff160147db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 shutdown_connections 2023-11-28T18:36:41.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:41.291+0000 7ff1655b5640 1 -- 172.21.15.7:0/995440994 wait complete. 2023-11-28T18:36:41.315 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:36:43.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:43.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb42278d640 1 Processor -- start 2023-11-28T18:36:43.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb42278d640 1 -- start start 2023-11-28T18:36:43.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb42278d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb41c151cd0 0x7fb41c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:43.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb42278d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c157b90 0x7fb41c0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:43.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb42278d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c157fe0 con 0x7fb41c06e560 2023-11-28T18:36:43.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb42278d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb41c070730 con 0x7fb41c151cd0 2023-11-28T18:36:43.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb42278d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c071e60 con 0x7fb41c157b90 2023-11-28T18:36:43.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb41bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb41c151cd0 0x7fb41c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:43.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb41bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb41c151cd0 0x7fb41c15bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38768/0 (socket says 172.21.15.7:38768) 2023-11-28T18:36:43.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb41bfff640 1 -- 172.21.15.7:0/3900715055 learned_addr learned my addr 172.21.15.7:0/3900715055 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb420d2a640 1 --2- 172.21.15.7:0/3900715055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c157b90 0x7fb41c0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb41bfff640 1 -- 172.21.15.7:0/3900715055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c157b90 msgr2=0x7fb41c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb41bfff640 1 --2- 172.21.15.7:0/3900715055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c157b90 0x7fb41c0701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb41bfff640 1 -- 172.21.15.7:0/3900715055 >> v1:172.21.15.7:6789/0 conn(0x7fb41c06e560 legacy=0x7fb41c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb41bfff640 1 -- 172.21.15.7:0/3900715055 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c070930 con 0x7fb41c151cd0 2023-11-28T18:36:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.479+0000 7fb420d2a640 1 --2- 172.21.15.7:0/3900715055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c157b90 0x7fb41c0701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:36:43.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb41bfff640 1 --2- 172.21.15.7:0/3900715055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c151cd0 0x7fb41c15bd00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009870 tx=0x7fb40c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8a3bce7a90a941 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:43.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb41b7fe640 1 -- 172.21.15.7:0/3900715055 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0ac600 con 0x7fb41c151cd0 2023-11-28T18:36:43.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb41b7fe640 1 -- 172.21.15.7:0/3900715055 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40c0a8050 con 0x7fb41c151cd0 2023-11-28T18:36:43.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb41b7fe640 1 -- 172.21.15.7:0/3900715055 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0ac980 con 0x7fb41c151cd0 2023-11-28T18:36:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb42278d640 1 -- 172.21.15.7:0/3900715055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c151cd0 msgr2=0x7fb41c15bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb42278d640 1 --2- 172.21.15.7:0/3900715055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c151cd0 0x7fb41c15bd00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009870 tx=0x7fb40c01c4b0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb42278d640 1 -- 172.21.15.7:0/3900715055 shutdown_connections 2023-11-28T18:36:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb42278d640 1 --2- 172.21.15.7:0/3900715055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c157b90 0x7fb41c0701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb42278d640 1 --2- 172.21.15.7:0/3900715055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c151cd0 0x7fb41c15bd00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb42278d640 1 -- 172.21.15.7:0/3900715055 >> 172.21.15.7:0/3900715055 conn(0x7fb41c0662e0 msgr2=0x7fb41c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb42278d640 1 -- 172.21.15.7:0/3900715055 shutdown_connections 2023-11-28T18:36:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.483+0000 7fb42278d640 1 -- 172.21.15.7:0/3900715055 wait complete. 2023-11-28T18:36:43.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 Processor -- start 2023-11-28T18:36:43.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 -- start start 2023-11-28T18:36:43.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06e560 0x7fb41c0c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:43.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c151cd0 0x7fb41c1f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:43.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb41c157b90 0x7fb41c156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:43.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c0c8eb0 con 0x7fb41c06e560 2023-11-28T18:36:43.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c0c8660 con 0x7fb41c157b90 2023-11-28T18:36:43.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c1f5e80 con 0x7fb41c151cd0 2023-11-28T18:36:43.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06e560 0x7fb41c0c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:43.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06e560 0x7fb41c0c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38780/0 (socket says 172.21.15.7:38780) 2023-11-28T18:36:43.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 -- 172.21.15.7:0/1840453380 learned_addr learned my addr 172.21.15.7:0/1840453380 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:43.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb413fff640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c151cd0 0x7fb41c1f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:43.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb420d2a640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb41c157b90 0x7fb41c156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:43.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 -- 172.21.15.7:0/1840453380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c151cd0 msgr2=0x7fb41c1f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:43.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c151cd0 0x7fb41c1f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 -- 172.21.15.7:0/1840453380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb41c157b90 msgr2=0x7fb41c156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb41c157b90 0x7fb41c156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 -- 172.21.15.7:0/1840453380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c0094d0 con 0x7fb41c06e560 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb41bfff640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06e560 0x7fb41c0c8120 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb40c01ab60 tx=0x7fb40c01acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb4197fa640 1 -- 172.21.15.7:0/1840453380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0b7650 con 0x7fb41c06e560 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb4197fa640 1 -- 172.21.15.7:0/1840453380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40c0ace90 con 0x7fb41c06e560 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb4197fa640 1 -- 172.21.15.7:0/1840453380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0b79f0 con 0x7fb41c06e560 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41c1f2cb0 con 0x7fb41c06e560 2023-11-28T18:36:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.487+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c0c8bd0 con 0x7fb41c06e560 2023-11-28T18:36:43.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.491+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41c1f5ce0 con 0x7fb41c06e560 2023-11-28T18:36:43.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.491+0000 7fb4197fa640 1 -- 172.21.15.7:0/1840453380 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb40c0b3070 con 0x7fb41c06e560 2023-11-28T18:36:43.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.491+0000 7fb4197fa640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3fc0668d0 0x7fb3fc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:43.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.491+0000 7fb4197fa640 1 -- 172.21.15.7:0/1840453380 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fb40c11ee30 con 0x7fb41c06e560 2023-11-28T18:36:43.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.491+0000 7fb413fff640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3fc0668d0 0x7fb3fc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:43.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.495+0000 7fb413fff640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3fc0668d0 0x7fb3fc068d90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb41c1613e0 tx=0x7fb404009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:43.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.499+0000 7fb4197fa640 1 -- 172.21.15.7:0/1840453380 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb41c1f5ce0 con 0x7fb41c06e560 2023-11-28T18:36:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.723+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb41c1f5ce0 con 0x7fb41c06e560 2023-11-28T18:36:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb4197fa640 1 -- 172.21.15.7:0/1840453380 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fb41c1f5ce0 con 0x7fb41c06e560 2023-11-28T18:36:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:43.728 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5700,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 59 2023-11-28T18:36:43.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3fc0668d0 msgr2=0x7fb3fc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:43.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3fc0668d0 0x7fb3fc068d90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb41c1613e0 tx=0x7fb404009040 comp rx=0 tx=0).stop 2023-11-28T18:36:43.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06e560 msgr2=0x7fb41c0c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:43.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06e560 0x7fb41c0c8120 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb40c01ab60 tx=0x7fb40c01acf0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 shutdown_connections 2023-11-28T18:36:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3fc0668d0 0x7fb3fc068d90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb41c157b90 0x7fb41c156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb41c151cd0 0x7fb41c1f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 --2- 172.21.15.7:0/1840453380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06e560 0x7fb41c0c8120 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 >> 172.21.15.7:0/1840453380 conn(0x7fb41c0662e0 msgr2=0x7fb41c0666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 shutdown_connections 2023-11-28T18:36:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:43.727+0000 7fb42278d640 1 -- 172.21.15.7:0/1840453380 wait complete. 2023-11-28T18:36:43.751 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:36:45.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc62cf3e640 1 Processor -- start 2023-11-28T18:36:45.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc62cf3e640 1 -- start start 2023-11-28T18:36:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc62cf3e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc628151cd0 0x7fc62815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc62cf3e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc628157b90 0x7fc6280701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc62cf3e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc628071e60 con 0x7fc62806e560 2023-11-28T18:36:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc62cf3e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc628070730 con 0x7fc628157b90 2023-11-28T18:36:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc62cf3e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc628070930 con 0x7fc628151cd0 2023-11-28T18:36:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc626d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc628157b90 0x7fc6280701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc625d9b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc62806e560 0x7fc6280c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47928/0 (socket says 172.21.15.7:47928) 2023-11-28T18:36:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc625d9b640 1 -- 172.21.15.7:0/2354593636 learned_addr learned my addr 172.21.15.7:0/2354593636 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:36:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc62659c640 1 --2- 172.21.15.7:0/2354593636 >> v2:172.21.15.193:3300/0 conn(0x7fc628151cd0 0x7fc62815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc626d9d640 1 -- 172.21.15.7:0/2354593636 >> v2:172.21.15.193:3300/0 conn(0x7fc628151cd0 msgr2=0x7fc62815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc626d9d640 1 --2- 172.21.15.7:0/2354593636 >> v2:172.21.15.193:3300/0 conn(0x7fc628151cd0 0x7fc62815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc626d9d640 1 -- 172.21.15.7:0/2354593636 >> v1:172.21.15.7:6789/0 conn(0x7fc62806e560 legacy=0x7fc6280c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.915+0000 7fc626d9d640 1 -- 172.21.15.7:0/2354593636 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc628157fa0 con 0x7fc628157b90 2023-11-28T18:36:45.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.919+0000 7fc626d9d640 1 --2- 172.21.15.7:0/2354593636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc6280701f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc62806fee0 tx=0x7fc62001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d79aceee8e19b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:45.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.919+0000 7fc62559a640 1 -- 172.21.15.7:0/2354593636 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6200a5830 con 0x7fc628157b90 2023-11-28T18:36:45.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.919+0000 7fc62559a640 1 -- 172.21.15.7:0/2354593636 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6200a5990 con 0x7fc628157b90 2023-11-28T18:36:45.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.919+0000 7fc62559a640 1 -- 172.21.15.7:0/2354593636 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6200a5d20 con 0x7fc628157b90 2023-11-28T18:36:45.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 -- 172.21.15.7:0/2354593636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 msgr2=0x7fc6280701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:45.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/2354593636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc6280701f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc62806fee0 tx=0x7fc62001c880 comp rx=0 tx=0).stop 2023-11-28T18:36:45.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 -- 172.21.15.7:0/2354593636 shutdown_connections 2023-11-28T18:36:45.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/2354593636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc6280701f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:45.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/2354593636 >> v2:172.21.15.193:3300/0 conn(0x7fc628151cd0 0x7fc62815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:45.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 -- 172.21.15.7:0/2354593636 >> 172.21.15.7:0/2354593636 conn(0x7fc6280662e0 msgr2=0x7fc6280666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:45.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 -- 172.21.15.7:0/2354593636 shutdown_connections 2023-11-28T18:36:45.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 -- 172.21.15.7:0/2354593636 wait complete. 2023-11-28T18:36:45.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 Processor -- start 2023-11-28T18:36:45.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.923+0000 7fc62cf3e640 1 -- start start 2023-11-28T18:36:45.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc62806e560 0x7fc6280cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:45.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc628151cd0 0x7fc628155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:45.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc628148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:45.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280c7b80 con 0x7fc628157b90 2023-11-28T18:36:45.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6281f3010 con 0x7fc62806e560 2023-11-28T18:36:45.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6281f2460 con 0x7fc628151cd0 2023-11-28T18:36:45.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc628148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:45.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc628148e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38788/0 (socket says 172.21.15.7:38788) 2023-11-28T18:36:45.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 -- 172.21.15.7:0/4188933347 learned_addr learned my addr 172.21.15.7:0/4188933347 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:45.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc625d9b640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc628151cd0 0x7fc628155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:45.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62659c640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc62806e560 0x7fc6280cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 -- 172.21.15.7:0/4188933347 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc628151cd0 msgr2=0x7fc628155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc628151cd0 0x7fc628155c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 -- 172.21.15.7:0/4188933347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc62806e560 msgr2=0x7fc6280cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc62806e560 0x7fc6280cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 -- 172.21.15.7:0/4188933347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc620009a70 con 0x7fc628157b90 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc626d9d640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc628148e50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc620019490 tx=0x7fc620019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc616ffd640 1 -- 172.21.15.7:0/4188933347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6200b9650 con 0x7fc628157b90 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc616ffd640 1 -- 172.21.15.7:0/4188933347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6200b97b0 con 0x7fc628157b90 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc616ffd640 1 -- 172.21.15.7:0/4188933347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6200b9b60 con 0x7fc628157b90 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6281f3430 con 0x7fc628157b90 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6281f2670 con 0x7fc628157b90 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.927+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6281f3150 con 0x7fc628157b90 2023-11-28T18:36:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.931+0000 7fc616ffd640 1 -- 172.21.15.7:0/4188933347 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc6200b9cc0 con 0x7fc628157b90 2023-11-28T18:36:45.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.931+0000 7fc616ffd640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6080668d0 0x7fc608068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:45.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.931+0000 7fc616ffd640 1 -- 172.21.15.7:0/4188933347 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fc62011f7e0 con 0x7fc628157b90 2023-11-28T18:36:45.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.931+0000 7fc62659c640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6080668d0 0x7fc608068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:45.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.931+0000 7fc62659c640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6080668d0 0x7fc608068d90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc618009b60 tx=0x7fc618009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:45.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:45.935+0000 7fc616ffd640 1 -- 172.21.15.7:0/4188933347 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc6281f3150 con 0x7fc628157b90 2023-11-28T18:36:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.139+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6281f3150 con 0x7fc628157b90 2023-11-28T18:36:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.139+0000 7fc616ffd640 1 -- 172.21.15.7:0/4188933347 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fc6281f3150 con 0x7fc628157b90 2023-11-28T18:36:46.141 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:46.142 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5700,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 59 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.139+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6080668d0 msgr2=0x7fc608068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.139+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6080668d0 0x7fc608068d90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc618009b60 tx=0x7fc618009190 comp rx=0 tx=0).stop 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.139+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 msgr2=0x7fc628148e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.139+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc628148e50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc620019490 tx=0x7fc620019620 comp rx=0 tx=0).stop 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.143+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 shutdown_connections 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.143+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6080668d0 0x7fc608068d90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.143+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc628157b90 0x7fc628148e50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.143+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc628151cd0 0x7fc628155c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.143+0000 7fc62cf3e640 1 --2- 172.21.15.7:0/4188933347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc62806e560 0x7fc6280cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.143+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 >> 172.21.15.7:0/4188933347 conn(0x7fc6280662e0 msgr2=0x7fc6281570d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.143+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 shutdown_connections 2023-11-28T18:36:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:46.143+0000 7fc62cf3e640 1 -- 172.21.15.7:0/4188933347 wait complete. 2023-11-28T18:36:46.166 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:36:48.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.331+0000 7f7fca03a640 1 Processor -- start 2023-11-28T18:36:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fca03a640 1 -- start start 2023-11-28T18:36:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fca03a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7fc415ce10 0x7f7fc40ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fca03a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc415a3a0 con 0x7f7fc415ca30 2023-11-28T18:36:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fca03a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc415a620 con 0x7f7fc40cafe0 2023-11-28T18:36:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fca03a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc415a900 con 0x7f7fc415ce10 2023-11-28T18:36:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fc9038640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7fc415ca30 0x7f7fc4159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47938/0 (socket says 172.21.15.7:47938) 2023-11-28T18:36:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fc9038640 1 -- 172.21.15.7:0/573909671 learned_addr learned my addr 172.21.15.7:0/573909671 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:36:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fc8837640 1 --2- 172.21.15.7:0/573909671 >> v2:172.21.15.193:3300/0 conn(0x7f7fc415ce10 0x7f7fc40ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fbbfff640 1 -- 172.21.15.7:0/573909671 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285116487 0 0) 0x7f7fc415a3a0 con 0x7f7fc415ca30 2023-11-28T18:36:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fbbfff640 1 -- 172.21.15.7:0/573909671 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0003530 con 0x7f7fc415ca30 2023-11-28T18:36:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fbbfff640 1 -- 172.21.15.7:0/573909671 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151382162 0 0) 0x7f7fc415a620 con 0x7f7fc40cafe0 2023-11-28T18:36:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fbbfff640 1 -- 172.21.15.7:0/573909671 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc415a3a0 con 0x7f7fc40cafe0 2023-11-28T18:36:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fc8837640 1 -- 172.21.15.7:0/573909671 >> v1:172.21.15.193:6789/0 conn(0x7f7fc40cafe0 legacy=0x7f7fc40cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fc8837640 1 -- 172.21.15.7:0/573909671 >> v1:172.21.15.7:6789/0 conn(0x7f7fc415ca30 legacy=0x7f7fc4159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fbbfff640 1 -- 172.21.15.7:0/573909671 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236593965 0 0) 0x7f7fb0003530 con 0x7f7fc415ca30 2023-11-28T18:36:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.335+0000 7f7fc8837640 1 -- 172.21.15.7:0/573909671 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc415abe0 con 0x7f7fc415ce10 2023-11-28T18:36:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fc8837640 1 --2- 172.21.15.7:0/573909671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 0x7f7fc40ca1d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40097c0 tx=0x7f7fb401c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbefc2a683ed4c30 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fbbfff640 1 -- 172.21.15.7:0/573909671 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0001680 con 0x7f7fc415ce10 2023-11-28T18:36:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fbbfff640 1 -- 172.21.15.7:0/573909671 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb40a7050 con 0x7f7fc415ce10 2023-11-28T18:36:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fbbfff640 1 -- 172.21.15.7:0/573909671 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb40ab600 con 0x7f7fc415ce10 2023-11-28T18:36:48.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fca03a640 1 -- 172.21.15.7:0/573909671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 msgr2=0x7f7fc40ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:48.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fca03a640 1 --2- 172.21.15.7:0/573909671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 0x7f7fc40ca1d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40097c0 tx=0x7f7fb401c370 comp rx=0 tx=0).stop 2023-11-28T18:36:48.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fca03a640 1 -- 172.21.15.7:0/573909671 shutdown_connections 2023-11-28T18:36:48.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fca03a640 1 --2- 172.21.15.7:0/573909671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 0x7f7fc40ca1d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:48.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fca03a640 1 -- 172.21.15.7:0/573909671 >> 172.21.15.7:0/573909671 conn(0x7f7fc406c060 msgr2=0x7f7fc40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:48.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fca03a640 1 -- 172.21.15.7:0/573909671 shutdown_connections 2023-11-28T18:36:48.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.339+0000 7f7fca03a640 1 -- 172.21.15.7:0/573909671 wait complete. 2023-11-28T18:36:48.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 Processor -- start 2023-11-28T18:36:48.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 -- start start 2023-11-28T18:36:48.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fc40cafe0 0x7f7fc40c5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:48.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fc415ca30 0x7f7fc41570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 0x7f7fc41f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40c5da0 con 0x7f7fc40cafe0 2023-11-28T18:36:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc4155010 con 0x7f7fc415ce10 2023-11-28T18:36:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40c2f60 con 0x7f7fc415ca30 2023-11-28T18:36:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fc40cafe0 0x7f7fc40c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fc40cafe0 0x7f7fc40c5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38804/0 (socket says 172.21.15.7:38804) 2023-11-28T18:36:48.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 -- 172.21.15.7:0/4249341684 learned_addr learned my addr 172.21.15.7:0/4249341684 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:48.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9839640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 0x7f7fc41f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:48.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc8837640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fc415ca30 0x7f7fc41570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:48.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 -- 172.21.15.7:0/4249341684 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fc415ca30 msgr2=0x7f7fc41570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:48.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fc415ca30 0x7f7fc41570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:48.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 -- 172.21.15.7:0/4249341684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 msgr2=0x7f7fc41f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:48.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 0x7f7fc41f61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:48.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 -- 172.21.15.7:0/4249341684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb4009420 con 0x7f7fc40cafe0 2023-11-28T18:36:48.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fc9038640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fc40cafe0 0x7f7fc40c5860 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7fc000c780 tx=0x7f7fc000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:48.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fb9ffb640 1 -- 172.21.15.7:0/4249341684 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0005ae0 con 0x7f7fc40cafe0 2023-11-28T18:36:48.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fb9ffb640 1 -- 172.21.15.7:0/4249341684 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc0005c40 con 0x7f7fc40cafe0 2023-11-28T18:36:48.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fb9ffb640 1 -- 172.21.15.7:0/4249341684 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0007070 con 0x7f7fc40cafe0 2023-11-28T18:36:48.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.343+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc41569a0 con 0x7f7fc40cafe0 2023-11-28T18:36:48.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.347+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc40c4780 con 0x7f7fc40cafe0 2023-11-28T18:36:48.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.347+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc404e890 con 0x7f7fc40cafe0 2023-11-28T18:36:48.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.347+0000 7f7fb9ffb640 1 -- 172.21.15.7:0/4249341684 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7fc00071d0 con 0x7f7fc40cafe0 2023-11-28T18:36:48.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.347+0000 7f7fb9ffb640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fa80668d0 0x7f7fa8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:48.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.347+0000 7f7fb9ffb640 1 -- 172.21.15.7:0/4249341684 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f7fc0083d50 con 0x7f7fc40cafe0 2023-11-28T18:36:48.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.347+0000 7f7fc8837640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fa80668d0 0x7f7fa8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:48.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.351+0000 7f7fc8837640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fa80668d0 0x7f7fa8068d90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40ad000 tx=0x7f7fb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:48.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.355+0000 7f7fb9ffb640 1 -- 172.21.15.7:0/4249341684 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7fc0087030 con 0x7f7fc40cafe0 2023-11-28T18:36:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fc41588d0 con 0x7f7fc40cafe0 2023-11-28T18:36:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fb9ffb640 1 -- 172.21.15.7:0/4249341684 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f7fc0017020 con 0x7f7fc40cafe0 2023-11-28T18:36:48.601 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:48.601 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5700,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 59 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fa80668d0 msgr2=0x7f7fa8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fa80668d0 0x7f7fa8068d90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40ad000 tx=0x7f7fb4008040 comp rx=0 tx=0).stop 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fc40cafe0 msgr2=0x7f7fc40c5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fc40cafe0 0x7f7fc40c5860 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7fc000c780 tx=0x7f7fc000cc40 comp rx=0 tx=0).stop 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 shutdown_connections 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fa80668d0 0x7f7fa8068d90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fc415ce10 0x7f7fc41f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fc415ca30 0x7f7fc41570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 --2- 172.21.15.7:0/4249341684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fc40cafe0 0x7f7fc40c5860 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 >> 172.21.15.7:0/4249341684 conn(0x7f7fc406c060 msgr2=0x7f7fc40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:48.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 shutdown_connections 2023-11-28T18:36:48.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:48.599+0000 7f7fca03a640 1 -- 172.21.15.7:0/4249341684 wait complete. 2023-11-28T18:36:48.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:36:50.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905dff8640 1 Processor -- start 2023-11-28T18:36:50.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905dff8640 1 -- start start 2023-11-28T18:36:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905dff8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9058159c90 0x7f905815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905dff8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f905815def0 0x7f90580ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905dff8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905815d240 con 0x7f9058153ee0 2023-11-28T18:36:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905dff8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905815d520 con 0x7f905815def0 2023-11-28T18:36:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905dff8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90580cbd10 con 0x7f9058159c90 2023-11-28T18:36:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905cff6640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9058153ee0 0x7f90580cb600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47950/0 (socket says 172.21.15.7:47950) 2023-11-28T18:36:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905cff6640 1 -- 172.21.15.7:0/280930222 learned_addr learned my addr 172.21.15.7:0/280930222 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:36:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f9057fff640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9058159c90 0x7f905815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905d7f7640 1 --2- 172.21.15.7:0/280930222 >> v2:172.21.15.193:3300/0 conn(0x7f905815def0 0x7f90580ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722370567 0 0) 0x7f905815d240 con 0x7f9058153ee0 2023-11-28T18:36:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9048003530 con 0x7f9058153ee0 2023-11-28T18:36:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681838738 0 0) 0x7f9048003530 con 0x7f9058153ee0 2023-11-28T18:36:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9058159c90 msgr2=0x7f905815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9058159c90 0x7f905815d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 >> v2:172.21.15.193:3300/0 conn(0x7f905815def0 msgr2=0x7f90580ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 --2- 172.21.15.7:0/280930222 >> v2:172.21.15.193:3300/0 conn(0x7f905815def0 0x7f90580ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580c4bc0 con 0x7f9058153ee0 2023-11-28T18:36:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f904c001680 con 0x7f9058153ee0 2023-11-28T18:36:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 >> v1:172.21.15.7:6789/0 conn(0x7f9058153ee0 legacy=0x7f90580cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9048003980 0x7f9048003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90480047d0 0x7f9048004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9048005190 0x7f904800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905815d520 con 0x7f90480047d0 2023-11-28T18:36:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90580cbd10 con 0x7f9048005190 2023-11-28T18:36:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904800bed0 con 0x7f9048003980 2023-11-28T18:36:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f905d7f7640 1 --2- 172.21.15.7:0/280930222 >> v2:172.21.15.193:3300/0 conn(0x7f905815def0 0x7f90580ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:36:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.779+0000 7f9057fff640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9058159c90 0x7f905815d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:36:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f905cff6640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9048003980 0x7f9048003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f9057fff640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90480047d0 0x7f9048004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f905d7f7640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9048005190 0x7f904800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f905cff6640 1 -- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9048005190 msgr2=0x7f904800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:50.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f905cff6640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9048005190 0x7f904800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:50.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f905cff6640 1 -- 172.21.15.7:0/280930222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90480047d0 msgr2=0x7f9048004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:50.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f905cff6640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90480047d0 0x7f9048004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:50.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f905cff6640 1 -- 172.21.15.7:0/280930222 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048003530 con 0x7f9048003980 2023-11-28T18:36:50.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f9057fff640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90480047d0 0x7f9048004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f905cff6640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9048003980 0x7f9048003da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f904c00c110 tx=0x7f904c01cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6073f6d2b8d4bb87 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c0058e0 con 0x7f9048003980 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c005a60 con 0x7f9048003980 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.783+0000 7f90577fe640 1 -- 172.21.15.7:0/280930222 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c005e50 con 0x7f9048003980 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9048003980 msgr2=0x7f9048003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 --2- 172.21.15.7:0/280930222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9048003980 0x7f9048003da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f904c00c110 tx=0x7f904c01cba0 comp rx=0 tx=0).stop 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905d7f7640 1 -- 172.21.15.7:0/280930222 reap_dead start 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- 172.21.15.7:0/280930222 shutdown_connections 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- 172.21.15.7:0/280930222 >> 172.21.15.7:0/280930222 conn(0x7f9058071e20 msgr2=0x7f9058072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- 172.21.15.7:0/280930222 shutdown_connections 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- 172.21.15.7:0/280930222 wait complete. 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 Processor -- start 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- start start 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9048003980 0x7f90581f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90581f2f30 0x7f905815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905815b350 0x7f9058145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905815b7d0 con 0x7f9048003980 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90581f3590 con 0x7f90581f2f30 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905dff8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90581f3330 con 0x7f905815b350 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905cff6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9048003980 0x7f90581f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905cff6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9048003980 0x7f90581f29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38836/0 (socket says 172.21.15.7:38836) 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905cff6640 1 -- 172.21.15.7:0/2856491919 learned_addr learned my addr 172.21.15.7:0/2856491919 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f905d7f7640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905815b350 0x7f9058145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.787+0000 7f9057fff640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90581f2f30 0x7f905815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905cff6640 1 -- 172.21.15.7:0/2856491919 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905815b350 msgr2=0x7f9058145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905cff6640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905815b350 0x7f9058145da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905cff6640 1 -- 172.21.15.7:0/2856491919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90581f2f30 msgr2=0x7f905815ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905cff6640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90581f2f30 0x7f905815ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905cff6640 1 -- 172.21.15.7:0/2856491919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580b4790 con 0x7f9048003980 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905cff6640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9048003980 0x7f90581f29f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f904c0ab040 tx=0x7f9048005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f90557fa640 1 -- 172.21.15.7:0/2856491919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c0a5d70 con 0x7f9048003980 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f90557fa640 1 -- 172.21.15.7:0/2856491919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c0047f0 con 0x7f9048003980 2023-11-28T18:36:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90580b4a70 con 0x7f9048003980 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f90557fa640 1 -- 172.21.15.7:0/2856491919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c004ba0 con 0x7f9048003980 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90580b5030 con 0x7f9048003980 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905804e890 con 0x7f9048003980 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.791+0000 7f90557fa640 1 -- 172.21.15.7:0/2856491919 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f904c004d00 con 0x7f9048003980 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.795+0000 7f90557fa640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f903c0668d0 0x7f903c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.795+0000 7f90557fa640 1 -- 172.21.15.7:0/2856491919 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f904c11e470 con 0x7f9048003980 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.795+0000 7f9057fff640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f903c0668d0 0x7f903c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.795+0000 7f9057fff640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f903c0668d0 0x7f903c068d90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f905815ca90 tx=0x7f9044009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:50.799+0000 7f90557fa640 1 -- 172.21.15.7:0/2856491919 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f904c0eb4d0 con 0x7f9048003980 2023-11-28T18:36:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.027+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9058149300 con 0x7f9048003980 2023-11-28T18:36:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.027+0000 7f90557fa640 1 -- 172.21.15.7:0/2856491919 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f904c0eeaf0 con 0x7f9048003980 2023-11-28T18:36:51.031 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:51.031 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5700,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:35:29.465560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"b","rank":4,"incarnation":28,"state":"up:active","state_seq":40,"addr":"172.21.15.193:6836/1685294311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1685294311},{"type":"v1","addr":"172.21.15.193:6836","nonce":1685294311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:51.032 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 59 2023-11-28T18:36:51.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f903c0668d0 msgr2=0x7f903c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:51.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f903c0668d0 0x7f903c068d90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f905815ca90 tx=0x7f9044009190 comp rx=0 tx=0).stop 2023-11-28T18:36:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9048003980 msgr2=0x7f90581f29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9048003980 0x7f90581f29f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f904c0ab040 tx=0x7f9048005800 comp rx=0 tx=0).stop 2023-11-28T18:36:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 shutdown_connections 2023-11-28T18:36:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f903c0668d0 0x7f903c068d90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905815b350 0x7f9058145da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90581f2f30 0x7f905815ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:51.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 --2- 172.21.15.7:0/2856491919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9048003980 0x7f90581f29f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:51.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 >> 172.21.15.7:0/2856491919 conn(0x7f9058071e20 msgr2=0x7f90580b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:51.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 shutdown_connections 2023-11-28T18:36:51.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:51.031+0000 7f905dff8640 1 -- 172.21.15.7:0/2856491919 wait complete. 2023-11-28T18:36:51.055 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-11-28T18:36:53.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.203+0000 7f03f149c640 1 Processor -- start 2023-11-28T18:36:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.203+0000 7f03f149c640 1 -- start start 2023-11-28T18:36:53.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03f149c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03ec15ca30 0x7f03ec159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03f149c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f03ec0cafe0 0x7f03ec0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03f149c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec15a620 con 0x7f03ec15ce10 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03f149c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec15a900 con 0x7f03ec15ca30 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03f149c640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec15abe0 con 0x7f03ec0cafe0 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03ebfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03ec15ca30 0x7f03ec159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03ebfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03ec15ca30 0x7f03ec159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38844/0 (socket says 172.21.15.7:38844) 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03ebfff640 1 -- 172.21.15.7:0/3310634770 learned_addr learned my addr 172.21.15.7:0/3310634770 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03f0c9b640 1 --2- 172.21.15.7:0/3310634770 >> v2:172.21.15.193:3300/0 conn(0x7f03ec0cafe0 0x7f03ec0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03ebfff640 1 -- 172.21.15.7:0/3310634770 >> v2:172.21.15.193:3300/0 conn(0x7f03ec0cafe0 msgr2=0x7f03ec0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03ebfff640 1 --2- 172.21.15.7:0/3310634770 >> v2:172.21.15.193:3300/0 conn(0x7f03ec0cafe0 0x7f03ec0cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03ebfff640 1 -- 172.21.15.7:0/3310634770 >> v1:172.21.15.193:6789/0 conn(0x7f03ec15ce10 legacy=0x7f03ec0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:53.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03ebfff640 1 -- 172.21.15.7:0/3310634770 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec0cbb10 con 0x7f03ec15ca30 2023-11-28T18:36:53.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.207+0000 7f03f0c9b640 1 --2- 172.21.15.7:0/3310634770 >> v2:172.21.15.193:3300/0 conn(0x7f03ec0cafe0 0x7f03ec0cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:36:53.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03ebfff640 1 --2- 172.21.15.7:0/3310634770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec15ca30 0x7f03ec159c90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0671d0 tx=0x7f03dc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0026a5e1d48885 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:53.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03eaffd640 1 -- 172.21.15.7:0/3310634770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0ac640 con 0x7f03ec15ca30 2023-11-28T18:36:53.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03eaffd640 1 -- 172.21.15.7:0/3310634770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc0ac7a0 con 0x7f03ec15ca30 2023-11-28T18:36:53.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03eaffd640 1 -- 172.21.15.7:0/3310634770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0acb30 con 0x7f03ec15ca30 2023-11-28T18:36:53.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 -- 172.21.15.7:0/3310634770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec15ca30 msgr2=0x7f03ec159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 --2- 172.21.15.7:0/3310634770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec15ca30 0x7f03ec159c90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0671d0 tx=0x7f03dc01c4d0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 -- 172.21.15.7:0/3310634770 shutdown_connections 2023-11-28T18:36:53.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 --2- 172.21.15.7:0/3310634770 >> v2:172.21.15.193:3300/0 conn(0x7f03ec0cafe0 0x7f03ec0cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 --2- 172.21.15.7:0/3310634770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec15ca30 0x7f03ec159c90 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 -- 172.21.15.7:0/3310634770 >> 172.21.15.7:0/3310634770 conn(0x7f03ec06c060 msgr2=0x7f03ec0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:53.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 -- 172.21.15.7:0/3310634770 shutdown_connections 2023-11-28T18:36:53.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 -- 172.21.15.7:0/3310634770 wait complete. 2023-11-28T18:36:53.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.211+0000 7f03f149c640 1 Processor -- start 2023-11-28T18:36:53.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 -- start start 2023-11-28T18:36:53.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec0cafe0 0x7f03ec0c2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03ec15ca30 0x7f03ec1432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ec15ce10 0x7f03ec1437e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec140f30 con 0x7f03ec0cafe0 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec1500a0 con 0x7f03ec15ce10 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec142ae0 con 0x7f03ec15ca30 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec0cafe0 0x7f03ec0c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec0cafe0 0x7f03ec0c2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38850/0 (socket says 172.21.15.7:38850) 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 -- 172.21.15.7:0/1169775325 learned_addr learned my addr 172.21.15.7:0/1169775325 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f0c9b640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ec15ce10 0x7f03ec1437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03eb7fe640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03ec15ca30 0x7f03ec1432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 -- 172.21.15.7:0/1169775325 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03ec15ca30 msgr2=0x7f03ec1432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03ec15ca30 0x7f03ec1432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 -- 172.21.15.7:0/1169775325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ec15ce10 msgr2=0x7f03ec1437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ec15ce10 0x7f03ec1437e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 -- 172.21.15.7:0/1169775325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0094d0 con 0x7f03ec0cafe0 2023-11-28T18:36:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03ebfff640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec0cafe0 0x7f03ec0c2260 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f03dc01a100 tx=0x7f03dc01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03e8ff9640 1 -- 172.21.15.7:0/1169775325 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0bc070 con 0x7f03ec0cafe0 2023-11-28T18:36:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03e8ff9640 1 -- 172.21.15.7:0/1169775325 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc0ac600 con 0x7f03ec0cafe0 2023-11-28T18:36:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03e8ff9640 1 -- 172.21.15.7:0/1169775325 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0b78c0 con 0x7f03ec0cafe0 2023-11-28T18:36:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ec158990 con 0x7f03ec0cafe0 2023-11-28T18:36:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.215+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ec141180 con 0x7f03ec0cafe0 2023-11-28T18:36:53.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.219+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ec15a1b0 con 0x7f03ec0cafe0 2023-11-28T18:36:53.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.219+0000 7f03e8ff9640 1 -- 172.21.15.7:0/1169775325 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f03dc0b3070 con 0x7f03ec0cafe0 2023-11-28T18:36:53.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.219+0000 7f03e8ff9640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03d80668d0 0x7f03d8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.219+0000 7f03e8ff9640 1 -- 172.21.15.7:0/1169775325 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f03dc11e570 con 0x7f03ec0cafe0 2023-11-28T18:36:53.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.219+0000 7f03eb7fe640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03d80668d0 0x7f03d8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.219+0000 7f03eb7fe640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03d80668d0 0x7f03d8068d90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f03ec1472e0 tx=0x7f03e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:53.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.227+0000 7f03e8ff9640 1 -- 172.21.15.7:0/1169775325 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03ec15a1b0 con 0x7f03ec0cafe0 2023-11-28T18:36:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.439+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03ec15a190 con 0x7f03ec0cafe0 2023-11-28T18:36:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.443+0000 7f03e8ff9640 1 -- 172.21.15.7:0/1169775325 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5761 (secure 0 0 0) 0x7f03dc0ef0b0 con 0x7f03ec0cafe0 2023-11-28T18:36:53.446 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:53.446 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:36:52.304088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 60 2023-11-28T18:36:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03d80668d0 msgr2=0x7f03d8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03d80668d0 0x7f03d8068d90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f03ec1472e0 tx=0x7f03e0009040 comp rx=0 tx=0).stop 2023-11-28T18:36:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec0cafe0 msgr2=0x7f03ec0c2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec0cafe0 0x7f03ec0c2260 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f03dc01a100 tx=0x7f03dc01ad10 comp rx=0 tx=0).stop 2023-11-28T18:36:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 shutdown_connections 2023-11-28T18:36:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f03d80668d0 0x7f03d8068d90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ec15ce10 0x7f03ec1437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f03ec15ca30 0x7f03ec1432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 --2- 172.21.15.7:0/1169775325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03ec0cafe0 0x7f03ec0c2260 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 >> 172.21.15.7:0/1169775325 conn(0x7f03ec06c060 msgr2=0x7f03ec14d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:53.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 shutdown_connections 2023-11-28T18:36:53.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.447+0000 7f03f149c640 1 -- 172.21.15.7:0/1169775325 wait complete. 2023-11-28T18:36:53.468 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-11-28T18:36:53.469 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:36:53.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:53.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edee3a640 1 Processor -- start 2023-11-28T18:36:53.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edee3a640 1 -- start start 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edee3a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ed8153ee0 0x7f4ed80cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edee3a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed80ca330 0x7f4ed80ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edee3a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed815d240 con 0x7f4ed8159c90 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edee3a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed815d520 con 0x7f4ed8153ee0 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edee3a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80c4bc0 con 0x7f4ed80ca330 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edde38640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ed8153ee0 0x7f4ed80cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edde38640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ed8153ee0 0x7f4ed80cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57222/0 (socket says 172.21.15.7:57222) 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edde38640 1 -- 172.21.15.7:0/4207357941 learned_addr learned my addr 172.21.15.7:0/4207357941 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edd637640 1 --1- 172.21.15.7:0/4207357941 >> v1:172.21.15.7:6789/0 conn(0x7f4ed8159c90 0x7f4ed815d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58394/0 (socket says 172.21.15.7:58394) 2023-11-28T18:36:53.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4ede639640 1 --2- 172.21.15.7:0/4207357941 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed80ca330 0x7f4ed80ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edde38640 1 -- 172.21.15.7:0/4207357941 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed80ca330 msgr2=0x7f4ed80ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:53.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edde38640 1 --2- 172.21.15.7:0/4207357941 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed80ca330 0x7f4ed80ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edde38640 1 -- 172.21.15.7:0/4207357941 >> v1:172.21.15.7:6789/0 conn(0x7f4ed8159c90 legacy=0x7f4ed815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.639+0000 7f4edde38640 1 -- 172.21.15.7:0/4207357941 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed815a240 con 0x7f4ed8153ee0 2023-11-28T18:36:53.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edde38640 1 --2- 172.21.15.7:0/4207357941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed8153ee0 0x7f4ed80cb600 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4009870 tx=0x7f4ed401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81030bdf59f6db56 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:53.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edce36640 1 -- 172.21.15.7:0/4207357941 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed40a8a30 con 0x7f4ed8153ee0 2023-11-28T18:36:53.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edce36640 1 -- 172.21.15.7:0/4207357941 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed40a8b90 con 0x7f4ed8153ee0 2023-11-28T18:36:53.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edce36640 1 -- 172.21.15.7:0/4207357941 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed40b0440 con 0x7f4ed8153ee0 2023-11-28T18:36:53.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edee3a640 1 -- 172.21.15.7:0/4207357941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed8153ee0 msgr2=0x7f4ed80cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edee3a640 1 --2- 172.21.15.7:0/4207357941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed8153ee0 0x7f4ed80cb600 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4009870 tx=0x7f4ed401ed30 comp rx=0 tx=0).stop 2023-11-28T18:36:53.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edee3a640 1 -- 172.21.15.7:0/4207357941 shutdown_connections 2023-11-28T18:36:53.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edee3a640 1 --2- 172.21.15.7:0/4207357941 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed80ca330 0x7f4ed80ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edee3a640 1 --2- 172.21.15.7:0/4207357941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed8153ee0 0x7f4ed80cb600 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edee3a640 1 -- 172.21.15.7:0/4207357941 >> 172.21.15.7:0/4207357941 conn(0x7f4ed8071e20 msgr2=0x7f4ed8072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:53.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edee3a640 1 -- 172.21.15.7:0/4207357941 shutdown_connections 2023-11-28T18:36:53.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.643+0000 7f4edee3a640 1 -- 172.21.15.7:0/4207357941 wait complete. 2023-11-28T18:36:53.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 Processor -- start 2023-11-28T18:36:53.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 -- start start 2023-11-28T18:36:53.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed80ca330 0x7f4ed814e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed8153ee0 0x7f4ed815c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ed8159c90 0x7f4ed815c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80b3940 con 0x7f4ed80ca330 2023-11-28T18:36:53.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8145900 con 0x7f4ed8159c90 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed814f3f0 con 0x7f4ed8153ee0 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed80ca330 0x7f4ed814e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed80ca330 0x7f4ed814e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57232/0 (socket says 172.21.15.7:57232) 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 -- 172.21.15.7:0/2861136781 learned_addr learned my addr 172.21.15.7:0/2861136781 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edd637640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed8153ee0 0x7f4ed815c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4ede639640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ed8159c90 0x7f4ed815c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 -- 172.21.15.7:0/2861136781 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed8153ee0 msgr2=0x7f4ed815c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed8153ee0 0x7f4ed815c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 -- 172.21.15.7:0/2861136781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ed8159c90 msgr2=0x7f4ed815c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ed8159c90 0x7f4ed815c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 -- 172.21.15.7:0/2861136781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40094d0 con 0x7f4ed80ca330 2023-11-28T18:36:53.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edde38640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed80ca330 0x7f4ed814e740 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4ed401d3e0 tx=0x7f4ed401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:53.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4ece7fc640 1 -- 172.21.15.7:0/2861136781 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed40ba600 con 0x7f4ed80ca330 2023-11-28T18:36:53.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4ece7fc640 1 -- 172.21.15.7:0/2861136781 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed40ba780 con 0x7f4ed80ca330 2023-11-28T18:36:53.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4ece7fc640 1 -- 172.21.15.7:0/2861136781 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed40bab30 con 0x7f4ed80ca330 2023-11-28T18:36:53.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed80b35a0 con 0x7f4ed80ca330 2023-11-28T18:36:53.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.647+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed81f04b0 con 0x7f4ed80ca330 2023-11-28T18:36:53.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.651+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed804e890 con 0x7f4ed80ca330 2023-11-28T18:36:53.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.651+0000 7f4ece7fc640 1 -- 172.21.15.7:0/2861136781 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4ed40bac90 con 0x7f4ed80ca330 2023-11-28T18:36:53.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.651+0000 7f4ece7fc640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ebc066900 0x7f4ebc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:53.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.651+0000 7f4ece7fc640 1 -- 172.21.15.7:0/2861136781 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f4ed4122ce0 con 0x7f4ed80ca330 2023-11-28T18:36:53.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.651+0000 7f4edd637640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ebc066900 0x7f4ebc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:53.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.651+0000 7f4edd637640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ebc066900 0x7f4ebc068dc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80cfe80 tx=0x7f4ec8007cb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:53.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.655+0000 7f4ece7fc640 1 -- 172.21.15.7:0/2861136781 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4ed40efbb0 con 0x7f4ed80ca330 2023-11-28T18:36:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.871+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ed81511b0 con 0x7f4ed80ca330 2023-11-28T18:36:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.871+0000 7f4ece7fc640 1 -- 172.21.15.7:0/2861136781 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5761 (secure 0 0 0) 0x7f4ed40b5020 con 0x7f4ed80ca330 2023-11-28T18:36:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:53.875 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:36:52.304088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:53.875 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 60 2023-11-28T18:36:53.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ebc066900 msgr2=0x7f4ebc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ebc066900 0x7f4ebc068dc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80cfe80 tx=0x7f4ec8007cb0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed80ca330 msgr2=0x7f4ed814e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed80ca330 0x7f4ed814e740 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4ed401d3e0 tx=0x7f4ed401bad0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 shutdown_connections 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ebc066900 0x7f4ebc068dc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ed8159c90 0x7f4ed815c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4ed8153ee0 0x7f4ed815c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 --2- 172.21.15.7:0/2861136781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed80ca330 0x7f4ed814e740 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 >> 172.21.15.7:0/2861136781 conn(0x7f4ed8071e20 msgr2=0x7f4ed8159200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 shutdown_connections 2023-11-28T18:36:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:53.875+0000 7f4edee3a640 1 -- 172.21.15.7:0/2861136781 wait complete. 2023-11-28T18:36:53.898 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 60, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 60, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 33 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:36:52.304088+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5359, "mds_2": 5307, "mds_3": 5086, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:36:55.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:56.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.055+0000 7f1cc8501640 1 Processor -- start 2023-11-28T18:36:56.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.055+0000 7f1cc8501640 1 -- start start 2023-11-28T18:36:56.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc8501640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1cc01540e0 0x7f1cc00ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:56.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc8501640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc00cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:56.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc8501640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc015e380 con 0x7f1cc015a8a0 2023-11-28T18:36:56.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc8501640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc015d7e0 con 0x7f1cc01540e0 2023-11-28T18:36:56.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc8501640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc00cab80 con 0x7f1cc00cf6c0 2023-11-28T18:36:56.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc6a9e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc00cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:56.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc629d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1cc01540e0 0x7f1cc00ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:56.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc6a9e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc00cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:41570/0 (socket says 172.21.15.7:41570) 2023-11-28T18:36:56.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc6a9e640 1 -- 172.21.15.7:0/3878855838 learned_addr learned my addr 172.21.15.7:0/3878855838 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc629d640 1 -- 172.21.15.7:0/3878855838 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 msgr2=0x7f1cc00cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc629d640 1 --2- 172.21.15.7:0/3878855838 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc00cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc629d640 1 -- 172.21.15.7:0/3878855838 >> v1:172.21.15.193:6789/0 conn(0x7f1cc015a8a0 legacy=0x7f1cc015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc629d640 1 -- 172.21.15.7:0/3878855838 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc00cb850 con 0x7f1cc01540e0 2023-11-28T18:36:56.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.059+0000 7f1cc6a9e640 1 --2- 172.21.15.7:0/3878855838 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc00cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:36:56.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc629d640 1 --2- 172.21.15.7:0/3878855838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc01540e0 0x7f1cc00ca4d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0097c0 tx=0x7f1cbc01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e85e9f2f96d312b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:56.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc529b640 1 -- 172.21.15.7:0/3878855838 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0ab600 con 0x7f1cc01540e0 2023-11-28T18:36:56.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc529b640 1 -- 172.21.15.7:0/3878855838 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cbc0a7050 con 0x7f1cc01540e0 2023-11-28T18:36:56.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc529b640 1 -- 172.21.15.7:0/3878855838 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0ab980 con 0x7f1cc01540e0 2023-11-28T18:36:56.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc8501640 1 -- 172.21.15.7:0/3878855838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc01540e0 msgr2=0x7f1cc00ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc8501640 1 --2- 172.21.15.7:0/3878855838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc01540e0 0x7f1cc00ca4d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0097c0 tx=0x7f1cbc01c480 comp rx=0 tx=0).stop 2023-11-28T18:36:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc8501640 1 -- 172.21.15.7:0/3878855838 shutdown_connections 2023-11-28T18:36:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc8501640 1 --2- 172.21.15.7:0/3878855838 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc00cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc8501640 1 --2- 172.21.15.7:0/3878855838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc01540e0 0x7f1cc00ca4d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc8501640 1 -- 172.21.15.7:0/3878855838 >> 172.21.15.7:0/3878855838 conn(0x7f1cc00662e0 msgr2=0x7f1cc00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc8501640 1 -- 172.21.15.7:0/3878855838 shutdown_connections 2023-11-28T18:36:56.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.063+0000 7f1cc8501640 1 -- 172.21.15.7:0/3878855838 wait complete. 2023-11-28T18:36:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 Processor -- start 2023-11-28T18:36:56.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 -- start start 2023-11-28T18:36:56.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc0157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:56.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc01540e0 0x7f1cc0151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:56.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc015a8a0 0x7f1cc01f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:56.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc014bcf0 con 0x7f1cc01540e0 2023-11-28T18:36:56.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc01f4150 con 0x7f1cc015a8a0 2023-11-28T18:36:56.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0157ad0 con 0x7f1cc00cf6c0 2023-11-28T18:36:56.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc01540e0 0x7f1cc0151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc629d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc0157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc01540e0 0x7f1cc0151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57236/0 (socket says 172.21.15.7:57236) 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 -- 172.21.15.7:0/4098294439 learned_addr learned my addr 172.21.15.7:0/4098294439 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc6a9e640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc015a8a0 0x7f1cc01f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 -- 172.21.15.7:0/4098294439 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 msgr2=0x7f1cc0157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc0157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 -- 172.21.15.7:0/4098294439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc015a8a0 msgr2=0x7f1cc01f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc015a8a0 0x7f1cc01f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 -- 172.21.15.7:0/4098294439 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8009b30 con 0x7f1cc01540e0 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc5a9c640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc01540e0 0x7f1cc0151d30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000c3c0 tx=0x7f1cb000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cb6ffd640 1 -- 172.21.15.7:0/4098294439 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb00052d0 con 0x7f1cc01540e0 2023-11-28T18:36:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cb6ffd640 1 -- 172.21.15.7:0/4098294439 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb0005430 con 0x7f1cc01540e0 2023-11-28T18:36:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cb6ffd640 1 -- 172.21.15.7:0/4098294439 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0003840 con 0x7f1cc01540e0 2023-11-28T18:36:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cbc009420 con 0x7f1cc01540e0 2023-11-28T18:36:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.067+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc0152270 con 0x7f1cc01540e0 2023-11-28T18:36:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.071+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc0152da0 con 0x7f1cc01540e0 2023-11-28T18:36:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.071+0000 7f1cb6ffd640 1 -- 172.21.15.7:0/4098294439 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1cb00039a0 con 0x7f1cc01540e0 2023-11-28T18:36:56.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.071+0000 7f1cb6ffd640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1ca40668d0 0x7f1ca4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:56.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.071+0000 7f1cb6ffd640 1 -- 172.21.15.7:0/4098294439 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f1cb0083ad0 con 0x7f1cc01540e0 2023-11-28T18:36:56.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.071+0000 7f1cc629d640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1ca40668d0 0x7f1ca4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:56.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.071+0000 7f1cc629d640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1ca40668d0 0x7f1ca4068d90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0ad000 tx=0x7f1cbc0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:56.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.075+0000 7f1cb6ffd640 1 -- 172.21.15.7:0/4098294439 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1cb0050a40 con 0x7f1cc01540e0 2023-11-28T18:36:56.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cc01ec740 con 0x7f1cc01540e0 2023-11-28T18:36:56.309 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:56.310 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:36:52.304088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:56.310 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 60 2023-11-28T18:36:56.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cb6ffd640 1 -- 172.21.15.7:0/4098294439 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5761 (secure 0 0 0) 0x7f1cb0017020 con 0x7f1cc01540e0 2023-11-28T18:36:56.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1ca40668d0 msgr2=0x7f1ca4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:56.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1ca40668d0 0x7f1ca4068d90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0ad000 tx=0x7f1cbc0a6000 comp rx=0 tx=0).stop 2023-11-28T18:36:56.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc01540e0 msgr2=0x7f1cc0151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:56.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc01540e0 0x7f1cc0151d30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000c3c0 tx=0x7f1cb000c880 comp rx=0 tx=0).stop 2023-11-28T18:36:56.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 shutdown_connections 2023-11-28T18:36:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1ca40668d0 0x7f1ca4068d90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cc015a8a0 0x7f1cc01f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc01540e0 0x7f1cc0151d30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 --2- 172.21.15.7:0/4098294439 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1cc00cf6c0 0x7f1cc0157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 >> 172.21.15.7:0/4098294439 conn(0x7f1cc00662e0 msgr2=0x7f1cc00666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 shutdown_connections 2023-11-28T18:36:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:56.307+0000 7f1cc8501640 1 -- 172.21.15.7:0/4098294439 wait complete. 2023-11-28T18:36:56.560 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:36:58.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:36:58.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f935209a640 1 Processor -- start 2023-11-28T18:36:58.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f935209a640 1 -- start start 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f935209a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c1542e0 0x7f934c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f935209a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f934c0cbb70 0x7f934c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f935209a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934c0ca920 con 0x7f934c0cc810 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f935209a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c071d50 con 0x7f934c0cbb70 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f935209a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c072030 con 0x7f934c1542e0 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f934affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f934c0cbb70 0x7f934c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f934affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f934c0cbb70 0x7f934c0cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57250/0 (socket says 172.21.15.7:57250) 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f934affd640 1 -- 172.21.15.7:0/2848117689 learned_addr learned my addr 172.21.15.7:0/2848117689 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f934b7fe640 1 --2- 172.21.15.7:0/2848117689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c1542e0 0x7f934c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f934affd640 1 -- 172.21.15.7:0/2848117689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c1542e0 msgr2=0x7f934c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:36:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f934affd640 1 --2- 172.21.15.7:0/2848117689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c1542e0 0x7f934c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f934affd640 1 -- 172.21.15.7:0/2848117689 >> v1:172.21.15.193:6789/0 conn(0x7f934c0cc810 legacy=0x7f934c072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:58.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.731+0000 7f934affd640 1 -- 172.21.15.7:0/2848117689 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c15e550 con 0x7f934c0cbb70 2023-11-28T18:36:58.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f934affd640 1 --2- 172.21.15.7:0/2848117689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c0cbb70 0x7f934c0cc2d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9340009870 tx=0x7f934001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220416e4ece631bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:58.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f934a7fc640 1 -- 172.21.15.7:0/2848117689 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93400a9400 con 0x7f934c0cbb70 2023-11-28T18:36:58.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f934a7fc640 1 -- 172.21.15.7:0/2848117689 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93400a9560 con 0x7f934c0cbb70 2023-11-28T18:36:58.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f934a7fc640 1 -- 172.21.15.7:0/2848117689 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93400a98f0 con 0x7f934c0cbb70 2023-11-28T18:36:58.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f935209a640 1 -- 172.21.15.7:0/2848117689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c0cbb70 msgr2=0x7f934c0cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:58.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f935209a640 1 --2- 172.21.15.7:0/2848117689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c0cbb70 0x7f934c0cc2d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9340009870 tx=0x7f934001c540 comp rx=0 tx=0).stop 2023-11-28T18:36:58.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f935209a640 1 -- 172.21.15.7:0/2848117689 shutdown_connections 2023-11-28T18:36:58.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f935209a640 1 --2- 172.21.15.7:0/2848117689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c0cbb70 0x7f934c0cc2d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f935209a640 1 --2- 172.21.15.7:0/2848117689 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c1542e0 0x7f934c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f935209a640 1 -- 172.21.15.7:0/2848117689 >> 172.21.15.7:0/2848117689 conn(0x7f934c0717e0 msgr2=0x7f934c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:58.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f935209a640 1 -- 172.21.15.7:0/2848117689 shutdown_connections 2023-11-28T18:36:58.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.735+0000 7f935209a640 1 -- 172.21.15.7:0/2848117689 wait complete. 2023-11-28T18:36:58.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 Processor -- start 2023-11-28T18:36:58.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 -- start start 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f934c0cbb70 0x7f934c1472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c0cc810 0x7f934c0c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c1542e0 0x7f934c13b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c141030 con 0x7f934c1542e0 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c14b780 con 0x7f934c0cbb70 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c14bef0 con 0x7f934c0cc810 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c1542e0 0x7f934c13b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c1542e0 0x7f934c13b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57256/0 (socket says 172.21.15.7:57256) 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 -- 172.21.15.7:0/3200105449 learned_addr learned my addr 172.21.15.7:0/3200105449 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934affd640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c0cc810 0x7f934c0c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934b7fe640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f934c0cbb70 0x7f934c1472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 -- 172.21.15.7:0/3200105449 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c0cc810 msgr2=0x7f934c0c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c0cc810 0x7f934c0c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 -- 172.21.15.7:0/3200105449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f934c0cbb70 msgr2=0x7f934c1472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f934c0cbb70 0x7f934c1472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 -- 172.21.15.7:0/3200105449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93400094d0 con 0x7f934c1542e0 2023-11-28T18:36:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f934bfff640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c1542e0 0x7f934c13b1b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f934400c770 tx=0x7f934400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f9333fff640 1 -- 172.21.15.7:0/3200105449 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344003b80 con 0x7f934c1542e0 2023-11-28T18:36:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f9333fff640 1 -- 172.21.15.7:0/3200105449 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9344003ce0 con 0x7f934c1542e0 2023-11-28T18:36:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.739+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c1414c0 con 0x7f934c1542e0 2023-11-28T18:36:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.743+0000 7f9333fff640 1 -- 172.21.15.7:0/3200105449 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934400aa90 con 0x7f934c1542e0 2023-11-28T18:36:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.743+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934c0c6d60 con 0x7f934c1542e0 2023-11-28T18:36:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.743+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934c06c670 con 0x7f934c1542e0 2023-11-28T18:36:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.743+0000 7f9333fff640 1 -- 172.21.15.7:0/3200105449 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f934400abf0 con 0x7f934c1542e0 2023-11-28T18:36:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.743+0000 7f9333fff640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93380668d0 0x7f9338068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:36:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.743+0000 7f9333fff640 1 -- 172.21.15.7:0/3200105449 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f9344084530 con 0x7f934c1542e0 2023-11-28T18:36:58.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.743+0000 7f934b7fe640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93380668d0 0x7f9338068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:36:58.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.747+0000 7f934b7fe640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93380668d0 0x7f9338068d90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f933c0079c0 tx=0x7f933c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:36:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.751+0000 7f9333fff640 1 -- 172.21.15.7:0/3200105449 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f93440514b0 con 0x7f934c1542e0 2023-11-28T18:36:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.967+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f934c04e890 con 0x7f934c1542e0 2023-11-28T18:36:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.967+0000 7f9333fff640 1 -- 172.21.15.7:0/3200105449 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5763 (secure 0 0 0) 0x7f9344054ad0 con 0x7f934c1542e0 2023-11-28T18:36:58.970 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:36:58.971 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:36:57.319430+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:resolve","state_seq":23,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:36:58.971 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 61 2023-11-28T18:36:58.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93380668d0 msgr2=0x7f9338068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:58.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93380668d0 0x7f9338068d90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f933c0079c0 tx=0x7f933c008040 comp rx=0 tx=0).stop 2023-11-28T18:36:58.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c1542e0 msgr2=0x7f934c13b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:36:58.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c1542e0 0x7f934c13b1b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f934400c770 tx=0x7f934400cc30 comp rx=0 tx=0).stop 2023-11-28T18:36:58.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 shutdown_connections 2023-11-28T18:36:58.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93380668d0 0x7f9338068d90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f934c1542e0 0x7f934c13b1b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f934c0cc810 0x7f934c0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 --2- 172.21.15.7:0/3200105449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f934c0cbb70 0x7f934c1472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:36:58.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 >> 172.21.15.7:0/3200105449 conn(0x7f934c0717e0 msgr2=0x7f934c15dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:36:58.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 shutdown_connections 2023-11-28T18:36:58.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:36:58.971+0000 7f935209a640 1 -- 172.21.15.7:0/3200105449 wait complete. 2023-11-28T18:36:58.995 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:37:00.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:37:01.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d95f5640 1 Processor -- start 2023-11-28T18:37:01.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d95f5640 1 -- start start 2023-11-28T18:37:01.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d95f5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3d4154070 0x7fc3d40c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:01.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d95f5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc3d406b620 0x7fc3d415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:01.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d95f5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d4154450 con 0x7fc3d40d0e50 2023-11-28T18:37:01.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d95f5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d40c63c0 con 0x7fc3d4154070 2023-11-28T18:37:01.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d95f5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d40cba90 con 0x7fc3d406b620 2023-11-28T18:37:01.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d2ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3d4154070 0x7fc3d40c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:01.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d2ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3d4154070 0x7fc3d40c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57272/0 (socket says 172.21.15.7:57272) 2023-11-28T18:37:01.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.147+0000 7fc3d2ffd640 1 -- 172.21.15.7:0/1011867759 learned_addr learned my addr 172.21.15.7:0/1011867759 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:01.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.151+0000 7fc3d27fc640 1 --2- 172.21.15.7:0/1011867759 >> v2:172.21.15.193:3300/0 conn(0x7fc3d406b620 0x7fc3d415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:01.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.151+0000 7fc3d2ffd640 1 -- 172.21.15.7:0/1011867759 >> v2:172.21.15.193:3300/0 conn(0x7fc3d406b620 msgr2=0x7fc3d415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.151+0000 7fc3d2ffd640 1 --2- 172.21.15.7:0/1011867759 >> v2:172.21.15.193:3300/0 conn(0x7fc3d406b620 0x7fc3d415b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.151+0000 7fc3d2ffd640 1 -- 172.21.15.7:0/1011867759 >> v1:172.21.15.193:6789/0 conn(0x7fc3d40d0e50 legacy=0x7fc3d415d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.151+0000 7fc3d2ffd640 1 -- 172.21.15.7:0/1011867759 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40c7ce0 con 0x7fc3d4154070 2023-11-28T18:37:01.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.151+0000 7fc3d2ffd640 1 --2- 172.21.15.7:0/1011867759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4154070 0x7fc3d40c5e80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc3d406dc00 tx=0x7fc3c401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e3b560efa564e5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d1ffb640 1 -- 172.21.15.7:0/1011867759 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c40a89f0 con 0x7fc3d4154070 2023-11-28T18:37:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d1ffb640 1 -- 172.21.15.7:0/1011867759 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c40a8b50 con 0x7fc3d4154070 2023-11-28T18:37:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d1ffb640 1 -- 172.21.15.7:0/1011867759 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c40b0440 con 0x7fc3d4154070 2023-11-28T18:37:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1011867759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4154070 msgr2=0x7fc3d40c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1011867759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4154070 0x7fc3d40c5e80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc3d406dc00 tx=0x7fc3c401ecf0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1011867759 shutdown_connections 2023-11-28T18:37:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1011867759 >> v2:172.21.15.193:3300/0 conn(0x7fc3d406b620 0x7fc3d415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1011867759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4154070 0x7fc3d40c5e80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1011867759 >> 172.21.15.7:0/1011867759 conn(0x7fc3d408cff0 msgr2=0x7fc3d408d400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1011867759 shutdown_connections 2023-11-28T18:37:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1011867759 wait complete. 2023-11-28T18:37:01.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 Processor -- start 2023-11-28T18:37:01.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.155+0000 7fc3d95f5640 1 -- start start 2023-11-28T18:37:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d406b620 0x7fc3d40cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc3d40d0e50 0x7fc3d415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d4154070 0x7fc3d414ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41f38a0 con 0x7fc3d406b620 2023-11-28T18:37:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41f3b00 con 0x7fc3d4154070 2023-11-28T18:37:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d415b050 con 0x7fc3d40d0e50 2023-11-28T18:37:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d406b620 0x7fc3d40cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d406b620 0x7fc3d40cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57284/0 (socket says 172.21.15.7:57284) 2023-11-28T18:37:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 -- 172.21.15.7:0/1029689900 learned_addr learned my addr 172.21.15.7:0/1029689900 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d27fc640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc3d40d0e50 0x7fc3d415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d3fff640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d4154070 0x7fc3d414ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 -- 172.21.15.7:0/1029689900 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc3d40d0e50 msgr2=0x7fc3d415dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc3d40d0e50 0x7fc3d415dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 -- 172.21.15.7:0/1029689900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d4154070 msgr2=0x7fc3d414ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d4154070 0x7fc3d414ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 -- 172.21.15.7:0/1029689900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c40094d0 con 0x7fc3d406b620 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d2ffd640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d406b620 0x7fc3d40cd4a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc3c401b900 tx=0x7fc3c401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3bb7fe640 1 -- 172.21.15.7:0/1029689900 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c40ba600 con 0x7fc3d406b620 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3bb7fe640 1 -- 172.21.15.7:0/1029689900 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c40ba760 con 0x7fc3d406b620 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3bb7fe640 1 -- 172.21.15.7:0/1029689900 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c40bab10 con 0x7fc3d406b620 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d415b330 con 0x7fc3d406b620 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d415b8f0 con 0x7fc3d406b620 2023-11-28T18:37:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.159+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d404e890 con 0x7fc3d406b620 2023-11-28T18:37:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.163+0000 7fc3bb7fe640 1 -- 172.21.15.7:0/1029689900 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc3c40bac70 con 0x7fc3d406b620 2023-11-28T18:37:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.163+0000 7fc3bb7fe640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc3c00668d0 0x7fc3c0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.163+0000 7fc3bb7fe640 1 -- 172.21.15.7:0/1029689900 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fc3c4122ba0 con 0x7fc3d406b620 2023-11-28T18:37:01.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.163+0000 7fc3d27fc640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc3c00668d0 0x7fc3c0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:01.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.163+0000 7fc3d27fc640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc3c00668d0 0x7fc3c0068d90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc3d41575a0 tx=0x7fc3c8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:01.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.167+0000 7fc3bb7fe640 1 -- 172.21.15.7:0/1029689900 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc3c40efb20 con 0x7fc3d406b620 2023-11-28T18:37:01.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:37:01.257 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:37:01.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3d41528a0 con 0x7fc3d406b620 2023-11-28T18:37:01.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3bb7fe640 1 -- 172.21.15.7:0/1029689900 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5763 (secure 0 0 0) 0x7fc3c40b5020 con 0x7fc3d406b620 2023-11-28T18:37:01.401 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:37:01.401 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:36:57.319430+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:resolve","state_seq":23,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:37:01.401 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 61 2023-11-28T18:37:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc3c00668d0 msgr2=0x7fc3c0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc3c00668d0 0x7fc3c0068d90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc3d41575a0 tx=0x7fc3c8009190 comp rx=0 tx=0).stop 2023-11-28T18:37:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d406b620 msgr2=0x7fc3d40cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d406b620 0x7fc3d40cd4a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc3c401b900 tx=0x7fc3c401ba90 comp rx=0 tx=0).stop 2023-11-28T18:37:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 shutdown_connections 2023-11-28T18:37:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc3c00668d0 0x7fc3c0068d90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d4154070 0x7fc3d414ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc3d40d0e50 0x7fc3d415dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 --2- 172.21.15.7:0/1029689900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d406b620 0x7fc3d40cd4a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 >> 172.21.15.7:0/1029689900 conn(0x7fc3d408cff0 msgr2=0x7fc3d4152ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 shutdown_connections 2023-11-28T18:37:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:01.399+0000 7fc3d95f5640 1 -- 172.21.15.7:0/1029689900 wait complete. 2023-11-28T18:37:01.427 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:37:03.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:37:03.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.583+0000 7f3156f9d640 1 Processor -- start 2023-11-28T18:37:03.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.583+0000 7f3156f9d640 1 -- start start 2023-11-28T18:37:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.583+0000 7f3156f9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f315015d350 0x7f315006a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.583+0000 7f3156f9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f315006ab30 0x7f31500c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.583+0000 7f3156f9d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31500c8750 con 0x7f31501540a0 2023-11-28T18:37:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.583+0000 7f3156f9d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31500c8a30 con 0x7f315015d350 2023-11-28T18:37:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.583+0000 7f3156f9d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31500731c0 con 0x7f315006ab30 2023-11-28T18:37:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f315015d350 0x7f315006a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f315015d350 0x7f315006a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51582/0 (socket says 172.21.15.7:51582) 2023-11-28T18:37:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f3154d39640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f31501540a0 0x7f3150072ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35546/0 (socket says 172.21.15.7:35546) 2023-11-28T18:37:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314ffff640 1 -- 172.21.15.7:0/2066482773 learned_addr learned my addr 172.21.15.7:0/2066482773 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f315553a640 1 --2- 172.21.15.7:0/2066482773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f315006ab30 0x7f31500c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314ffff640 1 -- 172.21.15.7:0/2066482773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f315006ab30 msgr2=0x7f31500c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314ffff640 1 --2- 172.21.15.7:0/2066482773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f315006ab30 0x7f31500c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314ffff640 1 -- 172.21.15.7:0/2066482773 >> v1:172.21.15.7:6789/0 conn(0x7f31501540a0 legacy=0x7f3150072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314ffff640 1 -- 172.21.15.7:0/2066482773 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315015d840 con 0x7f315015d350 2023-11-28T18:37:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314f7fe640 1 -- 172.21.15.7:0/2066482773 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116269689 0 0) 0x7f31500c8750 con 0x7f31501540a0 2023-11-28T18:37:03.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.587+0000 7f314ffff640 1 --2- 172.21.15.7:0/2066482773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315015d350 0x7f315006a5f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f313c009870 tx=0x7f313c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a03483d52ebcd9e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:03.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f314f7fe640 1 -- 172.21.15.7:0/2066482773 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c0ab600 con 0x7f315015d350 2023-11-28T18:37:03.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f314f7fe640 1 -- 172.21.15.7:0/2066482773 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313c0a7050 con 0x7f315015d350 2023-11-28T18:37:03.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f314f7fe640 1 -- 172.21.15.7:0/2066482773 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c0ab980 con 0x7f315015d350 2023-11-28T18:37:03.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- 172.21.15.7:0/2066482773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315015d350 msgr2=0x7f315006a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:03.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2066482773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315015d350 0x7f315006a5f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f313c009870 tx=0x7f313c01c540 comp rx=0 tx=0).stop 2023-11-28T18:37:03.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- 172.21.15.7:0/2066482773 shutdown_connections 2023-11-28T18:37:03.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2066482773 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f315006ab30 0x7f31500c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2066482773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315015d350 0x7f315006a5f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- 172.21.15.7:0/2066482773 >> 172.21.15.7:0/2066482773 conn(0x7f3150073cd0 msgr2=0x7f31500662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:03.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- 172.21.15.7:0/2066482773 shutdown_connections 2023-11-28T18:37:03.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- 172.21.15.7:0/2066482773 wait complete. 2023-11-28T18:37:03.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 Processor -- start 2023-11-28T18:37:03.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- start start 2023-11-28T18:37:03.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315006ab30 0x7f31501f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:03.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31501540a0 0x7f31500c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:03.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f315015d350 0x7f31501f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:03.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31501f42a0 con 0x7f315006ab30 2023-11-28T18:37:03.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31501573f0 con 0x7f315015d350 2023-11-28T18:37:03.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.591+0000 7f3156f9d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3150152b40 con 0x7f31501540a0 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315006ab30 0x7f31501f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315006ab30 0x7f31501f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51598/0 (socket says 172.21.15.7:51598) 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 -- 172.21.15.7:0/2033876382 learned_addr learned my addr 172.21.15.7:0/2033876382 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f315553a640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f315015d350 0x7f31501f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f314ffff640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31501540a0 0x7f31500c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 -- 172.21.15.7:0/2033876382 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31501540a0 msgr2=0x7f31500c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31501540a0 0x7f31500c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 -- 172.21.15.7:0/2033876382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f315015d350 msgr2=0x7f31501f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f315015d350 0x7f31501f46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 -- 172.21.15.7:0/2033876382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c0094d0 con 0x7f315006ab30 2023-11-28T18:37:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3154d39640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315006ab30 0x7f31501f6870 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f31440011f0 tx=0x7f314400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f314d7fa640 1 -- 172.21.15.7:0/2033876382 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31440074e0 con 0x7f315006ab30 2023-11-28T18:37:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f314d7fa640 1 -- 172.21.15.7:0/2033876382 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3144007640 con 0x7f315006ab30 2023-11-28T18:37:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f314d7fa640 1 -- 172.21.15.7:0/2033876382 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144004a10 con 0x7f315006ab30 2023-11-28T18:37:03.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31501f6db0 con 0x7f315006ab30 2023-11-28T18:37:03.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3150152860 con 0x7f315006ab30 2023-11-28T18:37:03.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315004e890 con 0x7f315006ab30 2023-11-28T18:37:03.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.595+0000 7f314d7fa640 1 -- 172.21.15.7:0/2033876382 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3144004b70 con 0x7f315006ab30 2023-11-28T18:37:03.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.599+0000 7f314d7fa640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31340668d0 0x7f3134068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:03.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.599+0000 7f314d7fa640 1 -- 172.21.15.7:0/2033876382 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f3144083960 con 0x7f315006ab30 2023-11-28T18:37:03.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.599+0000 7f314ffff640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31340668d0 0x7f3134068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:03.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.599+0000 7f314ffff640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31340668d0 0x7f3134068d90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f31500ce820 tx=0x7f313c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:03.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.603+0000 7f314d7fa640 1 -- 172.21.15.7:0/2033876382 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3144050980 con 0x7f315006ab30 2023-11-28T18:37:03.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.815+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3150072670 con 0x7f315006ab30 2023-11-28T18:37:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.815+0000 7f314d7fa640 1 -- 172.21.15.7:0/2033876382 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f3150072670 con 0x7f315006ab30 2023-11-28T18:37:03.819 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:37:03.819 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:03.312620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:37:03.820 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 62 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31340668d0 msgr2=0x7f3134068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31340668d0 0x7f3134068d90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f31500ce820 tx=0x7f313c0a6000 comp rx=0 tx=0).stop 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315006ab30 msgr2=0x7f31501f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315006ab30 0x7f31501f6870 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f31440011f0 tx=0x7f314400c9b0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 shutdown_connections 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31340668d0 0x7f3134068d90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f315015d350 0x7f31501f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31501540a0 0x7f31500c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 --2- 172.21.15.7:0/2033876382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315006ab30 0x7f31501f6870 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 >> 172.21.15.7:0/2033876382 conn(0x7f3150073cd0 msgr2=0x7f31500662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 shutdown_connections 2023-11-28T18:37:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:03.819+0000 7f3156f9d640 1 -- 172.21.15.7:0/2033876382 wait complete. 2023-11-28T18:37:03.843 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:37:05.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:37:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb39ebb640 1 Processor -- start 2023-11-28T18:37:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb39ebb640 1 -- start start 2023-11-28T18:37:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb39ebb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbb34154120 0x7fbb3406f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb39ebb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb340c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb39ebb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3415d3c0 con 0x7fbb340c9360 2023-11-28T18:37:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb39ebb640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb34159c30 con 0x7fbb34154120 2023-11-28T18:37:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb39ebb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3415d6a0 con 0x7fbb3406ed80 2023-11-28T18:37:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb33fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb340c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb38eb9640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbb34154120 0x7fbb3406f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb38eb9640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbb34154120 0x7fbb3406f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:59306/0 (socket says 172.21.15.7:59306) 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb38eb9640 1 -- 172.21.15.7:0/1383369107 learned_addr learned my addr 172.21.15.7:0/1383369107 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb396ba640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fbb340c9360 0x7fbb3415a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:44798/0 (socket says 172.21.15.7:44798) 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.991+0000 7fbb33fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb340c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:45922/0 (socket says 172.21.15.7:45922) 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb38eb9640 1 -- 172.21.15.7:0/1383369107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 msgr2=0x7fbb340c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb38eb9640 1 --2- 172.21.15.7:0/1383369107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb340c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb38eb9640 1 -- 172.21.15.7:0/1383369107 >> v1:172.21.15.193:6789/0 conn(0x7fbb340c9360 legacy=0x7fbb3415a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb38eb9640 1 -- 172.21.15.7:0/1383369107 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3415d980 con 0x7fbb34154120 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb337fe640 1 -- 172.21.15.7:0/1383369107 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187503051 0 0) 0x7fbb3415d3c0 con 0x7fbb340c9360 2023-11-28T18:37:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb33fff640 1 --2- 172.21.15.7:0/1383369107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb340c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:37:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb38eb9640 1 --2- 172.21.15.7:0/1383369107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb34154120 0x7fbb3406f5d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009870 tx=0x7fbb2801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=292318aa59c4cca2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:05.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb337fe640 1 -- 172.21.15.7:0/1383369107 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280a8a30 con 0x7fbb34154120 2023-11-28T18:37:05.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb337fe640 1 -- 172.21.15.7:0/1383369107 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb280a8b90 con 0x7fbb34154120 2023-11-28T18:37:05.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.995+0000 7fbb337fe640 1 -- 172.21.15.7:0/1383369107 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280b0440 con 0x7fbb34154120 2023-11-28T18:37:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 -- 172.21.15.7:0/1383369107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb34154120 msgr2=0x7fbb3406f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/1383369107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb34154120 0x7fbb3406f5d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009870 tx=0x7fbb2801ed30 comp rx=0 tx=0).stop 2023-11-28T18:37:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 -- 172.21.15.7:0/1383369107 shutdown_connections 2023-11-28T18:37:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/1383369107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb340c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/1383369107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb34154120 0x7fbb3406f5d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 -- 172.21.15.7:0/1383369107 >> 172.21.15.7:0/1383369107 conn(0x7fbb3406a5f0 msgr2=0x7fbb3406aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 -- 172.21.15.7:0/1383369107 shutdown_connections 2023-11-28T18:37:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 -- 172.21.15.7:0/1383369107 wait complete. 2023-11-28T18:37:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 Processor -- start 2023-11-28T18:37:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 -- start start 2023-11-28T18:37:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb3414fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340c9360 0x7fbb341ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:05.999+0000 7fbb39ebb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb34154120 0x7fbb341494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb39ebb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3414d930 con 0x7fbb34154120 2023-11-28T18:37:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb39ebb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb340c62d0 con 0x7fbb340c9360 2023-11-28T18:37:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb39ebb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3414d320 con 0x7fbb3406ed80 2023-11-28T18:37:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb396ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb34154120 0x7fbb341494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb396ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb34154120 0x7fbb341494a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51606/0 (socket says 172.21.15.7:51606) 2023-11-28T18:37:06.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb396ba640 1 -- 172.21.15.7:0/2643371736 learned_addr learned my addr 172.21.15.7:0/2643371736 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:06.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb38eb9640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb3414fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:06.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb33fff640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340c9360 0x7fbb341ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb38eb9640 1 -- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340c9360 msgr2=0x7fbb341ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb38eb9640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340c9360 0x7fbb341ef370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb38eb9640 1 -- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb34154120 msgr2=0x7fbb341494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb38eb9640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb34154120 0x7fbb341494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb38eb9640 1 -- 172.21.15.7:0/2643371736 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb20009420 con 0x7fbb3406ed80 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb396ba640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb34154120 0x7fbb341494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb38eb9640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb3414fc20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb280af000 tx=0x7fbb2801bad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb317fa640 1 -- 172.21.15.7:0/2643371736 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280be070 con 0x7fbb3406ed80 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb280094d0 con 0x7fbb3406ed80 2023-11-28T18:37:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb317fa640 1 -- 172.21.15.7:0/2643371736 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb280ba400 con 0x7fbb3406ed80 2023-11-28T18:37:06.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb317fa640 1 -- 172.21.15.7:0/2643371736 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280ba7d0 con 0x7fbb3406ed80 2023-11-28T18:37:06.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3414d5c0 con 0x7fbb3406ed80 2023-11-28T18:37:06.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.003+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3404e890 con 0x7fbb3406ed80 2023-11-28T18:37:06.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.007+0000 7fbb317fa640 1 -- 172.21.15.7:0/2643371736 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbb280c3410 con 0x7fbb3406ed80 2023-11-28T18:37:06.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.007+0000 7fbb317fa640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb180668d0 0x7fbb18068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:06.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.007+0000 7fbb317fa640 1 -- 172.21.15.7:0/2643371736 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fbb28122100 con 0x7fbb3406ed80 2023-11-28T18:37:06.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.007+0000 7fbb33fff640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb180668d0 0x7fbb18068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:06.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.007+0000 7fbb33fff640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb180668d0 0x7fbb18068d90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbb340cbed0 tx=0x7fbb20009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:06.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.011+0000 7fbb317fa640 1 -- 172.21.15.7:0/2643371736 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbb280ef080 con 0x7fbb3406ed80 2023-11-28T18:37:06.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.259+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb34073320 con 0x7fbb3406ed80 2023-11-28T18:37:06.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.259+0000 7fbb317fa640 1 -- 172.21.15.7:0/2643371736 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5765 (secure 0 0 0) 0x7fbb280f26a0 con 0x7fbb3406ed80 2023-11-28T18:37:06.261 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:37:06.262 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:03.312620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:37:06.262 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 62 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.259+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb180668d0 msgr2=0x7fbb18068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.259+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb180668d0 0x7fbb18068d90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbb340cbed0 tx=0x7fbb20009190 comp rx=0 tx=0).stop 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.259+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 msgr2=0x7fbb3414fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.259+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb3414fc20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb280af000 tx=0x7fbb2801bad0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.263+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 shutdown_connections 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.263+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb180668d0 0x7fbb18068d90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.263+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb34154120 0x7fbb341494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.263+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb340c9360 0x7fbb341ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.263+0000 7fbb39ebb640 1 --2- 172.21.15.7:0/2643371736 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb3406ed80 0x7fbb3414fc20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:06.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.263+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 >> 172.21.15.7:0/2643371736 conn(0x7fbb3406a5f0 msgr2=0x7fbb340cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:06.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.263+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 shutdown_connections 2023-11-28T18:37:06.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:06.263+0000 7fbb39ebb640 1 -- 172.21.15.7:0/2643371736 wait complete. 2023-11-28T18:37:06.286 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5086": { "addr": "172.21.15.193:6835/650087483", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 650087483, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 650087483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5086, "incarnation": 36, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 33 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:reconnect", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:37:03.312620+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5359, "mds_2": 5307, "mds_3": 5086, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:37:08.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:37:08.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.455+0000 7f822759e640 1 Processor -- start 2023-11-28T18:37:08.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.455+0000 7f822759e640 1 -- start start 2023-11-28T18:37:08.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.455+0000 7f822759e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8228154010 0x7f82280725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.455+0000 7f822759e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f822815b3c0 0x7f822815b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f822759e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82281543f0 con 0x7f822815dee0 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f822759e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8228072e80 con 0x7f822815b3c0 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f822759e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8228159fe0 con 0x7f8228154010 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f8226d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f822815b3c0 0x7f822815b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f821ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f822815dee0 0x7f822806a5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35560/0 (socket says 172.21.15.7:35560) 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f821ffff640 1 -- 172.21.15.7:0/3282461138 learned_addr learned my addr 172.21.15.7:0/3282461138 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f822659c640 1 --2- 172.21.15.7:0/3282461138 >> v2:172.21.15.193:3300/0 conn(0x7f8228154010 0x7f82280725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f8226d9d640 1 -- 172.21.15.7:0/3282461138 >> v2:172.21.15.193:3300/0 conn(0x7f8228154010 msgr2=0x7f82280725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f8226d9d640 1 --2- 172.21.15.7:0/3282461138 >> v2:172.21.15.193:3300/0 conn(0x7f8228154010 0x7f82280725b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f8226d9d640 1 -- 172.21.15.7:0/3282461138 >> v1:172.21.15.7:6789/0 conn(0x7f822815dee0 legacy=0x7f822806a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:08.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f8226d9d640 1 -- 172.21.15.7:0/3282461138 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822815a2c0 con 0x7f822815b3c0 2023-11-28T18:37:08.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f8225d9b640 1 -- 172.21.15.7:0/3282461138 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174095377 0 0) 0x7f82281543f0 con 0x7f822815dee0 2023-11-28T18:37:08.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.459+0000 7f822659c640 1 --2- 172.21.15.7:0/3282461138 >> v2:172.21.15.193:3300/0 conn(0x7f8228154010 0x7f82280725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:37:08.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f8226d9d640 1 --2- 172.21.15.7:0/3282461138 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f822815b7d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8228072540 tx=0x7f822001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8a4fae5cc203e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:08.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f8225d9b640 1 -- 172.21.15.7:0/3282461138 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82200a59a0 con 0x7f822815b3c0 2023-11-28T18:37:08.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f8225d9b640 1 -- 172.21.15.7:0/3282461138 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82200a5b00 con 0x7f822815b3c0 2023-11-28T18:37:08.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f8225d9b640 1 -- 172.21.15.7:0/3282461138 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82200a5e90 con 0x7f822815b3c0 2023-11-28T18:37:08.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 -- 172.21.15.7:0/3282461138 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 msgr2=0x7f822815b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:08.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 --2- 172.21.15.7:0/3282461138 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f822815b7d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8228072540 tx=0x7f822001c8e0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 -- 172.21.15.7:0/3282461138 shutdown_connections 2023-11-28T18:37:08.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 --2- 172.21.15.7:0/3282461138 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f822815b7d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 --2- 172.21.15.7:0/3282461138 >> v2:172.21.15.193:3300/0 conn(0x7f8228154010 0x7f82280725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 -- 172.21.15.7:0/3282461138 >> 172.21.15.7:0/3282461138 conn(0x7f8228073ea0 msgr2=0x7f82280662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:08.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 -- 172.21.15.7:0/3282461138 shutdown_connections 2023-11-28T18:37:08.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 -- 172.21.15.7:0/3282461138 wait complete. 2023-11-28T18:37:08.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 Processor -- start 2023-11-28T18:37:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.463+0000 7f822759e640 1 -- start start 2023-11-28T18:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8228154010 0x7f82280cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f8228158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f822815dee0 0x7f8228141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82281f2f20 con 0x7f822815b3c0 2023-11-28T18:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82280cb7f0 con 0x7f8228154010 2023-11-28T18:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82280c5fe0 con 0x7f822815dee0 2023-11-28T18:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f8228158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f8228158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51624/0 (socket says 172.21.15.7:51624) 2023-11-28T18:37:08.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 -- 172.21.15.7:0/413800590 learned_addr learned my addr 172.21.15.7:0/413800590 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822659c640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8228154010 0x7f82280cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f8226d9d640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f822815dee0 0x7f8228141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 -- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f822815dee0 msgr2=0x7f8228141d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f822815dee0 0x7f8228141d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 -- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8228154010 msgr2=0x7f82280cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8228154010 0x7f82280cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 -- 172.21.15.7:0/413800590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8220009a70 con 0x7f822815b3c0 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f8226d9d640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f822815dee0 0x7f8228141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822659c640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8228154010 0x7f82280cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821ffff640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f8228158f60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8210007fd0 tx=0x7f821000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821effd640 1 -- 172.21.15.7:0/413800590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210005cf0 con 0x7f822815b3c0 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821effd640 1 -- 172.21.15.7:0/413800590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8210005e70 con 0x7f822815b3c0 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82280cb530 con 0x7f822815b3c0 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821effd640 1 -- 172.21.15.7:0/413800590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82100072f0 con 0x7f822815b3c0 2023-11-28T18:37:08.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82281f2b40 con 0x7f822815b3c0 2023-11-28T18:37:08.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82280c6690 con 0x7f822815b3c0 2023-11-28T18:37:08.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.467+0000 7f821effd640 1 -- 172.21.15.7:0/413800590 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8210007450 con 0x7f822815b3c0 2023-11-28T18:37:08.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.471+0000 7f821effd640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f82080668d0 0x7f8208068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:08.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.471+0000 7f821effd640 1 -- 172.21.15.7:0/413800590 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f8210083dc0 con 0x7f822815b3c0 2023-11-28T18:37:08.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.471+0000 7f822659c640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f82080668d0 0x7f8208068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:08.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.471+0000 7f822659c640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f82080668d0 0x7f8208068d90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8218009980 tx=0x7f8218008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:08.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.475+0000 7f821effd640 1 -- 172.21.15.7:0/413800590 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f82280c6690 con 0x7f822815b3c0 2023-11-28T18:37:08.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.719+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82281f29a0 con 0x7f822815b3c0 2023-11-28T18:37:08.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.719+0000 7f821effd640 1 -- 172.21.15.7:0/413800590 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5762 (secure 0 0 0) 0x7f82281f29a0 con 0x7f822815b3c0 2023-11-28T18:37:08.723 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:37:08.723 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:08.323828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:37:08.723 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 63 2023-11-28T18:37:08.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f82080668d0 msgr2=0x7f8208068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:08.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f82080668d0 0x7f8208068d90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8218009980 tx=0x7f8218008040 comp rx=0 tx=0).stop 2023-11-28T18:37:08.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 msgr2=0x7f8228158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:08.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f8228158f60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8210007fd0 tx=0x7f821000c8e0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 shutdown_connections 2023-11-28T18:37:08.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f82080668d0 0x7f8208068d90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f822815dee0 0x7f8228141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822815b3c0 0x7f8228158f60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 --2- 172.21.15.7:0/413800590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8228154010 0x7f82280cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:08.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 >> 172.21.15.7:0/413800590 conn(0x7f8228073ea0 msgr2=0x7f8228149f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:08.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 shutdown_connections 2023-11-28T18:37:08.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:08.723+0000 7f822759e640 1 -- 172.21.15.7:0/413800590 wait complete. 2023-11-28T18:37:08.749 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:37:10.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:37:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff19f8640 1 Processor -- start 2023-11-28T18:37:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff19f8640 1 -- start start 2023-11-28T18:37:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff19f8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7fec15ca30 0x7f7fec159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff19f8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7fec15ce10 0x7f7fec0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff19f8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fec15a620 con 0x7f7fec0cafe0 2023-11-28T18:37:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff19f8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec15a900 con 0x7f7fec15ca30 2023-11-28T18:37:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff19f8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec15abe0 con 0x7f7fec15ce10 2023-11-28T18:37:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff09f6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7fec15ca30 0x7f7fec159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff09f6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7fec15ca30 0x7f7fec159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51638/0 (socket says 172.21.15.7:51638) 2023-11-28T18:37:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff09f6640 1 -- 172.21.15.7:0/2042636879 learned_addr learned my addr 172.21.15.7:0/2042636879 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:10.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7fe3fff640 1 --2- 172.21.15.7:0/2042636879 >> v2:172.21.15.193:3300/0 conn(0x7f7fec15ce10 0x7f7fec0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:10.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff09f6640 1 -- 172.21.15.7:0/2042636879 >> v2:172.21.15.193:3300/0 conn(0x7f7fec15ce10 msgr2=0x7f7fec0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:10.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff09f6640 1 --2- 172.21.15.7:0/2042636879 >> v2:172.21.15.193:3300/0 conn(0x7f7fec15ce10 0x7f7fec0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:10.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff09f6640 1 -- 172.21.15.7:0/2042636879 >> v1:172.21.15.193:6789/0 conn(0x7f7fec0cafe0 legacy=0x7f7fec0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:10.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.927+0000 7f7ff09f6640 1 -- 172.21.15.7:0/2042636879 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec0cbb10 con 0x7f7fec15ca30 2023-11-28T18:37:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.931+0000 7f7ff09f6640 1 --2- 172.21.15.7:0/2042636879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ca30 0x7f7fec159c90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0d0200 tx=0x7f7fe801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bb01bfaff51f03 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.931+0000 7f7fe37fe640 1 -- 172.21.15.7:0/2042636879 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe80ac640 con 0x7f7fec15ca30 2023-11-28T18:37:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.931+0000 7f7fe37fe640 1 -- 172.21.15.7:0/2042636879 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe80ac7a0 con 0x7f7fec15ca30 2023-11-28T18:37:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.931+0000 7f7fe37fe640 1 -- 172.21.15.7:0/2042636879 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe80acb30 con 0x7f7fec15ca30 2023-11-28T18:37:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 -- 172.21.15.7:0/2042636879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ca30 msgr2=0x7f7fec159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/2042636879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ca30 0x7f7fec159c90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0d0200 tx=0x7f7fe801c4d0 comp rx=0 tx=0).stop 2023-11-28T18:37:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 -- 172.21.15.7:0/2042636879 shutdown_connections 2023-11-28T18:37:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/2042636879 >> v2:172.21.15.193:3300/0 conn(0x7f7fec15ce10 0x7f7fec0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/2042636879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ca30 0x7f7fec159c90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 -- 172.21.15.7:0/2042636879 >> 172.21.15.7:0/2042636879 conn(0x7f7fec06c060 msgr2=0x7f7fec0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 -- 172.21.15.7:0/2042636879 shutdown_connections 2023-11-28T18:37:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 -- 172.21.15.7:0/2042636879 wait complete. 2023-11-28T18:37:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 Processor -- start 2023-11-28T18:37:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 -- start start 2023-11-28T18:37:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.935+0000 7f7ff19f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fec0cafe0 0x7f7fec1f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff19f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fec15ca30 0x7f7fec0c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff19f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ce10 0x7f7fec15d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff19f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0c35c0 con 0x7f7fec15ce10 2023-11-28T18:37:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff19f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec1572f0 con 0x7f7fec15ca30 2023-11-28T18:37:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff19f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec1f68f0 con 0x7f7fec0cafe0 2023-11-28T18:37:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff11f7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ce10 0x7f7fec15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7fe3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fec15ca30 0x7f7fec0c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff09f6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fec0cafe0 0x7f7fec1f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff11f7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ce10 0x7f7fec15d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51640/0 (socket says 172.21.15.7:51640) 2023-11-28T18:37:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff11f7640 1 -- 172.21.15.7:0/4003858612 learned_addr learned my addr 172.21.15.7:0/4003858612 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff09f6640 1 -- 172.21.15.7:0/4003858612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fec15ca30 msgr2=0x7f7fec0c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:10.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff09f6640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fec15ca30 0x7f7fec0c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:10.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff09f6640 1 -- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ce10 msgr2=0x7f7fec15d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:10.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff09f6640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ce10 0x7f7fec15d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:10.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff09f6640 1 -- 172.21.15.7:0/4003858612 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe80094d0 con 0x7f7fec0cafe0 2023-11-28T18:37:10.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff11f7640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ce10 0x7f7fec15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:37:10.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff09f6640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fec0cafe0 0x7f7fec1f5fb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0cd160 tx=0x7f7fe801ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7fe17fa640 1 -- 172.21.15.7:0/4003858612 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe80ac640 con 0x7f7fec0cafe0 2023-11-28T18:37:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7fe17fa640 1 -- 172.21.15.7:0/4003858612 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe80b5870 con 0x7f7fec0cafe0 2023-11-28T18:37:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fec1f6bd0 con 0x7f7fec0cafe0 2023-11-28T18:37:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7fe17fa640 1 -- 172.21.15.7:0/4003858612 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe80b5c20 con 0x7f7fec0cafe0 2023-11-28T18:37:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fec0c32e0 con 0x7f7fec0cafe0 2023-11-28T18:37:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.939+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fec04e890 con 0x7f7fec0cafe0 2023-11-28T18:37:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.943+0000 7f7fe17fa640 1 -- 172.21.15.7:0/4003858612 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7fe80b3070 con 0x7f7fec0cafe0 2023-11-28T18:37:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.943+0000 7f7fe17fa640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fd80668d0 0x7f7fd8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.943+0000 7f7fe17fa640 1 -- 172.21.15.7:0/4003858612 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f7fe811edf0 con 0x7f7fec0cafe0 2023-11-28T18:37:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.943+0000 7f7fe3fff640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fd80668d0 0x7f7fd8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:10.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.943+0000 7f7fe3fff640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fd80668d0 0x7f7fd8068d90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7fec1583f0 tx=0x7f7fdc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:10.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:10.947+0000 7f7fe17fa640 1 -- 172.21.15.7:0/4003858612 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7fe80ebea0 con 0x7f7fec0cafe0 2023-11-28T18:37:11.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.179+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fec0ca320 con 0x7f7fec0cafe0 2023-11-28T18:37:11.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.179+0000 7f7fe17fa640 1 -- 172.21.15.7:0/4003858612 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5762 (secure 0 0 0) 0x7f7fe80b1090 con 0x7f7fec0cafe0 2023-11-28T18:37:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:37:11.179 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:08.323828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:37:11.179 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 63 2023-11-28T18:37:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fd80668d0 msgr2=0x7f7fd8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fd80668d0 0x7f7fd8068d90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7fec1583f0 tx=0x7f7fdc008040 comp rx=0 tx=0).stop 2023-11-28T18:37:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fec0cafe0 msgr2=0x7f7fec1f5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fec0cafe0 0x7f7fec1f5fb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0cd160 tx=0x7f7fe801ad10 comp rx=0 tx=0).stop 2023-11-28T18:37:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 shutdown_connections 2023-11-28T18:37:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7fd80668d0 0x7f7fd8068d90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fec15ce10 0x7f7fec15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fec15ca30 0x7f7fec0c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 --2- 172.21.15.7:0/4003858612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7fec0cafe0 0x7f7fec1f5fb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 >> 172.21.15.7:0/4003858612 conn(0x7f7fec06c060 msgr2=0x7f7fec0cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 shutdown_connections 2023-11-28T18:37:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:11.183+0000 7f7ff19f8640 1 -- 172.21.15.7:0/4003858612 wait complete. 2023-11-28T18:37:11.203 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:37:13.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:37:13.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.367+0000 7fc8728fe640 1 Processor -- start 2023-11-28T18:37:13.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.367+0000 7fc8728fe640 1 -- start start 2023-11-28T18:37:13.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc8728fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c1540e0 0x7fc86c0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:13.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc8728fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc86c15a8a0 0x7fc86c15dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:13.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc8728fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc86c0cf6c0 0x7fc86c0cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:13.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc8728fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86c15d7e0 con 0x7fc86c0cf6c0 2023-11-28T18:37:13.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc8728fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86c0cab80 con 0x7fc86c15a8a0 2023-11-28T18:37:13.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc8728fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0cb850 con 0x7fc86c1540e0 2023-11-28T18:37:13.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc870e9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc86c0cf6c0 0x7fc86c0cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:13.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc870e9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc86c0cf6c0 0x7fc86c0cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51650/0 (socket says 172.21.15.7:51650) 2023-11-28T18:37:13.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc870e9b640 1 -- 172.21.15.7:0/1733639966 learned_addr learned my addr 172.21.15.7:0/1733639966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:13.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc86bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c1540e0 0x7fc86c0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:13.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc863fff640 1 --2- 172.21.15.7:0/1733639966 >> v2:172.21.15.193:3300/0 conn(0x7fc86c15a8a0 0x7fc86c15dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:13.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc870e9b640 1 -- 172.21.15.7:0/1733639966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c1540e0 msgr2=0x7fc86c0ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:13.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc870e9b640 1 --2- 172.21.15.7:0/1733639966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c1540e0 0x7fc86c0ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc870e9b640 1 -- 172.21.15.7:0/1733639966 >> v2:172.21.15.193:3300/0 conn(0x7fc86c15a8a0 msgr2=0x7fc86c15dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:13.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc870e9b640 1 --2- 172.21.15.7:0/1733639966 >> v2:172.21.15.193:3300/0 conn(0x7fc86c15a8a0 0x7fc86c15dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.371+0000 7fc870e9b640 1 -- 172.21.15.7:0/1733639966 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0cbb30 con 0x7fc86c0cf6c0 2023-11-28T18:37:13.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc870e9b640 1 --2- 172.21.15.7:0/1733639966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c0cf6c0 0x7fc86c0cfb50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc86c06b6d0 tx=0x7fc86401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede2a1a4bc3dad32 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:13.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc86b7fe640 1 -- 172.21.15.7:0/1733639966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8640a5760 con 0x7fc86c0cf6c0 2023-11-28T18:37:13.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc86b7fe640 1 -- 172.21.15.7:0/1733639966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8640a58c0 con 0x7fc86c0cf6c0 2023-11-28T18:37:13.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc86b7fe640 1 -- 172.21.15.7:0/1733639966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8640a5c50 con 0x7fc86c0cf6c0 2023-11-28T18:37:13.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 -- 172.21.15.7:0/1733639966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c0cf6c0 msgr2=0x7fc86c0cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:13.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1733639966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c0cf6c0 0x7fc86c0cfb50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc86c06b6d0 tx=0x7fc86401c9d0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 -- 172.21.15.7:0/1733639966 shutdown_connections 2023-11-28T18:37:13.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1733639966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c0cf6c0 0x7fc86c0cfb50 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1733639966 >> v2:172.21.15.193:3300/0 conn(0x7fc86c15a8a0 0x7fc86c15dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1733639966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c1540e0 0x7fc86c0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 -- 172.21.15.7:0/1733639966 >> 172.21.15.7:0/1733639966 conn(0x7fc86c0662e0 msgr2=0x7fc86c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:13.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 -- 172.21.15.7:0/1733639966 shutdown_connections 2023-11-28T18:37:13.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 -- 172.21.15.7:0/1733639966 wait complete. 2023-11-28T18:37:13.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.375+0000 7fc8728fe640 1 Processor -- start 2023-11-28T18:37:13.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 -- start start 2023-11-28T18:37:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c0cf6c0 0x7fc86c156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc86c1540e0 0x7fc86c1f9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c15a8a0 0x7fc86c15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c1fa900 con 0x7fc86c15a8a0 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0c8fe0 con 0x7fc86c1540e0 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0cf060 con 0x7fc86c0cf6c0 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c15a8a0 0x7fc86c15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c15a8a0 0x7fc86c15bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51666/0 (socket says 172.21.15.7:51666) 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 -- 172.21.15.7:0/1124343479 learned_addr learned my addr 172.21.15.7:0/1124343479 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc86bfff640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c0cf6c0 0x7fc86c156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc863fff640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc86c1540e0 0x7fc86c1f9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 -- 172.21.15.7:0/1124343479 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c0cf6c0 msgr2=0x7fc86c156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c0cf6c0 0x7fc86c156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 -- 172.21.15.7:0/1124343479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc86c1540e0 msgr2=0x7fc86c1f9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc86c1540e0 0x7fc86c1f9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 -- 172.21.15.7:0/1124343479 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc864009b30 con 0x7fc86c15a8a0 2023-11-28T18:37:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc870e9b640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c15a8a0 0x7fc86c15bff0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc8640af040 tx=0x7fc864019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8697fa640 1 -- 172.21.15.7:0/1124343479 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8640b8670 con 0x7fc86c15a8a0 2023-11-28T18:37:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8697fa640 1 -- 172.21.15.7:0/1124343479 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8640b87d0 con 0x7fc86c15a8a0 2023-11-28T18:37:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c1fa340 con 0x7fc86c15a8a0 2023-11-28T18:37:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8697fa640 1 -- 172.21.15.7:0/1124343479 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8640b8b80 con 0x7fc86c15a8a0 2023-11-28T18:37:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c1f79a0 con 0x7fc86c15a8a0 2023-11-28T18:37:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.379+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c1f7800 con 0x7fc86c15a8a0 2023-11-28T18:37:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.383+0000 7fc8697fa640 1 -- 172.21.15.7:0/1124343479 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc8640b8ce0 con 0x7fc86c15a8a0 2023-11-28T18:37:13.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.383+0000 7fc8697fa640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc84c066900 0x7fc84c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:13.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.383+0000 7fc8697fa640 1 -- 172.21.15.7:0/1124343479 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fc86411f450 con 0x7fc86c15a8a0 2023-11-28T18:37:13.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.383+0000 7fc86bfff640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc84c066900 0x7fc84c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:13.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.383+0000 7fc86bfff640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc84c066900 0x7fc84c068dc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009ae0 tx=0x7fc85c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:13.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.387+0000 7fc8697fa640 1 -- 172.21.15.7:0/1124343479 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc86c1f7800 con 0x7fc86c15a8a0 2023-11-28T18:37:13.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.603+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc86c1f7800 con 0x7fc86c15a8a0 2023-11-28T18:37:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.603+0000 7fc8697fa640 1 -- 172.21.15.7:0/1124343479 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5762 (secure 0 0 0) 0x7fc86c1f7800 con 0x7fc86c15a8a0 2023-11-28T18:37:13.603 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:37:13.603 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:37:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 64 2023-11-28T18:37:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc84c066900 msgr2=0x7fc84c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc84c066900 0x7fc84c068dc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009ae0 tx=0x7fc85c009190 comp rx=0 tx=0).stop 2023-11-28T18:37:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c15a8a0 msgr2=0x7fc86c15bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c15a8a0 0x7fc86c15bff0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc8640af040 tx=0x7fc864019770 comp rx=0 tx=0).stop 2023-11-28T18:37:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 shutdown_connections 2023-11-28T18:37:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc84c066900 0x7fc84c068dc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86c15a8a0 0x7fc86c15bff0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc86c1540e0 0x7fc86c1f9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 --2- 172.21.15.7:0/1124343479 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc86c0cf6c0 0x7fc86c156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 >> 172.21.15.7:0/1124343479 conn(0x7fc86c0662e0 msgr2=0x7fc86c0666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 shutdown_connections 2023-11-28T18:37:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:13.607+0000 7fc8728fe640 1 -- 172.21.15.7:0/1124343479 wait complete. 2023-11-28T18:37:13.631 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=4, replacing gid=4978 2023-11-28T18:37:13.632 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2023-11-28T18:37:16.633 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-11-28T18:37:16.633 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T18:37:16.633 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T18:37:16.638 INFO:tasks.ceph.mds.b:Started 2023-11-28T18:37:16.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:37:16.728 INFO:tasks.ceph.mds.b.smithi193.stdout:starting mds.b at 2023-11-28T18:37:16.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe35ac9a640 1 Processor -- start 2023-11-28T18:37:16.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe35ac9a640 1 -- start start 2023-11-28T18:37:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe35ac9a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe3541540e0 0x7fe3540ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe35ac9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe35415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe35ac9a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe3540cf6c0 0x7fe3540cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe35ac9a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35415d7e0 con 0x7fe3541540e0 2023-11-28T18:37:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe35ac9a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3540cab80 con 0x7fe3540cf6c0 2023-11-28T18:37:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe35ac9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3540cb850 con 0x7fe35415a8a0 2023-11-28T18:37:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe358a36640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe3541540e0 0x7fe3540ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe353fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe35415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe358a36640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe3541540e0 0x7fe3540ca4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55872/0 (socket says 172.21.15.7:55872) 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe359237640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe3540cf6c0 0x7fe3540cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe358a36640 1 -- 172.21.15.7:0/2123326335 learned_addr learned my addr 172.21.15.7:0/2123326335 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe353fff640 1 -- 172.21.15.7:0/2123326335 >> v2:172.21.15.193:3300/0 conn(0x7fe3540cf6c0 msgr2=0x7fe3540cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe353fff640 1 --2- 172.21.15.7:0/2123326335 >> v2:172.21.15.193:3300/0 conn(0x7fe3540cf6c0 0x7fe3540cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe353fff640 1 -- 172.21.15.7:0/2123326335 >> v2:172.21.15.7:3300/0 conn(0x7fe3541540e0 msgr2=0x7fe3540ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe353fff640 1 --2- 172.21.15.7:0/2123326335 >> v2:172.21.15.7:3300/0 conn(0x7fe3541540e0 0x7fe3540ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.814+0000 7fe353fff640 1 -- 172.21.15.7:0/2123326335 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540cbb30 con 0x7fe35415a8a0 2023-11-28T18:37:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe353fff640 1 --2- 172.21.15.7:0/2123326335 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe35415dc70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3540b5a00 tx=0x7fe34001c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d5502e05b0d0c51 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe3537fe640 1 -- 172.21.15.7:0/2123326335 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400ab600 con 0x7fe35415a8a0 2023-11-28T18:37:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe3537fe640 1 -- 172.21.15.7:0/2123326335 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3400a7050 con 0x7fe35415a8a0 2023-11-28T18:37:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe3537fe640 1 -- 172.21.15.7:0/2123326335 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400ab980 con 0x7fe35415a8a0 2023-11-28T18:37:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 -- 172.21.15.7:0/2123326335 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 msgr2=0x7fe35415dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/2123326335 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe35415dc70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3540b5a00 tx=0x7fe34001c460 comp rx=0 tx=0).stop 2023-11-28T18:37:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 -- 172.21.15.7:0/2123326335 shutdown_connections 2023-11-28T18:37:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/2123326335 >> v2:172.21.15.193:3300/0 conn(0x7fe3540cf6c0 0x7fe3540cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/2123326335 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe35415dc70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/2123326335 >> v2:172.21.15.7:3300/0 conn(0x7fe3541540e0 0x7fe3540ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 -- 172.21.15.7:0/2123326335 >> 172.21.15.7:0/2123326335 conn(0x7fe3540662e0 msgr2=0x7fe3540666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 -- 172.21.15.7:0/2123326335 shutdown_connections 2023-11-28T18:37:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.818+0000 7fe35ac9a640 1 -- 172.21.15.7:0/2123326335 wait complete. 2023-11-28T18:37:16.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe35ac9a640 1 Processor -- start 2023-11-28T18:37:16.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe35ac9a640 1 -- start start 2023-11-28T18:37:16.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe35ac9a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3540cf6c0 0x7fe354158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:16.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe35ac9a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3541540e0 0x7fe3541f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:16.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe35ac9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe3540ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:16.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe35ac9a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541599f0 con 0x7fe3541540e0 2023-11-28T18:37:16.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe35ac9a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35415cb90 con 0x7fe3540cf6c0 2023-11-28T18:37:16.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe35ac9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35415b6c0 con 0x7fe35415a8a0 2023-11-28T18:37:16.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe353fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3541540e0 0x7fe3541f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:16.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe353fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3541540e0 0x7fe3541f2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55882/0 (socket says 172.21.15.7:55882) 2023-11-28T18:37:16.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe353fff640 1 -- 172.21.15.7:0/4185965588 learned_addr learned my addr 172.21.15.7:0/4185965588 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:16.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe358a36640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3540cf6c0 0x7fe354158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:16.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.822+0000 7fe359237640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe3540ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:16.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe353fff640 1 -- 172.21.15.7:0/4185965588 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 msgr2=0x7fe3540ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:16.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe353fff640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe3540ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:16.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe353fff640 1 -- 172.21.15.7:0/4185965588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3540cf6c0 msgr2=0x7fe354158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:16.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe353fff640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3540cf6c0 0x7fe354158f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:16.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe353fff640 1 -- 172.21.15.7:0/4185965588 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe340009420 con 0x7fe3541540e0 2023-11-28T18:37:16.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe353fff640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3541540e0 0x7fe3541f2870 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe3400ae040 tx=0x7fe340019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:16.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe3517fa640 1 -- 172.21.15.7:0/4185965588 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400b4410 con 0x7fe3541540e0 2023-11-28T18:37:16.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe3517fa640 1 -- 172.21.15.7:0/4185965588 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3400b4590 con 0x7fe3541540e0 2023-11-28T18:37:16.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35415bce0 con 0x7fe3541540e0 2023-11-28T18:37:16.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe3517fa640 1 -- 172.21.15.7:0/4185965588 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400b4940 con 0x7fe3541540e0 2023-11-28T18:37:16.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35415c540 con 0x7fe3541540e0 2023-11-28T18:37:16.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35415e170 con 0x7fe3541540e0 2023-11-28T18:37:16.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe3517fa640 1 -- 172.21.15.7:0/4185965588 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe3400b2040 con 0x7fe3541540e0 2023-11-28T18:37:16.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.826+0000 7fe3517fa640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe338066900 0x7fe338068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:16.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.830+0000 7fe3517fa640 1 -- 172.21.15.7:0/4185965588 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fe34011e430 con 0x7fe3541540e0 2023-11-28T18:37:16.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.830+0000 7fe358a36640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe338066900 0x7fe338068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:16.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.830+0000 7fe358a36640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe338066900 0x7fe338068dc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe348009b30 tx=0x7fe348009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:16.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:16.834+0000 7fe3517fa640 1 -- 172.21.15.7:0/4185965588 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe3400eb440 con 0x7fe3541540e0 2023-11-28T18:37:17.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.122+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe35404e890 con 0x7fe3541540e0 2023-11-28T18:37:17.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.122+0000 7fe3517fa640 1 -- 172.21.15.7:0/4185965588 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5762 (secure 0 0 0) 0x7fe3400b0090 con 0x7fe3541540e0 2023-11-28T18:37:17.121 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:37:17.121 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:37:17.122 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 64 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.122+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe338066900 msgr2=0x7fe338068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.122+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe338066900 0x7fe338068dc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe348009b30 tx=0x7fe348009240 comp rx=0 tx=0).stop 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.122+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3541540e0 msgr2=0x7fe3541f2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.122+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3541540e0 0x7fe3541f2870 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe3400ae040 tx=0x7fe340019200 comp rx=0 tx=0).stop 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.126+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 shutdown_connections 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.126+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe338066900 0x7fe338068dc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.126+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35415a8a0 0x7fe3540ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.126+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3541540e0 0x7fe3541f2870 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.126+0000 7fe35ac9a640 1 --2- 172.21.15.7:0/4185965588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3540cf6c0 0x7fe354158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.126+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 >> 172.21.15.7:0/4185965588 conn(0x7fe3540662e0 msgr2=0x7fe3540c5830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:17.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.126+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 shutdown_connections 2023-11-28T18:37:17.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:17.126+0000 7fe35ac9a640 1 -- 172.21.15.7:0/4185965588 wait complete. 2023-11-28T18:37:17.145 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-11-28T18:37:19.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:37:19.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98d10fe640 1 Processor -- start 2023-11-28T18:37:19.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98d10fe640 1 -- start start 2023-11-28T18:37:19.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98d10fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc151cd0 0x7f98cc15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:19.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98d10fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f98cc06e560 0x7f98cc157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98d10fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f98cc0701f0 0x7f98cc0c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98d10fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc071e60 con 0x7f98cc06e560 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98d10fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc070930 con 0x7f98cc0701f0 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98d10fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc070660 con 0x7f98cc151cd0 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98c3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f98cc06e560 0x7f98cc157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98c3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f98cc06e560 0x7f98cc157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55884/0 (socket says 172.21.15.7:55884) 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98c3fff640 1 -- 172.21.15.7:0/2399628006 learned_addr learned my addr 172.21.15.7:0/2399628006 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98cb59e640 1 --2- 172.21.15.7:0/2399628006 >> v2:172.21.15.193:3300/0 conn(0x7f98cc0701f0 0x7f98cc0c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98cad9d640 1 --2- 172.21.15.7:0/2399628006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc151cd0 0x7f98cc15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98c3fff640 1 -- 172.21.15.7:0/2399628006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc151cd0 msgr2=0x7f98cc15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98c3fff640 1 --2- 172.21.15.7:0/2399628006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc151cd0 0x7f98cc15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98c3fff640 1 -- 172.21.15.7:0/2399628006 >> v2:172.21.15.193:3300/0 conn(0x7f98cc0701f0 msgr2=0x7f98cc0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:37:19.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98c3fff640 1 --2- 172.21.15.7:0/2399628006 >> v2:172.21.15.193:3300/0 conn(0x7f98cc0701f0 0x7f98cc0c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.318+0000 7f98c3fff640 1 -- 172.21.15.7:0/2399628006 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc070bf0 con 0x7f98cc06e560 2023-11-28T18:37:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98c3fff640 1 --2- 172.21.15.7:0/2399628006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc157b90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0c95f0 tx=0x7f98b401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9396696da16fe515 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:19.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98ca59c640 1 -- 172.21.15.7:0/2399628006 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b40ab600 con 0x7f98cc06e560 2023-11-28T18:37:19.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98ca59c640 1 -- 172.21.15.7:0/2399628006 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b40a7050 con 0x7f98cc06e560 2023-11-28T18:37:19.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98ca59c640 1 -- 172.21.15.7:0/2399628006 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b40ab980 con 0x7f98cc06e560 2023-11-28T18:37:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98d10fe640 1 -- 172.21.15.7:0/2399628006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 msgr2=0x7f98cc157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98d10fe640 1 --2- 172.21.15.7:0/2399628006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc157b90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0c95f0 tx=0x7f98b401c470 comp rx=0 tx=0).stop 2023-11-28T18:37:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98d10fe640 1 -- 172.21.15.7:0/2399628006 shutdown_connections 2023-11-28T18:37:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98d10fe640 1 --2- 172.21.15.7:0/2399628006 >> v2:172.21.15.193:3300/0 conn(0x7f98cc0701f0 0x7f98cc0c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98d10fe640 1 --2- 172.21.15.7:0/2399628006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc157b90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98d10fe640 1 --2- 172.21.15.7:0/2399628006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc151cd0 0x7f98cc15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.322+0000 7f98d10fe640 1 -- 172.21.15.7:0/2399628006 >> 172.21.15.7:0/2399628006 conn(0x7f98cc0662e0 msgr2=0x7f98cc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:19.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 -- 172.21.15.7:0/2399628006 shutdown_connections 2023-11-28T18:37:19.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 -- 172.21.15.7:0/2399628006 wait complete. 2023-11-28T18:37:19.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 Processor -- start 2023-11-28T18:37:19.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 -- start start 2023-11-28T18:37:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc0701f0 0x7f98cc0c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98cc151cd0 0x7f98cc0c38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1f6fa0 con 0x7f98cc06e560 2023-11-28T18:37:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc154450 con 0x7f98cc151cd0 2023-11-28T18:37:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98d10fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc154730 con 0x7f98cc0701f0 2023-11-28T18:37:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.326+0000 7f98cad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:19.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55896/0 (socket says 172.21.15.7:55896) 2023-11-28T18:37:19.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cad9d640 1 -- 172.21.15.7:0/617992746 learned_addr learned my addr 172.21.15.7:0/617992746 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:37:19.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cb59e640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98cc151cd0 0x7f98cc0c38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:19.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98c3fff640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc0701f0 0x7f98cc0c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cad9d640 1 -- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc0701f0 msgr2=0x7f98cc0c33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cad9d640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc0701f0 0x7f98cc0c33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cad9d640 1 -- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98cc151cd0 msgr2=0x7f98cc0c38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cad9d640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98cc151cd0 0x7f98cc0c38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cad9d640 1 -- 172.21.15.7:0/617992746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40094d0 con 0x7f98cc06e560 2023-11-28T18:37:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98c3fff640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc0701f0 0x7f98cc0c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:37:19.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cb59e640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98cc151cd0 0x7f98cc0c38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:37:19.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98cad9d640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc152dc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00e870 tx=0x7f98bc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:19.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98c37fe640 1 -- 172.21.15.7:0/617992746 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc009510 con 0x7f98cc06e560 2023-11-28T18:37:19.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98c37fe640 1 -- 172.21.15.7:0/617992746 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc00eec0 con 0x7f98cc06e560 2023-11-28T18:37:19.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98c37fe640 1 -- 172.21.15.7:0/617992746 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00cb40 con 0x7f98cc06e560 2023-11-28T18:37:19.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc154a10 con 0x7f98cc06e560 2023-11-28T18:37:19.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc154fd0 con 0x7f98cc06e560 2023-11-28T18:37:19.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc04e890 con 0x7f98cc06e560 2023-11-28T18:37:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.330+0000 7f98c37fe640 1 -- 172.21.15.7:0/617992746 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f98bc00cca0 con 0x7f98cc06e560 2023-11-28T18:37:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.334+0000 7f98c37fe640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f98ac0668d0 0x7f98ac068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:37:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.334+0000 7f98c37fe640 1 -- 172.21.15.7:0/617992746 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f98bc084750 con 0x7f98cc06e560 2023-11-28T18:37:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.334+0000 7f98c3fff640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f98ac0668d0 0x7f98ac068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:37:19.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.334+0000 7f98c3fff640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f98ac0668d0 0x7f98ac068d90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f98b40ad000 tx=0x7f98b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:37:19.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.338+0000 7f98c37fe640 1 -- 172.21.15.7:0/617992746 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f98bc051800 con 0x7f98cc06e560 2023-11-28T18:37:19.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.566+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98cc0c3190 con 0x7f98cc06e560 2023-11-28T18:37:19.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.566+0000 7f98c37fe640 1 -- 172.21.15.7:0/617992746 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f98bc054e20 con 0x7f98cc06e560 2023-11-28T18:37:19.567 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:37:19.567 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:37:19.567 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 65 2023-11-28T18:37:19.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f98ac0668d0 msgr2=0x7f98ac068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:19.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f98ac0668d0 0x7f98ac068d90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f98b40ad000 tx=0x7f98b4008040 comp rx=0 tx=0).stop 2023-11-28T18:37:19.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 msgr2=0x7f98cc152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:37:19.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc152dc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00e870 tx=0x7f98bc00ec70 comp rx=0 tx=0).stop 2023-11-28T18:37:19.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 shutdown_connections 2023-11-28T18:37:19.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f98ac0668d0 0x7f98ac068d90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98cc151cd0 0x7f98cc0c38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98cc0701f0 0x7f98cc0c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 --2- 172.21.15.7:0/617992746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98cc06e560 0x7f98cc152dc0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:37:19.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 >> 172.21.15.7:0/617992746 conn(0x7f98cc0662e0 msgr2=0x7f98cc1614a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:37:19.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 shutdown_connections 2023-11-28T18:37:19.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:37:19.570+0000 7f98d10fe640 1 -- 172.21.15.7:0/617992746 wait complete. 2023-11-28T18:37:19.593 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-11-28T18:37:19.593 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 63 secs before thrashing 2023-11-28T18:37:31.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:37:31.593 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:38:02.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:38:02.618 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:38:22.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:22.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.749+0000 7f52b3409640 1 Processor -- start 2023-11-28T18:38:22.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.749+0000 7f52b3409640 1 -- start start 2023-11-28T18:38:22.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.749+0000 7f52b3409640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac15ce10 0x7f52ac0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:22.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.749+0000 7f52b3409640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f52ac0cafe0 0x7f52ac0ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:22.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b3409640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac15a3a0 con 0x7f52ac15ca30 2023-11-28T18:38:22.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b3409640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ac15a620 con 0x7f52ac0cafe0 2023-11-28T18:38:22.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b3409640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac15a900 con 0x7f52ac15ce10 2023-11-28T18:38:22.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b2407640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f52ac15ca30 0x7f52ac159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52552/0 (socket says 172.21.15.7:52552) 2023-11-28T18:38:22.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b2407640 1 -- 172.21.15.7:0/149092958 learned_addr learned my addr 172.21.15.7:0/149092958 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:38:22.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b2c08640 1 --2- 172.21.15.7:0/149092958 >> v2:172.21.15.7:3300/0 conn(0x7f52ac0cafe0 0x7f52ac0ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:22.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b1c06640 1 --2- 172.21.15.7:0/149092958 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac15ce10 0x7f52ac0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:22.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b2c08640 1 -- 172.21.15.7:0/149092958 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac15ce10 msgr2=0x7f52ac0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:22.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b2c08640 1 --2- 172.21.15.7:0/149092958 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac15ce10 0x7f52ac0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:22.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b2c08640 1 -- 172.21.15.7:0/149092958 >> v1:172.21.15.7:6789/0 conn(0x7f52ac15ca30 legacy=0x7f52ac159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:22.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b2c08640 1 -- 172.21.15.7:0/149092958 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac15abe0 con 0x7f52ac0cafe0 2023-11-28T18:38:22.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b1405640 1 -- 172.21.15.7:0/149092958 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519799720 0 0) 0x7f52ac15a3a0 con 0x7f52ac15ca30 2023-11-28T18:38:22.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.753+0000 7f52b1c06640 1 --2- 172.21.15.7:0/149092958 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac15ce10 0x7f52ac0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:38:22.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b2c08640 1 --2- 172.21.15.7:0/149092958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac0cafe0 0x7f52ac0ca710 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009d90 tx=0x7f52a401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814d3664e32976b9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:22.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b1405640 1 -- 172.21.15.7:0/149092958 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40af600 con 0x7f52ac0cafe0 2023-11-28T18:38:22.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b1405640 1 -- 172.21.15.7:0/149092958 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a40ab050 con 0x7f52ac0cafe0 2023-11-28T18:38:22.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b1405640 1 -- 172.21.15.7:0/149092958 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40af980 con 0x7f52ac0cafe0 2023-11-28T18:38:22.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 -- 172.21.15.7:0/149092958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac0cafe0 msgr2=0x7f52ac0ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:22.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 --2- 172.21.15.7:0/149092958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac0cafe0 0x7f52ac0ca710 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009d90 tx=0x7f52a401f130 comp rx=0 tx=0).stop 2023-11-28T18:38:22.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 -- 172.21.15.7:0/149092958 shutdown_connections 2023-11-28T18:38:22.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 --2- 172.21.15.7:0/149092958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac0cafe0 0x7f52ac0ca710 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:22.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 --2- 172.21.15.7:0/149092958 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac15ce10 0x7f52ac0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:22.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 -- 172.21.15.7:0/149092958 >> 172.21.15.7:0/149092958 conn(0x7f52ac06c060 msgr2=0x7f52ac0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:22.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 -- 172.21.15.7:0/149092958 shutdown_connections 2023-11-28T18:38:22.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 -- 172.21.15.7:0/149092958 wait complete. 2023-11-28T18:38:22.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 Processor -- start 2023-11-28T18:38:22.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 -- start start 2023-11-28T18:38:22.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac0cafe0 0x7f52ac14e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.757+0000 7f52b3409640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52ac15ca30 0x7f52ac151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b3409640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac15ce10 0x7f52ac152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b3409640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac1516e0 con 0x7f52ac15ce10 2023-11-28T18:38:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b3409640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac154e50 con 0x7f52ac15ca30 2023-11-28T18:38:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b3409640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac154bb0 con 0x7f52ac0cafe0 2023-11-28T18:38:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac15ce10 0x7f52ac152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac15ce10 0x7f52ac152040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36072/0 (socket says 172.21.15.7:36072) 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 -- 172.21.15.7:0/3840933311 learned_addr learned my addr 172.21.15.7:0/3840933311 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2407640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac0cafe0 0x7f52ac14e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b1c06640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52ac15ca30 0x7f52ac151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 -- 172.21.15.7:0/3840933311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac0cafe0 msgr2=0x7f52ac14e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac0cafe0 0x7f52ac14e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 -- 172.21.15.7:0/3840933311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52ac15ca30 msgr2=0x7f52ac151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52ac15ca30 0x7f52ac151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 -- 172.21.15.7:0/3840933311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a4009a70 con 0x7f52ac15ce10 2023-11-28T18:38:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b2c08640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac15ce10 0x7f52ac152040 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f52a40b2040 tx=0x7f52a401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52a2ffd640 1 -- 172.21.15.7:0/3840933311 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40b9610 con 0x7f52ac15ce10 2023-11-28T18:38:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52a2ffd640 1 -- 172.21.15.7:0/3840933311 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a40b9770 con 0x7f52ac15ce10 2023-11-28T18:38:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52a2ffd640 1 -- 172.21.15.7:0/3840933311 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40b9b20 con 0x7f52ac15ce10 2023-11-28T18:38:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ac1f3380 con 0x7f52ac15ce10 2023-11-28T18:38:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ac151300 con 0x7f52ac15ce10 2023-11-28T18:38:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.761+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ac0b4060 con 0x7f52ac15ce10 2023-11-28T18:38:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.765+0000 7f52a2ffd640 1 -- 172.21.15.7:0/3840933311 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f52a40b6050 con 0x7f52ac15ce10 2023-11-28T18:38:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.765+0000 7f52a2ffd640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5290066920 0x7f5290068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.765+0000 7f52a2ffd640 1 -- 172.21.15.7:0/3840933311 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f52a4122970 con 0x7f52ac15ce10 2023-11-28T18:38:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.765+0000 7f52b2407640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5290066920 0x7f5290068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.765+0000 7f52b2407640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5290066920 0x7f5290068de0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f52a8007fd0 tx=0x7f52a8007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:22.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.769+0000 7f52a2ffd640 1 -- 172.21.15.7:0/3840933311 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f52a40efb90 con 0x7f52ac15ce10 2023-11-28T18:38:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.993+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52ac151820 con 0x7f52ac15ce10 2023-11-28T18:38:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52a2ffd640 1 -- 172.21.15.7:0/3840933311 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f52ac151820 con 0x7f52ac15ce10 2023-11-28T18:38:22.996 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:22.996 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:22.997 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 65 2023-11-28T18:38:22.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5290066920 msgr2=0x7f5290068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:22.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5290066920 0x7f5290068de0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f52a8007fd0 tx=0x7f52a8007d60 comp rx=0 tx=0).stop 2023-11-28T18:38:22.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac15ce10 msgr2=0x7f52ac152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:22.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac15ce10 0x7f52ac152040 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f52a40b2040 tx=0x7f52a401bed0 comp rx=0 tx=0).stop 2023-11-28T18:38:22.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 shutdown_connections 2023-11-28T18:38:22.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5290066920 0x7f5290068de0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac15ce10 0x7f52ac152040 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52ac15ca30 0x7f52ac151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 --2- 172.21.15.7:0/3840933311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52ac0cafe0 0x7f52ac14e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 >> 172.21.15.7:0/3840933311 conn(0x7f52ac06c060 msgr2=0x7f52ac0c9200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:23.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 shutdown_connections 2023-11-28T18:38:23.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:22.997+0000 7f52b3409640 1 -- 172.21.15.7:0/3840933311 wait complete. 2023-11-28T18:38:23.017 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=3) 2023-11-28T18:38:23.018 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-11-28T18:38:23.018 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:38:23.034 INFO:tasks.ceph.mds.f:Stopped 2023-11-28T18:38:23.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:23.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235cf2a640 1 Processor -- start 2023-11-28T18:38:23.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235cf2a640 1 -- start start 2023-11-28T18:38:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235cf2a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23581540e0 0x7f23580ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235cf2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235cf2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235815e380 con 0x7f235815a8a0 2023-11-28T18:38:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235cf2a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f235815d7e0 con 0x7f23581540e0 2023-11-28T18:38:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235cf2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23580cab80 con 0x7f23580cf6c0 2023-11-28T18:38:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23581540e0 0x7f23580ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23581540e0 0x7f23580ca4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36086/0 (socket says 172.21.15.7:36086) 2023-11-28T18:38:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235659c640 1 -- 172.21.15.7:0/727377883 learned_addr learned my addr 172.21.15.7:0/727377883 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f2356d9d640 1 --2- 172.21.15.7:0/727377883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235659c640 1 -- 172.21.15.7:0/727377883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 msgr2=0x7f23580cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235659c640 1 --2- 172.21.15.7:0/727377883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235659c640 1 -- 172.21.15.7:0/727377883 >> v1:172.21.15.193:6789/0 conn(0x7f235815a8a0 legacy=0x7f235815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f235659c640 1 -- 172.21.15.7:0/727377883 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580cb850 con 0x7f23581540e0 2023-11-28T18:38:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.189+0000 7f2356d9d640 1 --2- 172.21.15.7:0/727377883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:38:23.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235659c640 1 --2- 172.21.15.7:0/727377883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23581540e0 0x7f23580ca4d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2348009870 tx=0x7f234801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28e4ca0ae5bb05c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:23.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235559a640 1 -- 172.21.15.7:0/727377883 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23480a7070 con 0x7f23581540e0 2023-11-28T18:38:23.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235559a640 1 -- 172.21.15.7:0/727377883 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23480ab430 con 0x7f23581540e0 2023-11-28T18:38:23.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235559a640 1 -- 172.21.15.7:0/727377883 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23480ab7c0 con 0x7f23581540e0 2023-11-28T18:38:23.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235cf2a640 1 -- 172.21.15.7:0/727377883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23581540e0 msgr2=0x7f23580ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:23.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235cf2a640 1 --2- 172.21.15.7:0/727377883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23581540e0 0x7f23580ca4d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2348009870 tx=0x7f234801c540 comp rx=0 tx=0).stop 2023-11-28T18:38:23.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235cf2a640 1 -- 172.21.15.7:0/727377883 shutdown_connections 2023-11-28T18:38:23.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235cf2a640 1 --2- 172.21.15.7:0/727377883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235cf2a640 1 --2- 172.21.15.7:0/727377883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23581540e0 0x7f23580ca4d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.193+0000 7f235cf2a640 1 -- 172.21.15.7:0/727377883 >> 172.21.15.7:0/727377883 conn(0x7f23580662e0 msgr2=0x7f23580666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:23.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 -- 172.21.15.7:0/727377883 shutdown_connections 2023-11-28T18:38:23.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 -- 172.21.15.7:0/727377883 wait complete. 2023-11-28T18:38:23.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 Processor -- start 2023-11-28T18:38:23.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 -- start start 2023-11-28T18:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23581540e0 0x7f23580b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f235815a8a0 0x7f23581f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580b38e0 con 0x7f235815a8a0 2023-11-28T18:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580b3ea0 con 0x7f23581540e0 2023-11-28T18:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235cf2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235815c9c0 con 0x7f23580cf6c0 2023-11-28T18:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f2356d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f235815a8a0 0x7f23581f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f235659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f2356d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f235815a8a0 0x7f23581f6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36088/0 (socket says 172.21.15.7:36088) 2023-11-28T18:38:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f2356d9d640 1 -- 172.21.15.7:0/232195065 learned_addr learned my addr 172.21.15.7:0/232195065 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.197+0000 7f2355d9b640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23581540e0 0x7f23580b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2356d9d640 1 -- 172.21.15.7:0/232195065 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 msgr2=0x7f23580b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2356d9d640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2356d9d640 1 -- 172.21.15.7:0/232195065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23581540e0 msgr2=0x7f23580b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2356d9d640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23581540e0 0x7f23580b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2356d9d640 1 -- 172.21.15.7:0/232195065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23480094d0 con 0x7f235815a8a0 2023-11-28T18:38:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2356d9d640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f235815a8a0 0x7f23581f6100 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f235000cf20 tx=0x7f235000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2346ffd640 1 -- 172.21.15.7:0/232195065 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000d840 con 0x7f235815a8a0 2023-11-28T18:38:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2346ffd640 1 -- 172.21.15.7:0/232195065 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235000d9a0 con 0x7f235815a8a0 2023-11-28T18:38:23.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2346ffd640 1 -- 172.21.15.7:0/232195065 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000dd50 con 0x7f235815a8a0 2023-11-28T18:38:23.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235815cca0 con 0x7f235815a8a0 2023-11-28T18:38:23.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23580b3b30 con 0x7f235815a8a0 2023-11-28T18:38:23.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23580b4050 con 0x7f235815a8a0 2023-11-28T18:38:23.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.201+0000 7f2346ffd640 1 -- 172.21.15.7:0/232195065 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f235001e3b0 con 0x7f235815a8a0 2023-11-28T18:38:23.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.205+0000 7f2346ffd640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23380668d0 0x7f2338068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:23.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.205+0000 7f2346ffd640 1 -- 172.21.15.7:0/232195065 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f2350085090 con 0x7f235815a8a0 2023-11-28T18:38:23.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.205+0000 7f235659c640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23380668d0 0x7f2338068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:23.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.205+0000 7f235659c640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23380668d0 0x7f2338068d90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f23480af000 tx=0x7f2348008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:23.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.209+0000 7f2346ffd640 1 -- 172.21.15.7:0/232195065 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f23580b4050 con 0x7f235815a8a0 2023-11-28T18:38:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23580b4050 con 0x7f235815a8a0 2023-11-28T18:38:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f2346ffd640 1 -- 172.21.15.7:0/232195065 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f23580b4050 con 0x7f235815a8a0 2023-11-28T18:38:23.442 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:23.442 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 65 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23380668d0 msgr2=0x7f2338068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23380668d0 0x7f2338068d90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f23480af000 tx=0x7f2348008040 comp rx=0 tx=0).stop 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f235815a8a0 msgr2=0x7f23581f6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f235815a8a0 0x7f23581f6100 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f235000cf20 tx=0x7f235000a7d0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 shutdown_connections 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f23380668d0 0x7f2338068d90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f235815a8a0 0x7f23581f6100 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23581540e0 0x7f23580b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 --2- 172.21.15.7:0/232195065 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f23580cf6c0 0x7f23580b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.441+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 >> 172.21.15.7:0/232195065 conn(0x7f23580662e0 msgr2=0x7f23580666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.445+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 shutdown_connections 2023-11-28T18:38:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:23.445+0000 7f235cf2a640 1 -- 172.21.15.7:0/232195065 wait complete. 2023-11-28T18:38:23.465 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:38:25.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:25.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.629+0000 7f94d5417640 1 Processor -- start 2023-11-28T18:38:25.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.629+0000 7f94d5417640 1 -- start start 2023-11-28T18:38:25.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d5417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c4c80 0x7f94d0070820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:25.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d5417640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94d01515d0 0x7f94d0151a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:25.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d5417640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d0158a90 con 0x7f94d00c5060 2023-11-28T18:38:25.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d5417640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d0151fa0 con 0x7f94d01515d0 2023-11-28T18:38:25.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d5417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d0157ba0 con 0x7f94d00c4c80 2023-11-28T18:38:25.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d4c16640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94d01515d0 0x7f94d0151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:25.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d4c16640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94d01515d0 0x7f94d0151a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48760/0 (socket says 172.21.15.7:48760) 2023-11-28T18:38:25.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d4c16640 1 -- 172.21.15.7:0/4179876292 learned_addr learned my addr 172.21.15.7:0/4179876292 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:25.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94cffff640 1 --2- 172.21.15.7:0/4179876292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c4c80 0x7f94d0070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:25.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d4c16640 1 -- 172.21.15.7:0/4179876292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c4c80 msgr2=0x7f94d0070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:25.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d4c16640 1 --2- 172.21.15.7:0/4179876292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c4c80 0x7f94d0070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d4c16640 1 -- 172.21.15.7:0/4179876292 >> v1:172.21.15.193:6789/0 conn(0x7f94d00c5060 legacy=0x7f94d006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:25.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94d4c16640 1 -- 172.21.15.7:0/4179876292 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d0157e80 con 0x7f94d01515d0 2023-11-28T18:38:25.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94cffff640 1 --2- 172.21.15.7:0/4179876292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c4c80 0x7f94d0070820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:38:25.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.633+0000 7f94ceffd640 1 -- 172.21.15.7:0/4179876292 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952961195 0 0) 0x7f94c4001120 con 0x7f94d00c5060 2023-11-28T18:38:25.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d4c16640 1 --2- 172.21.15.7:0/4179876292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d01515d0 0x7f94d0151a60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009e50 tx=0x7f94c801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8596093564171ce3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:25.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94ceffd640 1 -- 172.21.15.7:0/4179876292 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c80af400 con 0x7f94d01515d0 2023-11-28T18:38:25.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94ceffd640 1 -- 172.21.15.7:0/4179876292 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c80ab050 con 0x7f94d01515d0 2023-11-28T18:38:25.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94ceffd640 1 -- 172.21.15.7:0/4179876292 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c80af780 con 0x7f94d01515d0 2023-11-28T18:38:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d5417640 1 -- 172.21.15.7:0/4179876292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d01515d0 msgr2=0x7f94d0151a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d5417640 1 --2- 172.21.15.7:0/4179876292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d01515d0 0x7f94d0151a60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009e50 tx=0x7f94c801f1f0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d5417640 1 -- 172.21.15.7:0/4179876292 shutdown_connections 2023-11-28T18:38:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d5417640 1 --2- 172.21.15.7:0/4179876292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d01515d0 0x7f94d0151a60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d5417640 1 --2- 172.21.15.7:0/4179876292 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c4c80 0x7f94d0070820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d5417640 1 -- 172.21.15.7:0/4179876292 >> 172.21.15.7:0/4179876292 conn(0x7f94d006c370 msgr2=0x7f94d00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d5417640 1 -- 172.21.15.7:0/4179876292 shutdown_connections 2023-11-28T18:38:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.637+0000 7f94d5417640 1 -- 172.21.15.7:0/4179876292 wait complete. 2023-11-28T18:38:25.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 Processor -- start 2023-11-28T18:38:25.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 -- start start 2023-11-28T18:38:25.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00c4c80 0x7f94d0149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:25.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c5060 0x7f94d01457e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d01515d0 0x7f94d0145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d014e760 con 0x7f94d00c4c80 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d014d560 con 0x7f94d01515d0 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d014d840 con 0x7f94d00c5060 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00c4c80 0x7f94d0149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00c4c80 0x7f94d0149f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48772/0 (socket says 172.21.15.7:48772) 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 -- 172.21.15.7:0/86720219 learned_addr learned my addr 172.21.15.7:0/86720219 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d4c16640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d01515d0 0x7f94d0145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cf7fe640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c5060 0x7f94d01457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:25.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 -- 172.21.15.7:0/86720219 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c5060 msgr2=0x7f94d01457e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:25.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c5060 0x7f94d01457e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 -- 172.21.15.7:0/86720219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d01515d0 msgr2=0x7f94d0145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d01515d0 0x7f94d0145d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 -- 172.21.15.7:0/86720219 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c8009b30 con 0x7f94d00c4c80 2023-11-28T18:38:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94cffff640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00c4c80 0x7f94d0149f70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f94c000e810 tx=0x7f94c000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94ccff9640 1 -- 172.21.15.7:0/86720219 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c000c8d0 con 0x7f94d00c4c80 2023-11-28T18:38:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94ccff9640 1 -- 172.21.15.7:0/86720219 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c000ca30 con 0x7f94d00c4c80 2023-11-28T18:38:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94ccff9640 1 -- 172.21.15.7:0/86720219 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c000cdc0 con 0x7f94d00c4c80 2023-11-28T18:38:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.641+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d014ddc0 con 0x7f94d00c4c80 2023-11-28T18:38:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.645+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d014e380 con 0x7f94d00c4c80 2023-11-28T18:38:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.645+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d004e890 con 0x7f94d00c4c80 2023-11-28T18:38:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.645+0000 7f94ccff9640 1 -- 172.21.15.7:0/86720219 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f94c0013410 con 0x7f94d00c4c80 2023-11-28T18:38:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.645+0000 7f94ccff9640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f94bc0668d0 0x7f94bc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:25.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.649+0000 7f94ccff9640 1 -- 172.21.15.7:0/86720219 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f94c0084570 con 0x7f94d00c4c80 2023-11-28T18:38:25.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.649+0000 7f94cf7fe640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f94bc0668d0 0x7f94bc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:25.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.649+0000 7f94cf7fe640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f94bc0668d0 0x7f94bc068d90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f94d01507c0 tx=0x7f94c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:25.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.653+0000 7f94ccff9640 1 -- 172.21.15.7:0/86720219 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f94c00514f0 con 0x7f94d00c4c80 2023-11-28T18:38:25.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.889+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94d014aa80 con 0x7f94d00c4c80 2023-11-28T18:38:25.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.889+0000 7f94ccff9640 1 -- 172.21.15.7:0/86720219 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f94c0054b10 con 0x7f94d00c4c80 2023-11-28T18:38:25.890 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:25.891 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:25.891 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 65 2023-11-28T18:38:25.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f94bc0668d0 msgr2=0x7f94bc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f94bc0668d0 0x7f94bc068d90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f94d01507c0 tx=0x7f94c4009040 comp rx=0 tx=0).stop 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00c4c80 msgr2=0x7f94d0149f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00c4c80 0x7f94d0149f70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f94c000e810 tx=0x7f94c000ecd0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 shutdown_connections 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f94bc0668d0 0x7f94bc068d90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d01515d0 0x7f94d0145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f94d00c5060 0x7f94d01457e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 --2- 172.21.15.7:0/86720219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00c4c80 0x7f94d0149f70 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 >> 172.21.15.7:0/86720219 conn(0x7f94d006c370 msgr2=0x7f94d014b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 shutdown_connections 2023-11-28T18:38:25.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:25.893+0000 7f94d5417640 1 -- 172.21.15.7:0/86720219 wait complete. 2023-11-28T18:38:25.913 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:38:27.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:28.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.089+0000 7f9e98af3640 1 Processor -- start 2023-11-28T18:38:28.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.089+0000 7f9e98af3640 1 -- start start 2023-11-28T18:38:28.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.089+0000 7f9e98af3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e8c0d9220 0x7f9e8c0df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:28.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.089+0000 7f9e98af3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c038d10 0x7f9e8c050160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:28.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.089+0000 7f9e98af3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8c038170 con 0x7f9e8c0379d0 2023-11-28T18:38:28.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.089+0000 7f9e98af3640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8c0dfef0 con 0x7f9e8c0d9220 2023-11-28T18:38:28.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.089+0000 7f9e98af3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c037310 con 0x7f9e8c038d10 2023-11-28T18:38:28.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.089+0000 7f9e937fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e8c0d9220 0x7f9e8c0df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:28.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e937fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e8c0d9220 0x7f9e8c0df9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48788/0 (socket says 172.21.15.7:48788) 2023-11-28T18:38:28.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e937fe640 1 -- 172.21.15.7:0/3449903608 learned_addr learned my addr 172.21.15.7:0/3449903608 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:28.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e93fff640 1 --2- 172.21.15.7:0/3449903608 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c038d10 0x7f9e8c050160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:28.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e92ffd640 1 --1- 172.21.15.7:0/3449903608 >> v1:172.21.15.193:6789/0 conn(0x7f9e8c0379d0 0x7f9e8c038600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:42138/0 (socket says 172.21.15.7:42138) 2023-11-28T18:38:28.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e937fe640 1 -- 172.21.15.7:0/3449903608 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c038d10 msgr2=0x7f9e8c050160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:28.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e937fe640 1 --2- 172.21.15.7:0/3449903608 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c038d10 0x7f9e8c050160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e937fe640 1 -- 172.21.15.7:0/3449903608 >> v1:172.21.15.193:6789/0 conn(0x7f9e8c0379d0 legacy=0x7f9e8c038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:28.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e937fe640 1 -- 172.21.15.7:0/3449903608 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c0375b0 con 0x7f9e8c0d9220 2023-11-28T18:38:28.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e937fe640 1 --2- 172.21.15.7:0/3449903608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0d9220 0x7f9e8c0df9b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009860 tx=0x7f9e8801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc3cc50d8293fcb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:28.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e927fc640 1 -- 172.21.15.7:0/3449903608 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880ab600 con 0x7f9e8c0d9220 2023-11-28T18:38:28.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e927fc640 1 -- 172.21.15.7:0/3449903608 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e880a7050 con 0x7f9e8c0d9220 2023-11-28T18:38:28.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.093+0000 7f9e927fc640 1 -- 172.21.15.7:0/3449903608 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880ab980 con 0x7f9e8c0d9220 2023-11-28T18:38:28.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 -- 172.21.15.7:0/3449903608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0d9220 msgr2=0x7f9e8c0df9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:28.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 --2- 172.21.15.7:0/3449903608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0d9220 0x7f9e8c0df9b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009860 tx=0x7f9e8801c3d0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 -- 172.21.15.7:0/3449903608 shutdown_connections 2023-11-28T18:38:28.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 --2- 172.21.15.7:0/3449903608 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c038d10 0x7f9e8c050160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 --2- 172.21.15.7:0/3449903608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0d9220 0x7f9e8c0df9b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 -- 172.21.15.7:0/3449903608 >> 172.21.15.7:0/3449903608 conn(0x7f9e8c0545f0 msgr2=0x7f9e8c054a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:28.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 -- 172.21.15.7:0/3449903608 shutdown_connections 2023-11-28T18:38:28.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 -- 172.21.15.7:0/3449903608 wait complete. 2023-11-28T18:38:28.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 Processor -- start 2023-11-28T18:38:28.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 -- start start 2023-11-28T18:38:28.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0379d0 0x7f9e8c0dd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:28.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e8c038d10 0x7f9e8c17ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:28.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.097+0000 7f9e98af3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c0d9220 0x7f9e8c0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:28.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e98af3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c0ddb70 con 0x7f9e8c0379d0 2023-11-28T18:38:28.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e98af3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c0d3aa0 con 0x7f9e8c038d10 2023-11-28T18:38:28.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e98af3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c04a140 con 0x7f9e8c0d9220 2023-11-28T18:38:28.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0379d0 0x7f9e8c0dd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:28.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0379d0 0x7f9e8c0dd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48790/0 (socket says 172.21.15.7:48790) 2023-11-28T18:38:28.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 -- 172.21.15.7:0/4242161444 learned_addr learned my addr 172.21.15.7:0/4242161444 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:28.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e92ffd640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e8c038d10 0x7f9e8c17ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e93fff640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c0d9220 0x7f9e8c0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 -- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c0d9220 msgr2=0x7f9e8c0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c0d9220 0x7f9e8c0ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 -- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e8c038d10 msgr2=0x7f9e8c17ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e8c038d10 0x7f9e8c17ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 -- 172.21.15.7:0/4242161444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e88009470 con 0x7f9e8c0379d0 2023-11-28T18:38:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e93fff640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c0d9220 0x7f9e8c0ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:38:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e937fe640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0379d0 0x7f9e8c0dd630 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9e880ae040 tx=0x7f9e88019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e7ffff640 1 -- 172.21.15.7:0/4242161444 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880b4640 con 0x7f9e8c0379d0 2023-11-28T18:38:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e7ffff640 1 -- 172.21.15.7:0/4242161444 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e880b47a0 con 0x7f9e8c0379d0 2023-11-28T18:38:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e7ffff640 1 -- 172.21.15.7:0/4242161444 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880b4b50 con 0x7f9e8c0379d0 2023-11-28T18:38:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8c0d34e0 con 0x7f9e8c0379d0 2023-11-28T18:38:28.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8c0ddf50 con 0x7f9e8c0379d0 2023-11-28T18:38:28.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8c0052a0 con 0x7f9e8c0379d0 2023-11-28T18:38:28.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.101+0000 7f9e7ffff640 1 -- 172.21.15.7:0/4242161444 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9e880b2070 con 0x7f9e8c0379d0 2023-11-28T18:38:28.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.105+0000 7f9e7ffff640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9e780668d0 0x7f9e78068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:28.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.105+0000 7f9e7ffff640 1 -- 172.21.15.7:0/4242161444 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f9e8811e650 con 0x7f9e8c0379d0 2023-11-28T18:38:28.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.105+0000 7f9e92ffd640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9e780668d0 0x7f9e78068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:28.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.105+0000 7f9e92ffd640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9e780668d0 0x7f9e78068d90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0cd970 tx=0x7f9e80007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:28.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.109+0000 7f9e7ffff640 1 -- 172.21.15.7:0/4242161444 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9e880eb6e0 con 0x7f9e8c0379d0 2023-11-28T18:38:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.321+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e8c04ff40 con 0x7f9e8c0379d0 2023-11-28T18:38:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e7ffff640 1 -- 172.21.15.7:0/4242161444 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f9e880eed00 con 0x7f9e8c0379d0 2023-11-28T18:38:28.324 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:28.324 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 65 2023-11-28T18:38:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9e780668d0 msgr2=0x7f9e78068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9e780668d0 0x7f9e78068d90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0cd970 tx=0x7f9e80007c90 comp rx=0 tx=0).stop 2023-11-28T18:38:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0379d0 msgr2=0x7f9e8c0dd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0379d0 0x7f9e8c0dd630 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9e880ae040 tx=0x7f9e88019170 comp rx=0 tx=0).stop 2023-11-28T18:38:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 shutdown_connections 2023-11-28T18:38:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9e780668d0 0x7f9e78068d90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9e8c0d9220 0x7f9e8c0ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e8c038d10 0x7f9e8c17ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 --2- 172.21.15.7:0/4242161444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e8c0379d0 0x7f9e8c0dd630 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 >> 172.21.15.7:0/4242161444 conn(0x7f9e8c0545f0 msgr2=0x7f9e8c052540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 shutdown_connections 2023-11-28T18:38:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:28.325+0000 7f9e98af3640 1 -- 172.21.15.7:0/4242161444 wait complete. 2023-11-28T18:38:28.346 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:38:30.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.505+0000 7f4bfedc4640 1 Processor -- start 2023-11-28T18:38:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfedc4640 1 -- start start 2023-11-28T18:38:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfedc4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf80cbb70 0x7f4bf80cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfedc4640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4bf80cc810 0x7f4bf8072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfedc4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf80ca920 con 0x7f4bf81542e0 2023-11-28T18:38:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfedc4640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf8071d50 con 0x7f4bf80cc810 2023-11-28T18:38:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfedc4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf8072030 con 0x7f4bf80cbb70 2023-11-28T18:38:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfcb60640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4bf81542e0 0x7f4bf815e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40370/0 (socket says 172.21.15.7:40370) 2023-11-28T18:38:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfcb60640 1 -- 172.21.15.7:0/3507062075 learned_addr learned my addr 172.21.15.7:0/3507062075 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:38:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf7fff640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf80cbb70 0x7f4bf80cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfd361640 1 --2- 172.21.15.7:0/3507062075 >> v2:172.21.15.193:3300/0 conn(0x7f4bf80cc810 0x7f4bf8072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295208651 0 0) 0x7f4bf80ca920 con 0x7f4bf81542e0 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be8003530 con 0x7f4bf81542e0 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3208423646 0 0) 0x7f4be8003530 con 0x7f4bf81542e0 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf80cbb70 msgr2=0x7f4bf80cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf80cbb70 0x7f4bf80cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 >> v2:172.21.15.193:3300/0 conn(0x7f4bf80cc810 msgr2=0x7f4bf8072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 --2- 172.21.15.7:0/3507062075 >> v2:172.21.15.193:3300/0 conn(0x7f4bf80cc810 0x7f4bf8072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf80c4cd0 con 0x7f4bf81542e0 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f4bec001680 con 0x7f4bf81542e0 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 >> v1:172.21.15.7:6789/0 conn(0x7f4bf81542e0 legacy=0x7f4bf815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf80cbb70 0x7f4be8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8003f50 0x7f4be80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4be8004d10 0x7f4be8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf8071d50 con 0x7f4be8004d10 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf8072030 con 0x7f4be8003f50 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be800b990 con 0x7f4bf80cbb70 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfd361640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4be8004d10 0x7f4be8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfcb60640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf80cbb70 0x7f4be8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bf7fff640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8003f50 0x7f4be80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfd361640 1 -- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf80cbb70 msgr2=0x7f4be8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfd361640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf80cbb70 0x7f4be8003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfd361640 1 -- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8003f50 msgr2=0x7f4be80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfd361640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8003f50 0x7f4be80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.509+0000 7f4bfd361640 1 -- 172.21.15.7:0/3507062075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be8003530 con 0x7f4be8004d10 2023-11-28T18:38:30.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bfd361640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4be8004d10 0x7f4be8005190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000e170 tx=0x7f4bf001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6315f31cdfc1899e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:30.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf00a5c90 con 0x7f4be8004d10 2023-11-28T18:38:30.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf000f040 con 0x7f4be8004d10 2023-11-28T18:38:30.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bf77fe640 1 -- 172.21.15.7:0/3507062075 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf00ad440 con 0x7f4be8004d10 2023-11-28T18:38:30.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bfedc4640 1 -- 172.21.15.7:0/3507062075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4be8004d10 msgr2=0x7f4be8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:30.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bfedc4640 1 --2- 172.21.15.7:0/3507062075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4be8004d10 0x7f4be8005190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000e170 tx=0x7f4bf001cc80 comp rx=0 tx=0).stop 2023-11-28T18:38:30.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bfd361640 1 -- 172.21.15.7:0/3507062075 reap_dead start 2023-11-28T18:38:30.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bfedc4640 1 -- 172.21.15.7:0/3507062075 shutdown_connections 2023-11-28T18:38:30.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bfedc4640 1 -- 172.21.15.7:0/3507062075 >> 172.21.15.7:0/3507062075 conn(0x7f4bf80717e0 msgr2=0x7f4bf80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:30.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bfedc4640 1 -- 172.21.15.7:0/3507062075 shutdown_connections 2023-11-28T18:38:30.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.513+0000 7f4bfedc4640 1 -- 172.21.15.7:0/3507062075 wait complete. 2023-11-28T18:38:30.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfedc4640 1 Processor -- start 2023-11-28T18:38:30.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfedc4640 1 -- start start 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfedc4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8004d10 0x7f4bf81f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfedc4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bf81549e0 0x7f4bf80c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfedc4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf8147420 0x7f4bf81478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfedc4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf8155b50 con 0x7f4bf81549e0 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfedc4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf8147df0 con 0x7f4be8004d10 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfedc4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf8156430 con 0x7f4bf8147420 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bf81549e0 0x7f4bf80c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bf81549e0 0x7f4bf80c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48806/0 (socket says 172.21.15.7:48806) 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bf7fff640 1 -- 172.21.15.7:0/2727948933 learned_addr learned my addr 172.21.15.7:0/2727948933 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfcb60640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8004d10 0x7f4bf81f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bfd361640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf8147420 0x7f4bf81478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bf7fff640 1 -- 172.21.15.7:0/2727948933 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf8147420 msgr2=0x7f4bf81478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bf7fff640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf8147420 0x7f4bf81478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bf7fff640 1 -- 172.21.15.7:0/2727948933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8004d10 msgr2=0x7f4bf81f3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bf7fff640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8004d10 0x7f4bf81f3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.517+0000 7f4bf7fff640 1 -- 172.21.15.7:0/2727948933 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf81f38b0 con 0x7f4bf81549e0 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bf7fff640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bf81549e0 0x7f4bf80c70b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4be4011990 tx=0x7f4be4011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bf57fa640 1 -- 172.21.15.7:0/2727948933 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be400fa70 con 0x7f4bf81549e0 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bf57fa640 1 -- 172.21.15.7:0/2727948933 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be400fbd0 con 0x7f4bf81549e0 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf80c7ab0 con 0x7f4bf81549e0 2023-11-28T18:38:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bf57fa640 1 -- 172.21.15.7:0/2727948933 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be4014440 con 0x7f4bf81549e0 2023-11-28T18:38:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf8156840 con 0x7f4bf81549e0 2023-11-28T18:38:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf806c670 con 0x7f4bf81549e0 2023-11-28T18:38:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bf57fa640 1 -- 172.21.15.7:0/2727948933 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4be40145a0 con 0x7f4bf81549e0 2023-11-28T18:38:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bf57fa640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bdc066900 0x7f4bdc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.521+0000 7f4bf57fa640 1 -- 172.21.15.7:0/2727948933 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f4be4084ce0 con 0x7f4bf81549e0 2023-11-28T18:38:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.525+0000 7f4bfcb60640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bdc066900 0x7f4bdc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.525+0000 7f4bfcb60640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bdc066900 0x7f4bdc068dc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0017b0 tx=0x7f4bec001de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.529+0000 7f4bf57fa640 1 -- 172.21.15.7:0/2727948933 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4be4051bd0 con 0x7f4bf81549e0 2023-11-28T18:38:30.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.765+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bf81f38b0 con 0x7f4bf81549e0 2023-11-28T18:38:30.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.765+0000 7f4bf57fa640 1 -- 172.21.15.7:0/2727948933 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f4bf81f38b0 con 0x7f4bf81549e0 2023-11-28T18:38:30.767 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:30.767 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:30.768 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 65 2023-11-28T18:38:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bdc066900 msgr2=0x7f4bdc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bdc066900 0x7f4bdc068dc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0017b0 tx=0x7f4bec001de0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bf81549e0 msgr2=0x7f4bf80c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bf81549e0 0x7f4bf80c70b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4be4011990 tx=0x7f4be4011e50 comp rx=0 tx=0).stop 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 shutdown_connections 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bdc066900 0x7f4bdc068dc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bf8147420 0x7f4bf81478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bf81549e0 0x7f4bf80c70b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 --2- 172.21.15.7:0/2727948933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8004d10 0x7f4bf81f3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 >> 172.21.15.7:0/2727948933 conn(0x7f4bf80717e0 msgr2=0x7f4bf80b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 shutdown_connections 2023-11-28T18:38:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:30.769+0000 7f4bfedc4640 1 -- 172.21.15.7:0/2727948933 wait complete. 2023-11-28T18:38:30.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:38:32.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:38:32.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:38:32.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:32.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe11966b640 1 Processor -- start 2023-11-28T18:38:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe11966b640 1 -- start start 2023-11-28T18:38:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe11966b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe114154080 0x7fe11415a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe11966b640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe11415ab30 0x7fe1140cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe11966b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11406a5f0 con 0x7fe1140b3900 2023-11-28T18:38:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe11966b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11406d8a0 con 0x7fe114154080 2023-11-28T18:38:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe11966b640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe114159e80 con 0x7fe11415ab30 2023-11-28T18:38:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe112ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe114154080 0x7fe11415a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe112ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe114154080 0x7fe11415a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48808/0 (socket says 172.21.15.7:48808) 2023-11-28T18:38:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe112ffd640 1 -- 172.21.15.7:0/3673929086 learned_addr learned my addr 172.21.15.7:0/3673929086 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe1127fc640 1 --2- 172.21.15.7:0/3673929086 >> v2:172.21.15.193:3300/0 conn(0x7fe11415ab30 0x7fe1140cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe112ffd640 1 -- 172.21.15.7:0/3673929086 >> v2:172.21.15.193:3300/0 conn(0x7fe11415ab30 msgr2=0x7fe1140cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe112ffd640 1 --2- 172.21.15.7:0/3673929086 >> v2:172.21.15.193:3300/0 conn(0x7fe11415ab30 0x7fe1140cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe112ffd640 1 -- 172.21.15.7:0/3673929086 >> v1:172.21.15.193:6789/0 conn(0x7fe1140b3900 legacy=0x7fe11415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.937+0000 7fe112ffd640 1 -- 172.21.15.7:0/3673929086 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11415a160 con 0x7fe114154080 2023-11-28T18:38:32.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe112ffd640 1 --2- 172.21.15.7:0/3673929086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe11415a5f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe11406ddc0 tx=0x7fe10401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef57903cfe22b046 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:32.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe111ffb640 1 -- 172.21.15.7:0/3673929086 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1040a89b0 con 0x7fe114154080 2023-11-28T18:38:32.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe111ffb640 1 -- 172.21.15.7:0/3673929086 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1040a8b10 con 0x7fe114154080 2023-11-28T18:38:32.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe111ffb640 1 -- 172.21.15.7:0/3673929086 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1040a8ea0 con 0x7fe114154080 2023-11-28T18:38:32.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe11966b640 1 -- 172.21.15.7:0/3673929086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 msgr2=0x7fe11415a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:32.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe11966b640 1 --2- 172.21.15.7:0/3673929086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe11415a5f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe11406ddc0 tx=0x7fe10401ecb0 comp rx=0 tx=0).stop 2023-11-28T18:38:32.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe11966b640 1 -- 172.21.15.7:0/3673929086 shutdown_connections 2023-11-28T18:38:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe11966b640 1 --2- 172.21.15.7:0/3673929086 >> v2:172.21.15.193:3300/0 conn(0x7fe11415ab30 0x7fe1140cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe11966b640 1 --2- 172.21.15.7:0/3673929086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe11415a5f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe11966b640 1 -- 172.21.15.7:0/3673929086 >> 172.21.15.7:0/3673929086 conn(0x7fe11408d3f0 msgr2=0x7fe1140662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe11966b640 1 -- 172.21.15.7:0/3673929086 shutdown_connections 2023-11-28T18:38:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.941+0000 7fe11966b640 1 -- 172.21.15.7:0/3673929086 wait complete. 2023-11-28T18:38:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe11966b640 1 Processor -- start 2023-11-28T18:38:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe11966b640 1 -- start start 2023-11-28T18:38:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe11966b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1140b3900 0x7fe1141e3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe11966b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe1141e2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe11966b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe11415ab30 0x7fe1141eff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe11966b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1141f3490 con 0x7fe114154080 2023-11-28T18:38:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe11966b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1141f3d30 con 0x7fe11415ab30 2023-11-28T18:38:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe11966b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe114150830 con 0x7fe1140b3900 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe1141e2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe1141e2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48824/0 (socket says 172.21.15.7:48824) 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 -- 172.21.15.7:0/3659078847 learned_addr learned my addr 172.21.15.7:0/3659078847 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 -- 172.21.15.7:0/3659078847 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1140b3900 msgr2=0x7fe1141e3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1140b3900 0x7fe1141e3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 -- 172.21.15.7:0/3659078847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe11415ab30 msgr2=0x7fe1141eff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe113fff640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe11415ab30 0x7fe1141eff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe11415ab30 0x7fe1141eff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 -- 172.21.15.7:0/3659078847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1040094d0 con 0x7fe114154080 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe113fff640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe11415ab30 0x7fe1141eff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:38:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe1127fc640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe1141e2670 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe10800d620 tx=0x7fe10800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:32.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.945+0000 7fe0fb7fe640 1 -- 172.21.15.7:0/3659078847 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10800b620 con 0x7fe114154080 2023-11-28T18:38:32.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe0fb7fe640 1 -- 172.21.15.7:0/3659078847 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10800b780 con 0x7fe114154080 2023-11-28T18:38:32.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe0fb7fe640 1 -- 172.21.15.7:0/3659078847 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10800bb10 con 0x7fe114154080 2023-11-28T18:38:32.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1141583c0 con 0x7fe114154080 2023-11-28T18:38:32.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1141f36e0 con 0x7fe114154080 2023-11-28T18:38:32.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1141e4270 con 0x7fe114154080 2023-11-28T18:38:32.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe0fb7fe640 1 -- 172.21.15.7:0/3659078847 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe10800bc70 con 0x7fe114154080 2023-11-28T18:38:32.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe0fb7fe640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1000668d0 0x7fe100068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:32.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe0fb7fe640 1 -- 172.21.15.7:0/3659078847 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fe108084bf0 con 0x7fe114154080 2023-11-28T18:38:32.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.949+0000 7fe112ffd640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1000668d0 0x7fe100068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:32.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.953+0000 7fe112ffd640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1000668d0 0x7fe100068d90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe1040b2000 tx=0x7fe104004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:32.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:32.957+0000 7fe0fb7fe640 1 -- 172.21.15.7:0/3659078847 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe108051b50 con 0x7fe114154080 2023-11-28T18:38:33.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.149+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe11404e890 con 0x7fe114154080 2023-11-28T18:38:33.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe0fb7fe640 1 -- 172.21.15.7:0/3659078847 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fe108055170 con 0x7fe114154080 2023-11-28T18:38:33.153 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:33.153 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:33.156 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 65 2023-11-28T18:38:33.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1000668d0 msgr2=0x7fe100068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:33.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1000668d0 0x7fe100068d90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe1040b2000 tx=0x7fe104004740 comp rx=0 tx=0).stop 2023-11-28T18:38:33.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 msgr2=0x7fe1141e2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:33.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe1141e2670 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe10800d620 tx=0x7fe10800dae0 comp rx=0 tx=0).stop 2023-11-28T18:38:33.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 shutdown_connections 2023-11-28T18:38:33.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe1000668d0 0x7fe100068d90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:33.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe11415ab30 0x7fe1141eff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:33.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe114154080 0x7fe1141e2670 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:33.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 --2- 172.21.15.7:0/3659078847 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe1140b3900 0x7fe1141e3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:33.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 >> 172.21.15.7:0/3659078847 conn(0x7fe11408d3f0 msgr2=0x7fe1140c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:33.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 shutdown_connections 2023-11-28T18:38:33.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:33.153+0000 7fe11966b640 1 -- 172.21.15.7:0/3659078847 wait complete. 2023-11-28T18:38:33.178 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:38:35.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f623d9b8640 1 Processor -- start 2023-11-28T18:38:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f623d9b8640 1 -- start start 2023-11-28T18:38:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f623d9b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62381542e0 0x7f623815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f623d9b8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f62380cbb70 0x7f62380cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f623d9b8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62380ca920 con 0x7f62380cc810 2023-11-28T18:38:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f623d9b8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6238071d50 con 0x7f62380cbb70 2023-11-28T18:38:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f623d9b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6238072030 con 0x7f62381542e0 2023-11-28T18:38:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f6236ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62381542e0 0x7f623815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f6236ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62381542e0 0x7f623815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:33762/0 (socket says 172.21.15.7:33762) 2023-11-28T18:38:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f6236ffd640 1 -- 172.21.15.7:0/3433305355 learned_addr learned my addr 172.21.15.7:0/3433305355 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.337+0000 7f62367fc640 1 --2- 172.21.15.7:0/3433305355 >> v2:172.21.15.193:3300/0 conn(0x7f62380cbb70 0x7f62380cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f6236ffd640 1 -- 172.21.15.7:0/3433305355 >> v2:172.21.15.193:3300/0 conn(0x7f62380cbb70 msgr2=0x7f62380cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f6236ffd640 1 --2- 172.21.15.7:0/3433305355 >> v2:172.21.15.193:3300/0 conn(0x7f62380cbb70 0x7f62380cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f6236ffd640 1 -- 172.21.15.7:0/3433305355 >> v1:172.21.15.193:6789/0 conn(0x7f62380cc810 legacy=0x7f6238072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:35.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f6236ffd640 1 -- 172.21.15.7:0/3433305355 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623815e550 con 0x7f62381542e0 2023-11-28T18:38:35.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f62367fc640 1 --2- 172.21.15.7:0/3433305355 >> v2:172.21.15.193:3300/0 conn(0x7f62380cbb70 0x7f62380cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:38:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f6236ffd640 1 --2- 172.21.15.7:0/3433305355 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62381542e0 0x7f623815e010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6228009870 tx=0x7f622801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=463538dc877c93da server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f6235ffb640 1 -- 172.21.15.7:0/3433305355 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62280ab650 con 0x7f62381542e0 2023-11-28T18:38:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f6235ffb640 1 -- 172.21.15.7:0/3433305355 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62280ab7b0 con 0x7f62381542e0 2023-11-28T18:38:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.341+0000 7f6235ffb640 1 -- 172.21.15.7:0/3433305355 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62280abb40 con 0x7f62381542e0 2023-11-28T18:38:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- 172.21.15.7:0/3433305355 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62381542e0 msgr2=0x7f623815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 --2- 172.21.15.7:0/3433305355 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62381542e0 0x7f623815e010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6228009870 tx=0x7f622801c540 comp rx=0 tx=0).stop 2023-11-28T18:38:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- 172.21.15.7:0/3433305355 shutdown_connections 2023-11-28T18:38:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 --2- 172.21.15.7:0/3433305355 >> v2:172.21.15.193:3300/0 conn(0x7f62380cbb70 0x7f62380cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 --2- 172.21.15.7:0/3433305355 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62381542e0 0x7f623815e010 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- 172.21.15.7:0/3433305355 >> 172.21.15.7:0/3433305355 conn(0x7f62380717e0 msgr2=0x7f62380662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- 172.21.15.7:0/3433305355 shutdown_connections 2023-11-28T18:38:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- 172.21.15.7:0/3433305355 wait complete. 2023-11-28T18:38:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 Processor -- start 2023-11-28T18:38:35.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- start start 2023-11-28T18:38:35.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62380cbb70 0x7f6238149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:35.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62380cc810 0x7f6238149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:35.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381542e0 0x7f62380cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623814a450 con 0x7f62381542e0 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62380c6d60 con 0x7f62380cbb70 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f623d9b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62380c7040 con 0x7f62380cc810 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f62377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381542e0 0x7f62380cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f6236ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62380cbb70 0x7f6238149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f62377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381542e0 0x7f62380cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43372/0 (socket says 172.21.15.7:43372) 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f62377fe640 1 -- 172.21.15.7:0/2243450350 learned_addr learned my addr 172.21.15.7:0/2243450350 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f6236ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62380cbb70 0x7f6238149f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:43406/0 (socket says 172.21.15.7:43406) 2023-11-28T18:38:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.345+0000 7f62367fc640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62380cc810 0x7f6238149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f6236ffd640 1 -- 172.21.15.7:0/2243450350 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62380cc810 msgr2=0x7f6238149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f6236ffd640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62380cc810 0x7f6238149270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f6236ffd640 1 -- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381542e0 msgr2=0x7f62380cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f6236ffd640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381542e0 0x7f62380cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f6236ffd640 1 -- 172.21.15.7:0/2243450350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c009420 con 0x7f62380cbb70 2023-11-28T18:38:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f62377fe640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381542e0 0x7f62380cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:38:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f6236ffd640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62380cbb70 0x7f6238149f10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f62280ac000 tx=0x7f62280192e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f621ffff640 1 -- 172.21.15.7:0/2243450350 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62280b5810 con 0x7f62380cbb70 2023-11-28T18:38:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f621ffff640 1 -- 172.21.15.7:0/2243450350 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62280b59b0 con 0x7f62380cbb70 2023-11-28T18:38:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f621ffff640 1 -- 172.21.15.7:0/2243450350 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62280b5d60 con 0x7f62380cbb70 2023-11-28T18:38:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62280094d0 con 0x7f62380cbb70 2023-11-28T18:38:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62380c7600 con 0x7f62380cbb70 2023-11-28T18:38:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623806c670 con 0x7f62380cbb70 2023-11-28T18:38:35.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.349+0000 7f621ffff640 1 -- 172.21.15.7:0/2243450350 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f62280de440 con 0x7f62380cbb70 2023-11-28T18:38:35.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.353+0000 7f621ffff640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6224066900 0x7f6224068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:35.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.353+0000 7f621ffff640 1 -- 172.21.15.7:0/2243450350 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f622811ef50 con 0x7f62380cbb70 2023-11-28T18:38:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.353+0000 7f62367fc640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6224066900 0x7f6224068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.353+0000 7f62367fc640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6224066900 0x7f6224068dc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f62380d07a0 tx=0x7f622c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:35.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.357+0000 7f621ffff640 1 -- 172.21.15.7:0/2243450350 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f62280ebfe0 con 0x7f62380cbb70 2023-11-28T18:38:35.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.561+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62380cd8d0 con 0x7f62380cbb70 2023-11-28T18:38:35.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f621ffff640 1 -- 172.21.15.7:0/2243450350 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f62280ef600 con 0x7f62380cbb70 2023-11-28T18:38:35.564 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:35.564 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:37:13.355614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5086,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5086":{"gid":5086,"name":"f","rank":3,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6835/650087483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":650087483},{"type":"v1","addr":"172.21.15.193:6835","nonce":650087483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:35.564 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 65 2023-11-28T18:38:35.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6224066900 msgr2=0x7f6224068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:35.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6224066900 0x7f6224068dc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f62380d07a0 tx=0x7f622c009190 comp rx=0 tx=0).stop 2023-11-28T18:38:35.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62380cbb70 msgr2=0x7f6238149f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62380cbb70 0x7f6238149f10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f62280ac000 tx=0x7f62280192e0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 shutdown_connections 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6224066900 0x7f6224068dc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381542e0 0x7f62380cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62380cc810 0x7f6238149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 --2- 172.21.15.7:0/2243450350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62380cbb70 0x7f6238149f10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 >> 172.21.15.7:0/2243450350 conn(0x7f62380717e0 msgr2=0x7f6238144f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 shutdown_connections 2023-11-28T18:38:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:35.565+0000 7f623d9b8640 1 -- 172.21.15.7:0/2243450350 wait complete. 2023-11-28T18:38:35.586 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:38:37.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b204fe640 1 Processor -- start 2023-11-28T18:38:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b204fe640 1 -- start start 2023-11-28T18:38:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b204fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b18164410 0x7f8b18166bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b204fe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b180c75a0 con 0x7f8b1816ab90 2023-11-28T18:38:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b204fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1816a140 con 0x7f8b180c6660 2023-11-28T18:38:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b204fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1816afe0 con 0x7f8b18164410 2023-11-28T18:38:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b1e29a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b18164410 0x7f8b18166bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b1e29a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b18164410 0x7f8b18166bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43388/0 (socket says 172.21.15.7:43388) 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b1e29a640 1 -- 172.21.15.7:0/181105227 learned_addr learned my addr 172.21.15.7:0/181105227 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b1da99640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8b1816ab90 0x7f8b180c2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36492/0 (socket says 172.21.15.7:36492) 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.761+0000 7f8b1d298640 1 -- 172.21.15.7:0/181105227 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317291150 0 0) 0x7f8b180c75a0 con 0x7f8b1816ab90 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.981+0000 7f8b1d298640 1 -- 172.21.15.7:0/181105227 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b04003510 con 0x7f8b1816ab90 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.981+0000 7f8b1d298640 1 -- 172.21.15.7:0/181105227 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625707171 0 0) 0x7f8b1816a140 con 0x7f8b180c6660 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.981+0000 7f8b1d298640 1 -- 172.21.15.7:0/181105227 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b180c75a0 con 0x7f8b180c6660 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.981+0000 7f8b1e29a640 1 -- 172.21.15.7:0/181105227 >> v1:172.21.15.193:6789/0 conn(0x7f8b180c6660 legacy=0x7f8b18171470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.981+0000 7f8b1e29a640 1 -- 172.21.15.7:0/181105227 >> v1:172.21.15.7:6789/0 conn(0x7f8b1816ab90 legacy=0x7f8b180c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.981+0000 7f8b1d298640 1 -- 172.21.15.7:0/181105227 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533843381 0 0) 0x7f8b04003510 con 0x7f8b1816ab90 2023-11-28T18:38:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.981+0000 7f8b1e29a640 1 -- 172.21.15.7:0/181105227 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b180c81d0 con 0x7f8b18164410 2023-11-28T18:38:37.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.985+0000 7f8b1e29a640 1 --2- 172.21.15.7:0/181105227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b18164410 0x7f8b18166bd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8b14008120 tx=0x7f8b1401c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c290f58e6d3a24 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:37.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.985+0000 7f8b1d298640 1 -- 172.21.15.7:0/181105227 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08001680 con 0x7f8b18164410 2023-11-28T18:38:37.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.985+0000 7f8b1d298640 1 -- 172.21.15.7:0/181105227 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b14004030 con 0x7f8b18164410 2023-11-28T18:38:37.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.985+0000 7f8b1d298640 1 -- 172.21.15.7:0/181105227 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b140043a0 con 0x7f8b18164410 2023-11-28T18:38:37.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.985+0000 7f8b204fe640 1 -- 172.21.15.7:0/181105227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b18164410 msgr2=0x7f8b18166bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:37.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.985+0000 7f8b204fe640 1 --2- 172.21.15.7:0/181105227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b18164410 0x7f8b18166bd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8b14008120 tx=0x7f8b1401c700 comp rx=0 tx=0).stop 2023-11-28T18:38:37.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 -- 172.21.15.7:0/181105227 shutdown_connections 2023-11-28T18:38:37.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 --2- 172.21.15.7:0/181105227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b18164410 0x7f8b18166bd0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:37.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 -- 172.21.15.7:0/181105227 >> 172.21.15.7:0/181105227 conn(0x7f8b180662e0 msgr2=0x7f8b180666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:37.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 -- 172.21.15.7:0/181105227 shutdown_connections 2023-11-28T18:38:37.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 -- 172.21.15.7:0/181105227 wait complete. 2023-11-28T18:38:37.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 Processor -- start 2023-11-28T18:38:37.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 -- start start 2023-11-28T18:38:37.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b180c6660 0x7f8b1815fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:37.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b18164410 0x7f8b181650a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:37.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b1816ab90 0x7f8b18165cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:37.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b181ff6c0 con 0x7f8b180c6660 2023-11-28T18:38:37.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b181ff9a0 con 0x7f8b18164410 2023-11-28T18:38:37.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.989+0000 7f8b204fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b181ffc80 con 0x7f8b1816ab90 2023-11-28T18:38:37.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b180c6660 0x7f8b1815fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:37.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b180c6660 0x7f8b1815fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43390/0 (socket says 172.21.15.7:43390) 2023-11-28T18:38:37.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 -- 172.21.15.7:0/1809225797 learned_addr learned my addr 172.21.15.7:0/1809225797 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:37.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1da99640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b18164410 0x7f8b181650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:37.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1ea9b640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b1816ab90 0x7f8b18165cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 -- 172.21.15.7:0/1809225797 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b1816ab90 msgr2=0x7f8b18165cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b1816ab90 0x7f8b18165cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 -- 172.21.15.7:0/1809225797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b18164410 msgr2=0x7f8b181650a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b18164410 0x7f8b181650a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 -- 172.21.15.7:0/1809225797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1400a580 con 0x7f8b180c6660 2023-11-28T18:38:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b1e29a640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b180c6660 0x7f8b1815fe20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8b140af040 tx=0x7f8b140194a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b0effd640 1 -- 172.21.15.7:0/1809225797 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b140bd070 con 0x7f8b180c6660 2023-11-28T18:38:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b0effd640 1 -- 172.21.15.7:0/1809225797 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b14003910 con 0x7f8b180c6660 2023-11-28T18:38:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b0effd640 1 -- 172.21.15.7:0/1809225797 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14003a80 con 0x7f8b180c6660 2023-11-28T18:38:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b181fff60 con 0x7f8b180c6660 2023-11-28T18:38:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b18161030 con 0x7f8b180c6660 2023-11-28T18:38:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.993+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1804e890 con 0x7f8b180c6660 2023-11-28T18:38:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.997+0000 7f8b0effd640 1 -- 172.21.15.7:0/1809225797 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8b14003be0 con 0x7f8b180c6660 2023-11-28T18:38:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.997+0000 7f8b0effd640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8afc0668d0 0x7f8afc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.997+0000 7f8b0effd640 1 -- 172.21.15.7:0/1809225797 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f8b1411f590 con 0x7f8b180c6660 2023-11-28T18:38:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.997+0000 7f8b1da99640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8afc0668d0 0x7f8afc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:37.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:37.997+0000 7f8b1da99640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8afc0668d0 0x7f8afc068d90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8b08001b60 tx=0x7f8b08001af0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.005+0000 7f8b0effd640 1 -- 172.21.15.7:0/1809225797 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8b140ec650 con 0x7f8b180c6660 2023-11-28T18:38:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.285+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b180c6140 con 0x7f8b180c6660 2023-11-28T18:38:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.285+0000 7f8b0effd640 1 -- 172.21.15.7:0/1809225797 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5764 (secure 0 0 0) 0x7f8b1400b040 con 0x7f8b180c6660 2023-11-28T18:38:38.286 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:38.286 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:37.328517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 66 2023-11-28T18:38:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8afc0668d0 msgr2=0x7f8afc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8afc0668d0 0x7f8afc068d90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8b08001b60 tx=0x7f8b08001af0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b180c6660 msgr2=0x7f8b1815fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b180c6660 0x7f8b1815fe20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8b140af040 tx=0x7f8b140194a0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 shutdown_connections 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8afc0668d0 0x7f8afc068d90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8b1816ab90 0x7f8b18165cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b18164410 0x7f8b181650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 --2- 172.21.15.7:0/1809225797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b180c6660 0x7f8b1815fe20 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 >> 172.21.15.7:0/1809225797 conn(0x7f8b180662e0 msgr2=0x7f8b180c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 shutdown_connections 2023-11-28T18:38:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.289+0000 7f8b204fe640 1 -- 172.21.15.7:0/1809225797 wait complete. 2023-11-28T18:38:38.311 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-11-28T18:38:38.311 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:38:38.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:38.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.473+0000 7fcce025a640 1 Processor -- start 2023-11-28T18:38:38.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.473+0000 7fcce025a640 1 -- start start 2023-11-28T18:38:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fcce025a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fcce025a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccd80cbb70 0x7fccd80cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fcce025a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd80ca920 con 0x7fccd80cc810 2023-11-28T18:38:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fcce025a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd8071d50 con 0x7fccd80cbb70 2023-11-28T18:38:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fcce025a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8072030 con 0x7fccd81542e0 2023-11-28T18:38:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fccdd7f5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccd80cbb70 0x7fccd80cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fccde7f7640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fccd80cc810 0x7fccd8072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36504/0 (socket says 172.21.15.7:36504) 2023-11-28T18:38:38.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fccde7f7640 1 -- 172.21.15.7:0/3247912770 learned_addr learned my addr 172.21.15.7:0/3247912770 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:38:38.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fccddff6640 1 --2- 172.21.15.7:0/3247912770 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:38.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fccdd7f5640 1 -- 172.21.15.7:0/3247912770 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 msgr2=0x7fccd815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:38.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fccdd7f5640 1 --2- 172.21.15.7:0/3247912770 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd815e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fccdd7f5640 1 -- 172.21.15.7:0/3247912770 >> v1:172.21.15.7:6789/0 conn(0x7fccd80cc810 legacy=0x7fccd8072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:38.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.477+0000 7fccdd7f5640 1 -- 172.21.15.7:0/3247912770 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd815e550 con 0x7fccd80cbb70 2023-11-28T18:38:38.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fccdd7f5640 1 --2- 172.21.15.7:0/3247912770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cbb70 0x7fccd80cc2d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fccc80097c0 tx=0x7fccc801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6ba26ec94112c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:38.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fccdcff4640 1 -- 172.21.15.7:0/3247912770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80ab680 con 0x7fccd80cbb70 2023-11-28T18:38:38.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fccdcff4640 1 -- 172.21.15.7:0/3247912770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc80ab7e0 con 0x7fccd80cbb70 2023-11-28T18:38:38.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fccdcff4640 1 -- 172.21.15.7:0/3247912770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80abb70 con 0x7fccd80cbb70 2023-11-28T18:38:38.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fcce025a640 1 -- 172.21.15.7:0/3247912770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cbb70 msgr2=0x7fccd80cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fcce025a640 1 --2- 172.21.15.7:0/3247912770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cbb70 0x7fccd80cc2d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fccc80097c0 tx=0x7fccc801c480 comp rx=0 tx=0).stop 2023-11-28T18:38:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fcce025a640 1 -- 172.21.15.7:0/3247912770 shutdown_connections 2023-11-28T18:38:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fcce025a640 1 --2- 172.21.15.7:0/3247912770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cbb70 0x7fccd80cc2d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fcce025a640 1 --2- 172.21.15.7:0/3247912770 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.481+0000 7fcce025a640 1 -- 172.21.15.7:0/3247912770 >> 172.21.15.7:0/3247912770 conn(0x7fccd80717e0 msgr2=0x7fccd80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 -- 172.21.15.7:0/3247912770 shutdown_connections 2023-11-28T18:38:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 -- 172.21.15.7:0/3247912770 wait complete. 2023-11-28T18:38:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 Processor -- start 2023-11-28T18:38:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 -- start start 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd80cbb70 0x7fccd81472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cc810 0x7fccd80c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd813b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8141030 con 0x7fccd80cc810 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd814b780 con 0x7fccd80cbb70 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fcce025a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd814bef0 con 0x7fccd81542e0 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cc810 0x7fccd80c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cc810 0x7fccd80c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43418/0 (socket says 172.21.15.7:43418) 2023-11-28T18:38:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 -- 172.21.15.7:0/1638683983 learned_addr learned my addr 172.21.15.7:0/1638683983 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccddff6640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd80cbb70 0x7fccd81472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccde7f7640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd813b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 -- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 msgr2=0x7fccd813b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd813b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 -- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd80cbb70 msgr2=0x7fccd81472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd80cbb70 0x7fccd81472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 -- 172.21.15.7:0/1638683983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8009420 con 0x7fccd80cc810 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccde7f7640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd813b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.485+0000 7fccdd7f5640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cc810 0x7fccd80c8790 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fccc80ad040 tx=0x7fccc801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fccce7fc640 1 -- 172.21.15.7:0/1638683983 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80ab680 con 0x7fccd80cc810 2023-11-28T18:38:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fccce7fc640 1 -- 172.21.15.7:0/1638683983 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc80b5810 con 0x7fccd80cc810 2023-11-28T18:38:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fccce7fc640 1 -- 172.21.15.7:0/1638683983 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80b5bc0 con 0x7fccd80cc810 2023-11-28T18:38:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd81414c0 con 0x7fccd80cc810 2023-11-28T18:38:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd80c6d60 con 0x7fccd80cc810 2023-11-28T18:38:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd806c670 con 0x7fccd80cc810 2023-11-28T18:38:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fccce7fc640 1 -- 172.21.15.7:0/1638683983 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fccc80b5d20 con 0x7fccd80cc810 2023-11-28T18:38:38.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fccce7fc640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccbc0668d0 0x7fccbc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:38.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.489+0000 7fccce7fc640 1 -- 172.21.15.7:0/1638683983 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fccc811f400 con 0x7fccd80cc810 2023-11-28T18:38:38.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.493+0000 7fccddff6640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccbc0668d0 0x7fccbc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:38.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.493+0000 7fccddff6640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccbc0668d0 0x7fccbc068d90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fccd400b930 tx=0x7fccd4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.497+0000 7fccce7fc640 1 -- 172.21.15.7:0/1638683983 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fccc80ec270 con 0x7fccd80cc810 2023-11-28T18:38:38.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccd804e890 con 0x7fccd80cc810 2023-11-28T18:38:38.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fccce7fc640 1 -- 172.21.15.7:0/1638683983 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5764 (secure 0 0 0) 0x7fccc80ef890 con 0x7fccd80cc810 2023-11-28T18:38:38.701 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:38.701 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:37.328517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:38.702 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 66 2023-11-28T18:38:38.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccbc0668d0 msgr2=0x7fccbc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:38.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccbc0668d0 0x7fccbc068d90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fccd400b930 tx=0x7fccd4008040 comp rx=0 tx=0).stop 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cc810 msgr2=0x7fccd80c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cc810 0x7fccd80c8790 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fccc80ad040 tx=0x7fccc801acc0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 shutdown_connections 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccbc0668d0 0x7fccbc068d90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccd81542e0 0x7fccd813b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd80cc810 0x7fccd80c8790 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 --2- 172.21.15.7:0/1638683983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd80cbb70 0x7fccd81472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 >> 172.21.15.7:0/1638683983 conn(0x7fccd80717e0 msgr2=0x7fccd815dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 shutdown_connections 2023-11-28T18:38:38.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:38.701+0000 7fcce025a640 1 -- 172.21.15.7:0/1638683983 wait complete. 2023-11-28T18:38:38.723 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 66, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 66, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 33 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:38:37.328517+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5359, "mds_2": 5307, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:38:40.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:40.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550f556640 1 Processor -- start 2023-11-28T18:38:40.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550f556640 1 -- start start 2023-11-28T18:38:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550f556640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55080cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550f556640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550815a3a0 con 0x7f550815ce10 2023-11-28T18:38:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550f556640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550815a620 con 0x7f550815ca30 2023-11-28T18:38:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550f556640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550815a900 con 0x7f55080cafe0 2023-11-28T18:38:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550dd53640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f550815ce10 0x7f55080ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36520/0 (socket says 172.21.15.7:36520) 2023-11-28T18:38:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550dd53640 1 -- 172.21.15.7:0/1258156612 learned_addr learned my addr 172.21.15.7:0/1258156612 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:38:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.881+0000 7f550ed55640 1 --2- 172.21.15.7:0/1258156612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55080cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.885+0000 7f550d552640 1 -- 172.21.15.7:0/1258156612 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701707744 0 0) 0x7f550815a3a0 con 0x7f550815ce10 2023-11-28T18:38:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.885+0000 7f550d552640 1 -- 172.21.15.7:0/1258156612 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f4003530 con 0x7f550815ce10 2023-11-28T18:38:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.885+0000 7f550d552640 1 -- 172.21.15.7:0/1258156612 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704458340 0 0) 0x7f550815a620 con 0x7f550815ca30 2023-11-28T18:38:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.885+0000 7f550d552640 1 -- 172.21.15.7:0/1258156612 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f550815a3a0 con 0x7f550815ca30 2023-11-28T18:38:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.885+0000 7f550ed55640 1 -- 172.21.15.7:0/1258156612 >> v1:172.21.15.193:6789/0 conn(0x7f550815ca30 legacy=0x7f5508159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.885+0000 7f550ed55640 1 -- 172.21.15.7:0/1258156612 >> v1:172.21.15.7:6789/0 conn(0x7f550815ce10 legacy=0x7f55080ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.885+0000 7f550ed55640 1 -- 172.21.15.7:0/1258156612 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550815abe0 con 0x7f55080cafe0 2023-11-28T18:38:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550ed55640 1 --2- 172.21.15.7:0/1258156612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55080cc5f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5500009dc0 tx=0x7f550001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c2b772b028d2aa1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550d552640 1 -- 172.21.15.7:0/1258156612 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55000af600 con 0x7f55080cafe0 2023-11-28T18:38:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550d552640 1 -- 172.21.15.7:0/1258156612 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55000ab050 con 0x7f55080cafe0 2023-11-28T18:38:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550d552640 1 -- 172.21.15.7:0/1258156612 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55000af980 con 0x7f55080cafe0 2023-11-28T18:38:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550f556640 1 -- 172.21.15.7:0/1258156612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 msgr2=0x7f55080cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550f556640 1 --2- 172.21.15.7:0/1258156612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55080cc5f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5500009dc0 tx=0x7f550001f150 comp rx=0 tx=0).stop 2023-11-28T18:38:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550f556640 1 -- 172.21.15.7:0/1258156612 shutdown_connections 2023-11-28T18:38:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550f556640 1 --2- 172.21.15.7:0/1258156612 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55080cc5f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550f556640 1 -- 172.21.15.7:0/1258156612 >> 172.21.15.7:0/1258156612 conn(0x7f550806c060 msgr2=0x7f55080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550f556640 1 -- 172.21.15.7:0/1258156612 shutdown_connections 2023-11-28T18:38:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.889+0000 7f550f556640 1 -- 172.21.15.7:0/1258156612 wait complete. 2023-11-28T18:38:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 Processor -- start 2023-11-28T18:38:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 -- start start 2023-11-28T18:38:40.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55081efd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:40.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f550815ca30 0x7f55080c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:40.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550815ce10 0x7f550814e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55080c45f0 con 0x7f550815ca30 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55080c2df0 con 0x7f550815ce10 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5508155d40 con 0x7f55080cafe0 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f550815ca30 0x7f55080c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f550815ca30 0x7f55080c40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43428/0 (socket says 172.21.15.7:43428) 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 -- 172.21.15.7:0/3491847472 learned_addr learned my addr 172.21.15.7:0/3491847472 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550e554640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55081efd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550ed55640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550815ce10 0x7f550814e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 -- 172.21.15.7:0/3491847472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 msgr2=0x7f55081efd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55081efd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 -- 172.21.15.7:0/3491847472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550815ce10 msgr2=0x7f550814e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550815ce10 0x7f550814e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:40.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 -- 172.21.15.7:0/3491847472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5500009a70 con 0x7f550815ca30 2023-11-28T18:38:40.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550dd53640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f550815ca30 0x7f55080c40b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f54f800c450 tx=0x7f54f800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:40.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f54feffd640 1 -- 172.21.15.7:0/3491847472 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8005cf0 con 0x7f550815ca30 2023-11-28T18:38:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f54feffd640 1 -- 172.21.15.7:0/3491847472 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f8005e50 con 0x7f550815ca30 2023-11-28T18:38:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5508152720 con 0x7f550815ca30 2023-11-28T18:38:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f54feffd640 1 -- 172.21.15.7:0/3491847472 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8007270 con 0x7f550815ca30 2023-11-28T18:38:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.893+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55080c32e0 con 0x7f550815ca30 2023-11-28T18:38:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.897+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5508155ed0 con 0x7f550815ca30 2023-11-28T18:38:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.897+0000 7f54feffd640 1 -- 172.21.15.7:0/3491847472 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f54f80073d0 con 0x7f550815ca30 2023-11-28T18:38:40.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.897+0000 7f54feffd640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54ec0668d0 0x7f54ec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:40.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.897+0000 7f54feffd640 1 -- 172.21.15.7:0/3491847472 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f54f8083b20 con 0x7f550815ca30 2023-11-28T18:38:40.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.897+0000 7f550e554640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54ec0668d0 0x7f54ec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:40.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.897+0000 7f550e554640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54ec0668d0 0x7f54ec068d90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f55040011f0 tx=0x7f5504009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:40.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:40.901+0000 7f54feffd640 1 -- 172.21.15.7:0/3491847472 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f54f8050b70 con 0x7f550815ca30 2023-11-28T18:38:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f550804e890 con 0x7f550815ca30 2023-11-28T18:38:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f54feffd640 1 -- 172.21.15.7:0/3491847472 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5764 (secure 0 0 0) 0x7f54f8017020 con 0x7f550815ca30 2023-11-28T18:38:41.113 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:41.113 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:37.328517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:41.114 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 66 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54ec0668d0 msgr2=0x7f54ec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54ec0668d0 0x7f54ec068d90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f55040011f0 tx=0x7f5504009040 comp rx=0 tx=0).stop 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f550815ca30 msgr2=0x7f55080c40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f550815ca30 0x7f55080c40b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f54f800c450 tx=0x7f54f800c910 comp rx=0 tx=0).stop 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 shutdown_connections 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f54ec0668d0 0x7f54ec068d90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550815ce10 0x7f550814e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f550815ca30 0x7f55080c40b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 --2- 172.21.15.7:0/3491847472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f55080cafe0 0x7f55081efd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.113+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 >> 172.21.15.7:0/3491847472 conn(0x7f550806c060 msgr2=0x7f55081596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.117+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 shutdown_connections 2023-11-28T18:38:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:41.117+0000 7f550f556640 1 -- 172.21.15.7:0/3491847472 wait complete. 2023-11-28T18:38:41.135 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:38:43.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:44.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.461+0000 7f2d1e0c8640 1 Processor -- start 2023-11-28T18:38:44.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.461+0000 7f2d1e0c8640 1 -- start start 2023-11-28T18:38:44.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1e0c8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d1815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:44.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1e0c8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1815d240 con 0x7f2d18153ee0 2023-11-28T18:38:44.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1e0c8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1815d520 con 0x7f2d1815def0 2023-11-28T18:38:44.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1e0c8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180cbd10 con 0x7f2d18159c90 2023-11-28T18:38:44.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1d0c6640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2d18153ee0 0x7f2d180cb600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54004/0 (socket says 172.21.15.7:54004) 2023-11-28T18:38:44.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1d0c6640 1 -- 172.21.15.7:0/266958270 learned_addr learned my addr 172.21.15.7:0/266958270 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:38:44.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1c8c5640 1 --2- 172.21.15.7:0/266958270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d1815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:44.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d0ffff640 1 -- 172.21.15.7:0/266958270 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956015150 0 0) 0x7f2d1815d240 con 0x7f2d18153ee0 2023-11-28T18:38:44.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d0ffff640 1 -- 172.21.15.7:0/266958270 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d04003530 con 0x7f2d18153ee0 2023-11-28T18:38:44.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d0ffff640 1 -- 172.21.15.7:0/266958270 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551967998 0 0) 0x7f2d1815d520 con 0x7f2d1815def0 2023-11-28T18:38:44.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d0ffff640 1 -- 172.21.15.7:0/266958270 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d1815d240 con 0x7f2d1815def0 2023-11-28T18:38:44.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1c8c5640 1 -- 172.21.15.7:0/266958270 >> v1:172.21.15.193:6789/0 conn(0x7f2d1815def0 legacy=0x7f2d180ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:44.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1c8c5640 1 -- 172.21.15.7:0/266958270 >> v1:172.21.15.7:6789/0 conn(0x7f2d18153ee0 legacy=0x7f2d180cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:44.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.465+0000 7f2d1c8c5640 1 -- 172.21.15.7:0/266958270 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d180c4bc0 con 0x7f2d18159c90 2023-11-28T18:38:44.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d1c8c5640 1 --2- 172.21.15.7:0/266958270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d1815d9b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2d080097f0 tx=0x7f2d0801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0535ac0ff8bb06f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:44.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d0ffff640 1 -- 172.21.15.7:0/266958270 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d080a89b0 con 0x7f2d18159c90 2023-11-28T18:38:44.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d0ffff640 1 -- 172.21.15.7:0/266958270 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d080a8b10 con 0x7f2d18159c90 2023-11-28T18:38:44.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d0ffff640 1 -- 172.21.15.7:0/266958270 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d080a8ea0 con 0x7f2d18159c90 2023-11-28T18:38:44.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/266958270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 msgr2=0x7f2d1815d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:44.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d1e0c8640 1 --2- 172.21.15.7:0/266958270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d1815d9b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2d080097f0 tx=0x7f2d0801ecb0 comp rx=0 tx=0).stop 2023-11-28T18:38:44.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/266958270 shutdown_connections 2023-11-28T18:38:44.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d1e0c8640 1 --2- 172.21.15.7:0/266958270 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d1815d9b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:44.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/266958270 >> 172.21.15.7:0/266958270 conn(0x7f2d18071e20 msgr2=0x7f2d18072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:44.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/266958270 shutdown_connections 2023-11-28T18:38:44.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.469+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/266958270 wait complete. 2023-11-28T18:38:44.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 Processor -- start 2023-11-28T18:38:44.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 -- start start 2023-11-28T18:38:44.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d18153ee0 0x7f2d180ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d180cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d1815def0 0x7f2d18149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d181f0660 con 0x7f2d18153ee0 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180cf340 con 0x7f2d1815def0 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180cf080 con 0x7f2d18159c90 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d18153ee0 0x7f2d180ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d18153ee0 0x7f2d180ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49790/0 (socket says 172.21.15.7:49790) 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 -- 172.21.15.7:0/2228023236 learned_addr learned my addr 172.21.15.7:0/2228023236 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:44.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1c8c5640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d180cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:44.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d8c7640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d1815def0 0x7f2d18149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:44.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 -- 172.21.15.7:0/2228023236 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 msgr2=0x7f2d180cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:44.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d180cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:44.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 -- 172.21.15.7:0/2228023236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d1815def0 msgr2=0x7f2d18149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:44.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d1815def0 0x7f2d18149d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:44.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 -- 172.21.15.7:0/2228023236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d08009420 con 0x7f2d18153ee0 2023-11-28T18:38:44.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1d0c6640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d18153ee0 0x7f2d180ce4f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400c4d0 tx=0x7f2d1400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:44.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d0dffb640 1 -- 172.21.15.7:0/2228023236 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14005cf0 con 0x7f2d18153ee0 2023-11-28T18:38:44.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d0dffb640 1 -- 172.21.15.7:0/2228023236 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d14005e50 con 0x7f2d18153ee0 2023-11-28T18:38:44.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d0dffb640 1 -- 172.21.15.7:0/2228023236 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d140072b0 con 0x7f2d18153ee0 2023-11-28T18:38:44.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d18153710 con 0x7f2d18153ee0 2023-11-28T18:38:44.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.473+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d181f0280 con 0x7f2d18153ee0 2023-11-28T18:38:44.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.477+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1804e890 con 0x7f2d18153ee0 2023-11-28T18:38:44.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.477+0000 7f2d0dffb640 1 -- 172.21.15.7:0/2228023236 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2d14007410 con 0x7f2d18153ee0 2023-11-28T18:38:44.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.477+0000 7f2d0dffb640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cfc0668d0 0x7f2cfc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:44.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.477+0000 7f2d0dffb640 1 -- 172.21.15.7:0/2228023236 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f2d14083e20 con 0x7f2d18153ee0 2023-11-28T18:38:44.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.477+0000 7f2d1c8c5640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cfc0668d0 0x7f2cfc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:44.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.477+0000 7f2d1c8c5640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cfc0668d0 0x7f2cfc068d90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2d080b2000 tx=0x7f2d08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.481+0000 7f2d0dffb640 1 -- 172.21.15.7:0/2228023236 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2d14050cb0 con 0x7f2d18153ee0 2023-11-28T18:38:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d180cae90 con 0x7f2d18153ee0 2023-11-28T18:38:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d0dffb640 1 -- 172.21.15.7:0/2228023236 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5766 (secure 0 0 0) 0x7f2d14017070 con 0x7f2d18153ee0 2023-11-28T18:38:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:44.733 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:42.349232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:resolve","state_seq":23,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 67 2023-11-28T18:38:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cfc0668d0 msgr2=0x7f2cfc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cfc0668d0 0x7f2cfc068d90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2d080b2000 tx=0x7f2d08008040 comp rx=0 tx=0).stop 2023-11-28T18:38:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d18153ee0 msgr2=0x7f2d180ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d18153ee0 0x7f2d180ce4f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400c4d0 tx=0x7f2d1400c990 comp rx=0 tx=0).stop 2023-11-28T18:38:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 shutdown_connections 2023-11-28T18:38:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2cfc0668d0 0x7f2cfc068d90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d1815def0 0x7f2d18149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2d18159c90 0x7f2d180cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 --2- 172.21.15.7:0/2228023236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d18153ee0 0x7f2d180ce4f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 >> 172.21.15.7:0/2228023236 conn(0x7f2d18071e20 msgr2=0x7f2d180cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 shutdown_connections 2023-11-28T18:38:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:44.733+0000 7f2d1e0c8640 1 -- 172.21.15.7:0/2228023236 wait complete. 2023-11-28T18:38:44.756 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:38:46.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f759e640 1 Processor -- start 2023-11-28T18:38:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f759e640 1 -- start start 2023-11-28T18:38:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f759e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ca30 0x7f63f8159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f759e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f63f815ce10 0x7f63f80ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f759e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63f80cafe0 0x7f63f80ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f759e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f815a620 con 0x7f63f80cafe0 2023-11-28T18:38:46.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f759e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f815a900 con 0x7f63f815ce10 2023-11-28T18:38:46.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f759e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f815abe0 con 0x7f63f815ca30 2023-11-28T18:38:46.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f6d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63f80cafe0 0x7f63f80ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:46.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f6d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63f80cafe0 0x7f63f80ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49798/0 (socket says 172.21.15.7:49798) 2023-11-28T18:38:46.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f6d9d640 1 -- 172.21.15.7:0/888680954 learned_addr learned my addr 172.21.15.7:0/888680954 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:46.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f5d9b640 1 --2- 172.21.15.7:0/888680954 >> v2:172.21.15.193:3300/0 conn(0x7f63f815ce10 0x7f63f80ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:46.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f659c640 1 --2- 172.21.15.7:0/888680954 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ca30 0x7f63f8159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:46.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f6d9d640 1 -- 172.21.15.7:0/888680954 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ca30 msgr2=0x7f63f8159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:46.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f6d9d640 1 --2- 172.21.15.7:0/888680954 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ca30 0x7f63f8159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:46.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f6d9d640 1 -- 172.21.15.7:0/888680954 >> v2:172.21.15.193:3300/0 conn(0x7f63f815ce10 msgr2=0x7f63f80ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:46.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f6d9d640 1 --2- 172.21.15.7:0/888680954 >> v2:172.21.15.193:3300/0 conn(0x7f63f815ce10 0x7f63f80ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:46.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.909+0000 7f63f6d9d640 1 -- 172.21.15.7:0/888680954 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f80cbb10 con 0x7f63f80cafe0 2023-11-28T18:38:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f6d9d640 1 --2- 172.21.15.7:0/888680954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f80ca710 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f63f806c690 tx=0x7f63f001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d92123ccba0d01 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f559a640 1 -- 172.21.15.7:0/888680954 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00af400 con 0x7f63f80cafe0 2023-11-28T18:38:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f559a640 1 -- 172.21.15.7:0/888680954 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00ab050 con 0x7f63f80cafe0 2023-11-28T18:38:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f559a640 1 -- 172.21.15.7:0/888680954 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00af780 con 0x7f63f80cafe0 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 -- 172.21.15.7:0/888680954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 msgr2=0x7f63f80ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 --2- 172.21.15.7:0/888680954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f80ca710 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f63f806c690 tx=0x7f63f001f1c0 comp rx=0 tx=0).stop 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 -- 172.21.15.7:0/888680954 shutdown_connections 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 --2- 172.21.15.7:0/888680954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f80ca710 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 --2- 172.21.15.7:0/888680954 >> v2:172.21.15.193:3300/0 conn(0x7f63f815ce10 0x7f63f80ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 --2- 172.21.15.7:0/888680954 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ca30 0x7f63f8159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 -- 172.21.15.7:0/888680954 >> 172.21.15.7:0/888680954 conn(0x7f63f806c060 msgr2=0x7f63f80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 -- 172.21.15.7:0/888680954 shutdown_connections 2023-11-28T18:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.913+0000 7f63f759e640 1 -- 172.21.15.7:0/888680954 wait complete. 2023-11-28T18:38:46.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 Processor -- start 2023-11-28T18:38:46.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 -- start start 2023-11-28T18:38:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f81e5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f815ca30 0x7f63f81f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ce10 0x7f63f81f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f81e4d30 con 0x7f63f80cafe0 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8154910 con 0x7f63f815ca30 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80c42f0 con 0x7f63f815ce10 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f81e5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f81e5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49810/0 (socket says 172.21.15.7:49810) 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 -- 172.21.15.7:0/1761267696 learned_addr learned my addr 172.21.15.7:0/1761267696 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f6d9d640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ce10 0x7f63f81f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f5d9b640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f815ca30 0x7f63f81f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 -- 172.21.15.7:0/1761267696 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ce10 msgr2=0x7f63f81f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ce10 0x7f63f81f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 -- 172.21.15.7:0/1761267696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f815ca30 msgr2=0x7f63f81f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f815ca30 0x7f63f81f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 -- 172.21.15.7:0/1761267696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f0009b30 con 0x7f63f80cafe0 2023-11-28T18:38:46.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f659c640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f81e5150 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f63e800d800 tx=0x7f63e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:46.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63e6ffd640 1 -- 172.21.15.7:0/1761267696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e80096a0 con 0x7f63f80cafe0 2023-11-28T18:38:46.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63e6ffd640 1 -- 172.21.15.7:0/1761267696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8009800 con 0x7f63f80cafe0 2023-11-28T18:38:46.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63e6ffd640 1 -- 172.21.15.7:0/1761267696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8009b70 con 0x7f63f80cafe0 2023-11-28T18:38:46.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.917+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f80cdc70 con 0x7f63f80cafe0 2023-11-28T18:38:46.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.921+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f81f3840 con 0x7f63f80cafe0 2023-11-28T18:38:46.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.921+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f804e890 con 0x7f63f80cafe0 2023-11-28T18:38:46.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.921+0000 7f63e6ffd640 1 -- 172.21.15.7:0/1761267696 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f63e8009cd0 con 0x7f63f80cafe0 2023-11-28T18:38:46.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.921+0000 7f63e6ffd640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63d80668d0 0x7f63d8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:46.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.921+0000 7f63e6ffd640 1 -- 172.21.15.7:0/1761267696 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f63e8084dd0 con 0x7f63f80cafe0 2023-11-28T18:38:46.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.921+0000 7f63f5d9b640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63d80668d0 0x7f63d8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:46.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.925+0000 7f63f5d9b640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63d80668d0 0x7f63d8068d90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f63f8154360 tx=0x7f63ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:46.925+0000 7f63e6ffd640 1 -- 172.21.15.7:0/1761267696 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f63e8051cf0 con 0x7f63f80cafe0 2023-11-28T18:38:47.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63f80c8c10 con 0x7f63f80cafe0 2023-11-28T18:38:47.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63e6ffd640 1 -- 172.21.15.7:0/1761267696 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5766 (secure 0 0 0) 0x7f63e8055310 con 0x7f63f80cafe0 2023-11-28T18:38:47.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:47.138 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:42.349232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:resolve","state_seq":23,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:47.138 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 67 2023-11-28T18:38:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63d80668d0 msgr2=0x7f63d8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63d80668d0 0x7f63d8068d90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f63f8154360 tx=0x7f63ec008040 comp rx=0 tx=0).stop 2023-11-28T18:38:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 msgr2=0x7f63f81e5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f81e5150 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f63e800d800 tx=0x7f63e800dcc0 comp rx=0 tx=0).stop 2023-11-28T18:38:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 shutdown_connections 2023-11-28T18:38:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f63d80668d0 0x7f63d8068d90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f63f815ce10 0x7f63f81f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f815ca30 0x7f63f81f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 --2- 172.21.15.7:0/1761267696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63f80cafe0 0x7f63f81e5150 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.137+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 >> 172.21.15.7:0/1761267696 conn(0x7f63f806c060 msgr2=0x7f63f8158610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.141+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 shutdown_connections 2023-11-28T18:38:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:47.141+0000 7f63f759e640 1 -- 172.21.15.7:0/1761267696 wait complete. 2023-11-28T18:38:47.164 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:38:49.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:49.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e6510640 1 Processor -- start 2023-11-28T18:38:49.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e6510640 1 -- start start 2023-11-28T18:38:49.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e6510640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e015ca30 0x7f18e0159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:49.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e6510640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f18e00cafe0 0x7f18e00cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:49.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e6510640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e015a620 con 0x7f18e015ce10 2023-11-28T18:38:49.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e6510640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e015a900 con 0x7f18e00cafe0 2023-11-28T18:38:49.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e6510640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e015abe0 con 0x7f18e015ca30 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e5d0f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f18e00cafe0 0x7f18e00cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e550e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e015ca30 0x7f18e0159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e550e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e015ca30 0x7f18e0159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:56164/0 (socket says 172.21.15.7:56164) 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e550e640 1 -- 172.21.15.7:0/4170788329 learned_addr learned my addr 172.21.15.7:0/4170788329 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e5d0f640 1 --2- 172.21.15.7:0/4170788329 >> v2:172.21.15.193:3300/0 conn(0x7f18e00cafe0 0x7f18e00cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:56808/0 (socket says 172.21.15.7:56808) 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e4d0d640 1 --1- 172.21.15.7:0/4170788329 >> v1:172.21.15.193:6789/0 conn(0x7f18e015ce10 0x7f18e00ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:58436/0 (socket says 172.21.15.7:58436) 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e550e640 1 -- 172.21.15.7:0/4170788329 >> v2:172.21.15.193:3300/0 conn(0x7f18e00cafe0 msgr2=0x7f18e00cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e550e640 1 --2- 172.21.15.7:0/4170788329 >> v2:172.21.15.193:3300/0 conn(0x7f18e00cafe0 0x7f18e00cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e550e640 1 -- 172.21.15.7:0/4170788329 >> v1:172.21.15.193:6789/0 conn(0x7f18e015ce10 legacy=0x7f18e00ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:49.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e550e640 1 -- 172.21.15.7:0/4170788329 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e00cbb10 con 0x7f18e015ca30 2023-11-28T18:38:49.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.337+0000 7f18e5d0f640 1 --2- 172.21.15.7:0/4170788329 >> v2:172.21.15.193:3300/0 conn(0x7f18e00cafe0 0x7f18e00cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:38:49.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.341+0000 7f18e550e640 1 --2- 172.21.15.7:0/4170788329 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e015ca30 0x7f18e0159c90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009870 tx=0x7f18dc01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1232d1b3c4000e5a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.341+0000 7f18d7fff640 1 -- 172.21.15.7:0/4170788329 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0a8a30 con 0x7f18e015ca30 2023-11-28T18:38:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.341+0000 7f18d7fff640 1 -- 172.21.15.7:0/4170788329 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc0a8b90 con 0x7f18e015ca30 2023-11-28T18:38:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.341+0000 7f18d7fff640 1 -- 172.21.15.7:0/4170788329 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0b0440 con 0x7f18e015ca30 2023-11-28T18:38:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.341+0000 7f18e6510640 1 -- 172.21.15.7:0/4170788329 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e015ca30 msgr2=0x7f18e0159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.341+0000 7f18e6510640 1 --2- 172.21.15.7:0/4170788329 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e015ca30 0x7f18e0159c90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009870 tx=0x7f18dc01ed30 comp rx=0 tx=0).stop 2023-11-28T18:38:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 -- 172.21.15.7:0/4170788329 shutdown_connections 2023-11-28T18:38:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 --2- 172.21.15.7:0/4170788329 >> v2:172.21.15.193:3300/0 conn(0x7f18e00cafe0 0x7f18e00cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 --2- 172.21.15.7:0/4170788329 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e015ca30 0x7f18e0159c90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 -- 172.21.15.7:0/4170788329 >> 172.21.15.7:0/4170788329 conn(0x7f18e006c060 msgr2=0x7f18e00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:49.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 -- 172.21.15.7:0/4170788329 shutdown_connections 2023-11-28T18:38:49.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 -- 172.21.15.7:0/4170788329 wait complete. 2023-11-28T18:38:49.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 Processor -- start 2023-11-28T18:38:49.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 -- start start 2023-11-28T18:38:49.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e00cafe0 0x7f18e0153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:49.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e015ca30 0x7f18e01e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:49.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e015ce10 0x7f18e01e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:49.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e01e4c10 con 0x7f18e015ca30 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e0154300 con 0x7f18e015ce10 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e6510640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e01e2110 con 0x7f18e00cafe0 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e4d0d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e015ca30 0x7f18e01e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e4d0d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e015ca30 0x7f18e01e5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49824/0 (socket says 172.21.15.7:49824) 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e4d0d640 1 -- 172.21.15.7:0/3977337269 learned_addr learned my addr 172.21.15.7:0/3977337269 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e550e640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e00cafe0 0x7f18e0153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e5d0f640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e015ce10 0x7f18e01e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e4d0d640 1 -- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e00cafe0 msgr2=0x7f18e0153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e4d0d640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e00cafe0 0x7f18e0153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e4d0d640 1 -- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e015ce10 msgr2=0x7f18e01e62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e4d0d640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e015ce10 0x7f18e01e62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e4d0d640 1 -- 172.21.15.7:0/3977337269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d8009a70 con 0x7f18e015ca30 2023-11-28T18:38:49.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.345+0000 7f18e550e640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e00cafe0 0x7f18e0153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:38:49.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18e4d0d640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e015ca30 0x7f18e01e5d60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f18d000c430 tx=0x7f18d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:49.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18d5ffb640 1 -- 172.21.15.7:0/3977337269 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d00074e0 con 0x7f18e015ca30 2023-11-28T18:38:49.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18d5ffb640 1 -- 172.21.15.7:0/3977337269 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d0007640 con 0x7f18e015ca30 2023-11-28T18:38:49.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18d5ffb640 1 -- 172.21.15.7:0/3977337269 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0004a10 con 0x7f18e015ca30 2023-11-28T18:38:49.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc0094d0 con 0x7f18e015ca30 2023-11-28T18:38:49.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e01f01f0 con 0x7f18e015ca30 2023-11-28T18:38:49.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e00c9f90 con 0x7f18e015ca30 2023-11-28T18:38:49.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18d5ffb640 1 -- 172.21.15.7:0/3977337269 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f18d0004b70 con 0x7f18e015ca30 2023-11-28T18:38:49.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18d5ffb640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c40668d0 0x7f18c4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:49.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.349+0000 7f18d5ffb640 1 -- 172.21.15.7:0/3977337269 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f18d0083fe0 con 0x7f18e015ca30 2023-11-28T18:38:49.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.353+0000 7f18e550e640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c40668d0 0x7f18c4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:49.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.353+0000 7f18e550e640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c40668d0 0x7f18c4068d90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0b2000 tx=0x7f18dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.357+0000 7f18d5ffb640 1 -- 172.21.15.7:0/3977337269 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f18d0050ff0 con 0x7f18e015ca30 2023-11-28T18:38:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.585+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18e004e890 con 0x7f18e015ca30 2023-11-28T18:38:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.585+0000 7f18d5ffb640 1 -- 172.21.15.7:0/3977337269 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5768 (secure 0 0 0) 0x7f18d0054610 con 0x7f18e015ca30 2023-11-28T18:38:49.586 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:49.586 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:48.339347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 68 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c40668d0 msgr2=0x7f18c4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c40668d0 0x7f18c4068d90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0b2000 tx=0x7f18dc009040 comp rx=0 tx=0).stop 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e015ca30 msgr2=0x7f18e01e5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e015ca30 0x7f18e01e5d60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f18d000c430 tx=0x7f18d000c8f0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 shutdown_connections 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18c40668d0 0x7f18c4068d90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e015ce10 0x7f18e01e62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e015ca30 0x7f18e01e5d60 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 --2- 172.21.15.7:0/3977337269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18e00cafe0 0x7f18e0153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 >> 172.21.15.7:0/3977337269 conn(0x7f18e006c060 msgr2=0x7f18e01599f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 shutdown_connections 2023-11-28T18:38:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:49.589+0000 7f18e6510640 1 -- 172.21.15.7:0/3977337269 wait complete. 2023-11-28T18:38:49.610 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:38:51.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.785+0000 7faf192fe640 1 Processor -- start 2023-11-28T18:38:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.785+0000 7faf192fe640 1 -- start start 2023-11-28T18:38:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.785+0000 7faf192fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7faf14151cd0 0x7faf1415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.785+0000 7faf192fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faf14157b90 0x7faf140701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf192fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf14071e60 con 0x7faf1406e560 2023-11-28T18:38:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf192fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf14070730 con 0x7faf14157b90 2023-11-28T18:38:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf192fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf14070930 con 0x7faf14151cd0 2023-11-28T18:38:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf137fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faf14157b90 0x7faf140701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf137fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faf14157b90 0x7faf140701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49838/0 (socket says 172.21.15.7:49838) 2023-11-28T18:38:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf137fe640 1 -- 172.21.15.7:0/1534177120 learned_addr learned my addr 172.21.15.7:0/1534177120 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf12ffd640 1 --2- 172.21.15.7:0/1534177120 >> v2:172.21.15.193:3300/0 conn(0x7faf14151cd0 0x7faf1415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf137fe640 1 -- 172.21.15.7:0/1534177120 >> v2:172.21.15.193:3300/0 conn(0x7faf14151cd0 msgr2=0x7faf1415bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf137fe640 1 --2- 172.21.15.7:0/1534177120 >> v2:172.21.15.193:3300/0 conn(0x7faf14151cd0 0x7faf1415bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf137fe640 1 -- 172.21.15.7:0/1534177120 >> v1:172.21.15.193:6789/0 conn(0x7faf1406e560 legacy=0x7faf140c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.789+0000 7faf137fe640 1 -- 172.21.15.7:0/1534177120 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf14157fa0 con 0x7faf14157b90 2023-11-28T18:38:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf137fe640 1 --2- 172.21.15.7:0/1534177120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140701f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7faf1406fee0 tx=0x7faf0c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd33a1555c87d05 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf127fc640 1 -- 172.21.15.7:0/1534177120 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c0a56f0 con 0x7faf14157b90 2023-11-28T18:38:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf127fc640 1 -- 172.21.15.7:0/1534177120 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c0a5850 con 0x7faf14157b90 2023-11-28T18:38:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf127fc640 1 -- 172.21.15.7:0/1534177120 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c0a5be0 con 0x7faf14157b90 2023-11-28T18:38:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 -- 172.21.15.7:0/1534177120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 msgr2=0x7faf140701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 --2- 172.21.15.7:0/1534177120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140701f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7faf1406fee0 tx=0x7faf0c01c940 comp rx=0 tx=0).stop 2023-11-28T18:38:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 -- 172.21.15.7:0/1534177120 shutdown_connections 2023-11-28T18:38:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 --2- 172.21.15.7:0/1534177120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140701f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 --2- 172.21.15.7:0/1534177120 >> v2:172.21.15.193:3300/0 conn(0x7faf14151cd0 0x7faf1415bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 -- 172.21.15.7:0/1534177120 >> 172.21.15.7:0/1534177120 conn(0x7faf140662e0 msgr2=0x7faf140666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 -- 172.21.15.7:0/1534177120 shutdown_connections 2023-11-28T18:38:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 -- 172.21.15.7:0/1534177120 wait complete. 2023-11-28T18:38:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 Processor -- start 2023-11-28T18:38:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.793+0000 7faf192fe640 1 -- start start 2023-11-28T18:38:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf1406e560 0x7faf14142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf14151cd0 0x7faf140c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf140c9c80 con 0x7faf14157b90 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf140c96c0 con 0x7faf1406e560 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf140c99a0 con 0x7faf14151cd0 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf12ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf1406e560 0x7faf14142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49846/0 (socket says 172.21.15.7:49846) 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 -- 172.21.15.7:0/1086186357 learned_addr learned my addr 172.21.15.7:0/1086186357 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf0bfff640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf14151cd0 0x7faf140c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 -- 172.21.15.7:0/1086186357 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf14151cd0 msgr2=0x7faf140c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf14151cd0 0x7faf140c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 -- 172.21.15.7:0/1086186357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf1406e560 msgr2=0x7faf14142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf1406e560 0x7faf14142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 -- 172.21.15.7:0/1086186357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c009b30 con 0x7faf14157b90 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf137fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140c4010 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0af040 tx=0x7faf0c0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf0b7fe640 1 -- 172.21.15.7:0/1086186357 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c0b8670 con 0x7faf14157b90 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf0b7fe640 1 -- 172.21.15.7:0/1086186357 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c0b87d0 con 0x7faf14157b90 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf0b7fe640 1 -- 172.21.15.7:0/1086186357 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c0b8b80 con 0x7faf14157b90 2023-11-28T18:38:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf14150ad0 con 0x7faf14157b90 2023-11-28T18:38:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf14162240 con 0x7faf14157b90 2023-11-28T18:38:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.797+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1404e890 con 0x7faf14157b90 2023-11-28T18:38:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.801+0000 7faf0b7fe640 1 -- 172.21.15.7:0/1086186357 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7faf0c0b8ce0 con 0x7faf14157b90 2023-11-28T18:38:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.801+0000 7faf0b7fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faef4066900 0x7faef4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.801+0000 7faf0b7fe640 1 -- 172.21.15.7:0/1086186357 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7faf0c11fb00 con 0x7faf14157b90 2023-11-28T18:38:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.801+0000 7faf12ffd640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faef4066900 0x7faef4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.801+0000 7faf12ffd640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faef4066900 0x7faef4068dc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faf04007a60 tx=0x7faf04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:51.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:51.805+0000 7faf0b7fe640 1 -- 172.21.15.7:0/1086186357 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faf0c0ecb50 con 0x7faf14157b90 2023-11-28T18:38:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.009+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf14150f80 con 0x7faf14157b90 2023-11-28T18:38:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.009+0000 7faf0b7fe640 1 -- 172.21.15.7:0/1086186357 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5768 (secure 0 0 0) 0x7faf0c009510 con 0x7faf14157b90 2023-11-28T18:38:52.011 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:48.339347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 68 2023-11-28T18:38:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faef4066900 msgr2=0x7faef4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faef4066900 0x7faef4068dc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faf04007a60 tx=0x7faf04008040 comp rx=0 tx=0).stop 2023-11-28T18:38:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 msgr2=0x7faf140c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140c4010 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0af040 tx=0x7faf0c0196e0 comp rx=0 tx=0).stop 2023-11-28T18:38:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 shutdown_connections 2023-11-28T18:38:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faef4066900 0x7faef4068dc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf14157b90 0x7faf140c4010 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf14151cd0 0x7faf140c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 --2- 172.21.15.7:0/1086186357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf1406e560 0x7faf14142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 >> 172.21.15.7:0/1086186357 conn(0x7faf140662e0 msgr2=0x7faf140724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 shutdown_connections 2023-11-28T18:38:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:52.013+0000 7faf192fe640 1 -- 172.21.15.7:0/1086186357 wait complete. 2023-11-28T18:38:52.042 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 68, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 68, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5307": { "addr": "172.21.15.7:6835/1029915019", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 1029915019, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1029915019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5307, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 33 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:reconnect", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:38:48.339347+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5359, "mds_2": 5307, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:38:54.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:54.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.201+0000 7f4422e9f640 1 Processor -- start 2023-11-28T18:38:54.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.201+0000 7f4422e9f640 1 -- start start 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f4422e9f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f441c15ca30 0x7f441c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f4422e9f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f441c0cafe0 0x7f441c0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f4422e9f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c15a620 con 0x7f441c15ce10 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f4422e9f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c15a900 con 0x7f441c0cafe0 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f4422e9f640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c15abe0 con 0x7f441c15ca30 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f442169c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f441c15ce10 0x7f441c0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46128/0 (socket says 172.21.15.7:46128) 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f442169c640 1 -- 172.21.15.7:0/2348268240 learned_addr learned my addr 172.21.15.7:0/2348268240 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f442269e640 1 --2- 172.21.15.7:0/2348268240 >> v2:172.21.15.7:3300/0 conn(0x7f441c0cafe0 0x7f441c0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f4421e9d640 1 --2- 172.21.15.7:0/2348268240 >> v2:172.21.15.193:3300/0 conn(0x7f441c15ca30 0x7f441c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:54.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f442269e640 1 -- 172.21.15.7:0/2348268240 >> v2:172.21.15.193:3300/0 conn(0x7f441c15ca30 msgr2=0x7f441c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:54.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f442269e640 1 --2- 172.21.15.7:0/2348268240 >> v2:172.21.15.193:3300/0 conn(0x7f441c15ca30 0x7f441c159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f442269e640 1 -- 172.21.15.7:0/2348268240 >> v1:172.21.15.7:6789/0 conn(0x7f441c15ce10 legacy=0x7f441c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:54.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f442269e640 1 -- 172.21.15.7:0/2348268240 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c0cbb10 con 0x7f441c0cafe0 2023-11-28T18:38:54.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.205+0000 7f4420e9b640 1 -- 172.21.15.7:0/2348268240 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129391291 0 0) 0x7f441c15a620 con 0x7f441c15ce10 2023-11-28T18:38:54.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f442269e640 1 --2- 172.21.15.7:0/2348268240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0cc5f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f441c06c660 tx=0x7f441401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5abae329dd327f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:54.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4420e9b640 1 -- 172.21.15.7:0/2348268240 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44140a58b0 con 0x7f441c0cafe0 2023-11-28T18:38:54.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4420e9b640 1 -- 172.21.15.7:0/2348268240 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44140a5a10 con 0x7f441c0cafe0 2023-11-28T18:38:54.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4420e9b640 1 -- 172.21.15.7:0/2348268240 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44140a5da0 con 0x7f441c0cafe0 2023-11-28T18:38:54.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4422e9f640 1 -- 172.21.15.7:0/2348268240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 msgr2=0x7f441c0cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:54.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4422e9f640 1 --2- 172.21.15.7:0/2348268240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0cc5f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f441c06c660 tx=0x7f441401c8e0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4422e9f640 1 -- 172.21.15.7:0/2348268240 shutdown_connections 2023-11-28T18:38:54.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4422e9f640 1 --2- 172.21.15.7:0/2348268240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0cc5f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4422e9f640 1 --2- 172.21.15.7:0/2348268240 >> v2:172.21.15.193:3300/0 conn(0x7f441c15ca30 0x7f441c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4422e9f640 1 -- 172.21.15.7:0/2348268240 >> 172.21.15.7:0/2348268240 conn(0x7f441c06c060 msgr2=0x7f441c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:54.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4422e9f640 1 -- 172.21.15.7:0/2348268240 shutdown_connections 2023-11-28T18:38:54.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.209+0000 7f4422e9f640 1 -- 172.21.15.7:0/2348268240 wait complete. 2023-11-28T18:38:54.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 Processor -- start 2023-11-28T18:38:54.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 -- start start 2023-11-28T18:38:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f441c15ca30 0x7f441c1ec590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f441c15ce10 0x7f441c1f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c1ebb40 con 0x7f441c0cafe0 2023-11-28T18:38:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c1ec100 con 0x7f441c15ce10 2023-11-28T18:38:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c1571d0 con 0x7f441c15ca30 2023-11-28T18:38:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43684/0 (socket says 172.21.15.7:43684) 2023-11-28T18:38:54.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 -- 172.21.15.7:0/1006730558 learned_addr learned my addr 172.21.15.7:0/1006730558 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:54.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f442169c640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f441c15ca30 0x7f441c1ec590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:54.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f442269e640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f441c15ce10 0x7f441c1f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:54.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 -- 172.21.15.7:0/1006730558 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f441c15ca30 msgr2=0x7f441c1ec590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:54.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f441c15ca30 0x7f441c1ec590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 -- 172.21.15.7:0/1006730558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f441c15ce10 msgr2=0x7f441c1f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:54.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f441c15ce10 0x7f441c1f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 -- 172.21.15.7:0/1006730558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4414009a70 con 0x7f441c0cafe0 2023-11-28T18:38:54.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4421e9d640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0c3ec0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f441800e8a0 tx=0x7f441800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:54.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f44127fc640 1 -- 172.21.15.7:0/1006730558 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418009510 con 0x7f441c0cafe0 2023-11-28T18:38:54.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f44127fc640 1 -- 172.21.15.7:0/1006730558 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441800c940 con 0x7f441c0cafe0 2023-11-28T18:38:54.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f44127fc640 1 -- 172.21.15.7:0/1006730558 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441800ccb0 con 0x7f441c0cafe0 2023-11-28T18:38:54.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c157470 con 0x7f441c0cafe0 2023-11-28T18:38:54.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.213+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c1f33d0 con 0x7f441c0cafe0 2023-11-28T18:38:54.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.217+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c04e890 con 0x7f441c0cafe0 2023-11-28T18:38:54.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.217+0000 7f44127fc640 1 -- 172.21.15.7:0/1006730558 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4418013470 con 0x7f441c0cafe0 2023-11-28T18:38:54.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.217+0000 7f44127fc640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4400066900 0x7f4400068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:54.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.217+0000 7f44127fc640 1 -- 172.21.15.7:0/1006730558 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f4418084840 con 0x7f441c0cafe0 2023-11-28T18:38:54.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.217+0000 7f442169c640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4400066900 0x7f4400068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:54.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.217+0000 7f442169c640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4400066900 0x7f4400068dc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f441c164440 tx=0x7f440c007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:54.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.225+0000 7f44127fc640 1 -- 172.21.15.7:0/1006730558 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4418051770 con 0x7f441c0cafe0 2023-11-28T18:38:54.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.441+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f441c0cad10 con 0x7f441c0cafe0 2023-11-28T18:38:54.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.441+0000 7f44127fc640 1 -- 172.21.15.7:0/1006730558 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f4418017090 con 0x7f441c0cafe0 2023-11-28T18:38:54.441 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:54.441 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:53.431020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:54.442 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 69 2023-11-28T18:38:54.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4400066900 msgr2=0x7f4400068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:54.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4400066900 0x7f4400068dc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f441c164440 tx=0x7f440c007d60 comp rx=0 tx=0).stop 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 msgr2=0x7f441c0c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0c3ec0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f441800e8a0 tx=0x7f441800eca0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 shutdown_connections 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4400066900 0x7f4400068dc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f441c15ce10 0x7f441c1f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f441c15ca30 0x7f441c1ec590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 --2- 172.21.15.7:0/1006730558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f441c0cafe0 0x7f441c0c3ec0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 >> 172.21.15.7:0/1006730558 conn(0x7f441c06c060 msgr2=0x7f441c1593c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 shutdown_connections 2023-11-28T18:38:54.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:54.445+0000 7f4422e9f640 1 -- 172.21.15.7:0/1006730558 wait complete. 2023-11-28T18:38:54.467 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:38:56.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0e4b56640 1 Processor -- start 2023-11-28T18:38:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0e4b56640 1 -- start start 2023-11-28T18:38:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0e4b56640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ca30 0x7ff0e0159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0e4b56640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff0e00cafe0 0x7ff0e00cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0e4b56640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e015a620 con 0x7ff0e015ce10 2023-11-28T18:38:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0e4b56640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e015a900 con 0x7ff0e00cafe0 2023-11-28T18:38:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0e4b56640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e015abe0 con 0x7ff0e015ca30 2023-11-28T18:38:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0deffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff0e015ce10 0x7ff0e00ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46138/0 (socket says 172.21.15.7:46138) 2023-11-28T18:38:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0deffd640 1 -- 172.21.15.7:0/3876646006 learned_addr learned my addr 172.21.15.7:0/3876646006 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0dffff640 1 --2- 172.21.15.7:0/3876646006 >> v2:172.21.15.193:3300/0 conn(0x7ff0e00cafe0 0x7ff0e00cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0df7fe640 1 --2- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ca30 0x7ff0e0159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0de7fc640 1 -- 172.21.15.7:0/3876646006 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625854114 0 0) 0x7ff0e015a620 con 0x7ff0e015ce10 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0de7fc640 1 -- 172.21.15.7:0/3876646006 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c8003530 con 0x7ff0e015ce10 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0dffff640 1 -- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ca30 msgr2=0x7ff0e0159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0dffff640 1 --2- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ca30 0x7ff0e0159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0dffff640 1 -- 172.21.15.7:0/3876646006 >> v1:172.21.15.7:6789/0 conn(0x7ff0e015ce10 legacy=0x7ff0e00ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0dffff640 1 -- 172.21.15.7:0/3876646006 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e00cbb10 con 0x7ff0e00cafe0 2023-11-28T18:38:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.633+0000 7ff0df7fe640 1 --2- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ca30 0x7ff0e0159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:38:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0dffff640 1 --2- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 0x7ff0e00cc5f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009d90 tx=0x7ff0d801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ef42c98056865fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0de7fc640 1 -- 172.21.15.7:0/3876646006 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80af600 con 0x7ff0e00cafe0 2023-11-28T18:38:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0de7fc640 1 -- 172.21.15.7:0/3876646006 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d80ab050 con 0x7ff0e00cafe0 2023-11-28T18:38:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0de7fc640 1 -- 172.21.15.7:0/3876646006 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80af980 con 0x7ff0e00cafe0 2023-11-28T18:38:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0e4b56640 1 -- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 msgr2=0x7ff0e00cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 0x7ff0e00cc5f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009d90 tx=0x7ff0d801f130 comp rx=0 tx=0).stop 2023-11-28T18:38:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0e4b56640 1 -- 172.21.15.7:0/3876646006 shutdown_connections 2023-11-28T18:38:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 0x7ff0e00cc5f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/3876646006 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ca30 0x7ff0e0159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0e4b56640 1 -- 172.21.15.7:0/3876646006 >> 172.21.15.7:0/3876646006 conn(0x7ff0e006c060 msgr2=0x7ff0e00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0e4b56640 1 -- 172.21.15.7:0/3876646006 shutdown_connections 2023-11-28T18:38:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.637+0000 7ff0e4b56640 1 -- 172.21.15.7:0/3876646006 wait complete. 2023-11-28T18:38:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 Processor -- start 2023-11-28T18:38:56.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 -- start start 2023-11-28T18:38:56.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 0x7ff0e01f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e015ca30 0x7ff0e0159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ce10 0x7ff0e01e2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00c38e0 con 0x7ff0e015ca30 2023-11-28T18:38:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e01f0650 con 0x7ff0e00cafe0 2023-11-28T18:38:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e01ef1e0 con 0x7ff0e015ce10 2023-11-28T18:38:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e015ca30 0x7ff0e0159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e015ca30 0x7ff0e0159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43700/0 (socket says 172.21.15.7:43700) 2023-11-28T18:38:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 -- 172.21.15.7:0/388909232 learned_addr learned my addr 172.21.15.7:0/388909232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0dffff640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ce10 0x7ff0e01e2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0df7fe640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 0x7ff0e01f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 -- 172.21.15.7:0/388909232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ce10 msgr2=0x7ff0e01e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ce10 0x7ff0e01e2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 -- 172.21.15.7:0/388909232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 msgr2=0x7ff0e01f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 0x7ff0e01f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 -- 172.21.15.7:0/388909232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d0009420 con 0x7ff0e015ca30 2023-11-28T18:38:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0deffd640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e015ca30 0x7ff0e0159c50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400c450 tx=0x7ff0d400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0c7fff640 1 -- 172.21.15.7:0/388909232 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4005cf0 con 0x7ff0e015ca30 2023-11-28T18:38:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0c7fff640 1 -- 172.21.15.7:0/388909232 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d4005e50 con 0x7ff0e015ca30 2023-11-28T18:38:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0c7fff640 1 -- 172.21.15.7:0/388909232 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4007270 con 0x7ff0e015ca30 2023-11-28T18:38:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d8009a70 con 0x7ff0e015ca30 2023-11-28T18:38:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.641+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e01ef480 con 0x7ff0e015ca30 2023-11-28T18:38:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.645+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e004e890 con 0x7ff0e015ca30 2023-11-28T18:38:56.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.645+0000 7ff0c7fff640 1 -- 172.21.15.7:0/388909232 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff0d40073d0 con 0x7ff0e015ca30 2023-11-28T18:38:56.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.645+0000 7ff0c7fff640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff0cc0668d0 0x7ff0cc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:56.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.645+0000 7ff0c7fff640 1 -- 172.21.15.7:0/388909232 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7ff0d4083b50 con 0x7ff0e015ca30 2023-11-28T18:38:56.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.645+0000 7ff0df7fe640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff0cc0668d0 0x7ff0cc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:56.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.645+0000 7ff0df7fe640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff0cc0668d0 0x7ff0cc068d90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009cf0 tx=0x7ff0d0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:56.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.649+0000 7ff0c7fff640 1 -- 172.21.15.7:0/388909232 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff0d4050b00 con 0x7ff0e015ca30 2023-11-28T18:38:56.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.921+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0e00cad10 con 0x7ff0e015ca30 2023-11-28T18:38:56.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.921+0000 7ff0c7fff640 1 -- 172.21.15.7:0/388909232 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5765 (secure 0 0 0) 0x7ff0d4017020 con 0x7ff0e015ca30 2023-11-28T18:38:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:56.926 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:53.431020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 69 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff0cc0668d0 msgr2=0x7ff0cc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff0cc0668d0 0x7ff0cc068d90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009cf0 tx=0x7ff0d0009190 comp rx=0 tx=0).stop 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e015ca30 msgr2=0x7ff0e0159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e015ca30 0x7ff0e0159c50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400c450 tx=0x7ff0d400c910 comp rx=0 tx=0).stop 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 shutdown_connections 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff0cc0668d0 0x7ff0cc068d90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0e015ce10 0x7ff0e01e2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e015ca30 0x7ff0e0159c50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 --2- 172.21.15.7:0/388909232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0e00cafe0 0x7ff0e01f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:56.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 >> 172.21.15.7:0/388909232 conn(0x7ff0e006c060 msgr2=0x7ff0e00ca1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:56.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 shutdown_connections 2023-11-28T18:38:56.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:56.925+0000 7ff0e4b56640 1 -- 172.21.15.7:0/388909232 wait complete. 2023-11-28T18:38:56.951 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:38:58.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:38:59.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e3f12a640 1 Processor -- start 2023-11-28T18:38:59.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e3f12a640 1 -- start start 2023-11-28T18:38:59.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e3f12a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e380c24f0 0x7f8e381575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:59.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e3f12a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e380c6140 con 0x7f8e380c7980 2023-11-28T18:38:59.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e3f12a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e38157b00 con 0x7f8e38150b30 2023-11-28T18:38:59.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e3f12a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e380c72b0 con 0x7f8e380c24f0 2023-11-28T18:38:59.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e37fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e380c24f0 0x7f8e381575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:59.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e3d6c7640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8e380c7980 0x7f8e380c7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46146/0 (socket says 172.21.15.7:46146) 2023-11-28T18:38:59.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e3d6c7640 1 -- 172.21.15.7:0/346978402 learned_addr learned my addr 172.21.15.7:0/346978402 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:38:59.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e37fff640 1 -- 172.21.15.7:0/346978402 >> v1:172.21.15.193:6789/0 conn(0x7f8e38150b30 legacy=0x7f8e380c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:59.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e37fff640 1 -- 172.21.15.7:0/346978402 >> v1:172.21.15.7:6789/0 conn(0x7f8e380c7980 legacy=0x7f8e380c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:59.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e377fe640 1 -- 172.21.15.7:0/346978402 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954189936 0 0) 0x7f8e380c6140 con 0x7f8e380c7980 2023-11-28T18:38:59.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.121+0000 7f8e37fff640 1 -- 172.21.15.7:0/346978402 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e380c2960 con 0x7f8e380c24f0 2023-11-28T18:38:59.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e37fff640 1 --2- 172.21.15.7:0/346978402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e380c24f0 0x7f8e381575c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8e240097c0 tx=0x7f8e2401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747f2df6c14a6e24 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:59.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e377fe640 1 -- 172.21.15.7:0/346978402 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e240a8970 con 0x7f8e380c24f0 2023-11-28T18:38:59.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e377fe640 1 -- 172.21.15.7:0/346978402 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e240a8ad0 con 0x7f8e380c24f0 2023-11-28T18:38:59.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e377fe640 1 -- 172.21.15.7:0/346978402 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e240a8e60 con 0x7f8e380c24f0 2023-11-28T18:38:59.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e3f12a640 1 -- 172.21.15.7:0/346978402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e380c24f0 msgr2=0x7f8e381575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:59.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e3f12a640 1 --2- 172.21.15.7:0/346978402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e380c24f0 0x7f8e381575c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8e240097c0 tx=0x7f8e2401ec70 comp rx=0 tx=0).stop 2023-11-28T18:38:59.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e3f12a640 1 -- 172.21.15.7:0/346978402 shutdown_connections 2023-11-28T18:38:59.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e3f12a640 1 --2- 172.21.15.7:0/346978402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e380c24f0 0x7f8e381575c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:59.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e3f12a640 1 -- 172.21.15.7:0/346978402 >> 172.21.15.7:0/346978402 conn(0x7f8e380662e0 msgr2=0x7f8e380666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:59.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e3f12a640 1 -- 172.21.15.7:0/346978402 shutdown_connections 2023-11-28T18:38:59.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.125+0000 7f8e3f12a640 1 -- 172.21.15.7:0/346978402 wait complete. 2023-11-28T18:38:59.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3f12a640 1 Processor -- start 2023-11-28T18:38:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3f12a640 1 -- start start 2023-11-28T18:38:59.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3f12a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e380c24f0 0x7f8e38153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3f12a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e380c7980 0x7f8e38155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3f12a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e38150b30 0x7f8e380714e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3f12a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e38155740 con 0x7f8e38150b30 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3f12a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e38153040 con 0x7f8e380c24f0 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3f12a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e381ebc00 con 0x7f8e380c7980 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e380c24f0 0x7f8e38153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3d6c7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e38150b30 0x7f8e380714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e37fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e380c7980 0x7f8e38155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e380c24f0 0x7f8e38153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:60428/0 (socket says 172.21.15.7:60428) 2023-11-28T18:38:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 -- 172.21.15.7:0/3175324443 learned_addr learned my addr 172.21.15.7:0/3175324443 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:38:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 -- 172.21.15.7:0/3175324443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e380c7980 msgr2=0x7f8e38155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e380c7980 0x7f8e38155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 -- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e38150b30 msgr2=0x7f8e380714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e38150b30 0x7f8e380714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:59.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 -- 172.21.15.7:0/3175324443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24009420 con 0x7f8e380c24f0 2023-11-28T18:38:59.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.129+0000 7f8e3cec6640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e380c24f0 0x7f8e38153b70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00c3f0 tx=0x7f8e2c00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:59.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e357fa640 1 -- 172.21.15.7:0/3175324443 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0052d0 con 0x7f8e380c24f0 2023-11-28T18:38:59.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e357fa640 1 -- 172.21.15.7:0/3175324443 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2c005430 con 0x7f8e380c24f0 2023-11-28T18:38:59.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e357fa640 1 -- 172.21.15.7:0/3175324443 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c003840 con 0x7f8e380c24f0 2023-11-28T18:38:59.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e381540b0 con 0x7f8e380c24f0 2023-11-28T18:38:59.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e38155460 con 0x7f8e380c24f0 2023-11-28T18:38:59.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3814dc60 con 0x7f8e380c24f0 2023-11-28T18:38:59.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e357fa640 1 -- 172.21.15.7:0/3175324443 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8e2c0039a0 con 0x7f8e380c24f0 2023-11-28T18:38:59.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e357fa640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e1c0668d0 0x7f8e1c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:38:59.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e357fa640 1 -- 172.21.15.7:0/3175324443 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f8e2c083c30 con 0x7f8e380c24f0 2023-11-28T18:38:59.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.133+0000 7f8e37fff640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e1c0668d0 0x7f8e1c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:38:59.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.137+0000 7f8e37fff640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e1c0668d0 0x7f8e1c068d90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8e240b2000 tx=0x7f8e24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:38:59.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.137+0000 7f8e357fa640 1 -- 172.21.15.7:0/3175324443 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8e2c050bf0 con 0x7f8e380c24f0 2023-11-28T18:38:59.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.341+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e3804e890 con 0x7f8e380c24f0 2023-11-28T18:38:59.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.341+0000 7f8e357fa640 1 -- 172.21.15.7:0/3175324443 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f8e2c017090 con 0x7f8e380c24f0 2023-11-28T18:38:59.343 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:38:59.343 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:38:59.343 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 70 2023-11-28T18:38:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e1c0668d0 msgr2=0x7f8e1c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e1c0668d0 0x7f8e1c068d90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8e240b2000 tx=0x7f8e24008040 comp rx=0 tx=0).stop 2023-11-28T18:38:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e380c24f0 msgr2=0x7f8e38153b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e380c24f0 0x7f8e38153b70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00c3f0 tx=0x7f8e2c00c8b0 comp rx=0 tx=0).stop 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 shutdown_connections 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e1c0668d0 0x7f8e1c068d90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e38150b30 0x7f8e380714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e380c7980 0x7f8e38155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 --2- 172.21.15.7:0/3175324443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e380c24f0 0x7f8e38153b70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 >> 172.21.15.7:0/3175324443 conn(0x7f8e380662e0 msgr2=0x7f8e3814fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 shutdown_connections 2023-11-28T18:38:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:38:59.345+0000 7f8e3f12a640 1 -- 172.21.15.7:0/3175324443 wait complete. 2023-11-28T18:38:59.367 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=3, replacing gid=5086 2023-11-28T18:38:59.367 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.f 2023-11-28T18:39:01.369 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-11-28T18:39:01.369 INFO:tasks.ceph.mds.f:Restarting daemon 2023-11-28T18:39:01.369 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-11-28T18:39:01.373 INFO:tasks.ceph.mds.f:Started 2023-11-28T18:39:01.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:39:01.457 INFO:tasks.ceph.mds.f.smithi193.stdout:starting mds.f at 2023-11-28T18:39:01.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.525+0000 7fa156d0f640 1 Processor -- start 2023-11-28T18:39:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.525+0000 7fa156d0f640 1 -- start start 2023-11-28T18:39:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa156d0f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 0x7fa15006ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa156d0f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa15006f470 0x7fa1500c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa156d0f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1501523b0 con 0x7fa150151fd0 2023-11-28T18:39:01.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa156d0f640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15015bf70 con 0x7fa15006f470 2023-11-28T18:39:01.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa156d0f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500c4420 con 0x7fa150158690 2023-11-28T18:39:01.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa154aab640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa150151fd0 0x7fa1500c3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46152/0 (socket says 172.21.15.7:46152) 2023-11-28T18:39:01.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa154aab640 1 -- 172.21.15.7:0/151885226 learned_addr learned my addr 172.21.15.7:0/151885226 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:39:01.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14ffff640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 0x7fa15006ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295748628 0 0) 0x7fa1501523b0 con 0x7fa150151fd0 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa1552ac640 1 --2- 172.21.15.7:0/151885226 >> v2:172.21.15.193:3300/0 conn(0x7fa15006f470 0x7fa1500c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa140003530 con 0x7fa150151fd0 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703897030 0 0) 0x7fa140003530 con 0x7fa150151fd0 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 msgr2=0x7fa15006ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 0x7fa15006ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 >> v2:172.21.15.193:3300/0 conn(0x7fa15006f470 msgr2=0x7fa1500c5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 --2- 172.21.15.7:0/151885226 >> v2:172.21.15.193:3300/0 conn(0x7fa15006f470 0x7fa1500c5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15015b360 con 0x7fa150151fd0 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14ffff640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 0x7fa15006ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fa144001680 con 0x7fa150151fd0 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 >> v1:172.21.15.7:6789/0 conn(0x7fa150151fd0 legacy=0x7fa1500c3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140003980 0x7fa140003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 0x7fa1400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa140004d10 0x7fa140005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15015bf70 con 0x7fa140004d10 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500c4420 con 0x7fa140003980 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14000fda0 con 0x7fa150158690 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa1552ac640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa140004d10 0x7fa140005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa154aab640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140003980 0x7fa140003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa14ffff640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 0x7fa1400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa1552ac640 1 -- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 msgr2=0x7fa1400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa1552ac640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150158690 0x7fa1400047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa1552ac640 1 -- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140003980 msgr2=0x7fa140003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa1552ac640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140003980 0x7fa140003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.529+0000 7fa1552ac640 1 -- 172.21.15.7:0/151885226 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c009420 con 0x7fa140004d10 2023-11-28T18:39:01.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa1552ac640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa140004d10 0x7fa140005190 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa148009fc0 tx=0x7fa14801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f53cf97918a850 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:39:01.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1480a5cb0 con 0x7fa140004d10 2023-11-28T18:39:01.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14800f040 con 0x7fa140004d10 2023-11-28T18:39:01.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa14f7fe640 1 -- 172.21.15.7:0/151885226 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1480ad680 con 0x7fa140004d10 2023-11-28T18:39:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa156d0f640 1 -- 172.21.15.7:0/151885226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa140004d10 msgr2=0x7fa140005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:39:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa156d0f640 1 --2- 172.21.15.7:0/151885226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa140004d10 0x7fa140005190 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa148009fc0 tx=0x7fa14801cca0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa1552ac640 1 -- 172.21.15.7:0/151885226 reap_dead start 2023-11-28T18:39:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa156d0f640 1 -- 172.21.15.7:0/151885226 shutdown_connections 2023-11-28T18:39:01.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa156d0f640 1 -- 172.21.15.7:0/151885226 >> 172.21.15.7:0/151885226 conn(0x7fa1500cc490 msgr2=0x7fa1500cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:39:01.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa156d0f640 1 -- 172.21.15.7:0/151885226 shutdown_connections 2023-11-28T18:39:01.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.533+0000 7fa156d0f640 1 -- 172.21.15.7:0/151885226 wait complete. 2023-11-28T18:39:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 Processor -- start 2023-11-28T18:39:01.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 -- start start 2023-11-28T18:39:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140004d10 0x7fa1501eb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15015adc0 0x7fa1501427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150072dd0 0x7fa150144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa150158f50 con 0x7fa15015adc0 2023-11-28T18:39:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa150159230 con 0x7fa140004d10 2023-11-28T18:39:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa150159510 con 0x7fa150072dd0 2023-11-28T18:39:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15015adc0 0x7fa1501427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15015adc0 0x7fa1501427e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43744/0 (socket says 172.21.15.7:43744) 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 -- 172.21.15.7:0/800560687 learned_addr learned my addr 172.21.15.7:0/800560687 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa154aab640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140004d10 0x7fa1501eb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa1552ac640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150072dd0 0x7fa150144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 -- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150072dd0 msgr2=0x7fa150144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150072dd0 0x7fa150144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 -- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140004d10 msgr2=0x7fa1501eb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140004d10 0x7fa1501eb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 -- 172.21.15.7:0/800560687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1501597f0 con 0x7fa15015adc0 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa1552ac640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150072dd0 0x7fa150144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:39:01.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14ffff640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15015adc0 0x7fa1501427e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa13c011cd0 tx=0x7fa13c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:39:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14d7fa640 1 -- 172.21.15.7:0/800560687 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c00fd20 con 0x7fa15015adc0 2023-11-28T18:39:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14d7fa640 1 -- 172.21.15.7:0/800560687 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13c00fe80 con 0x7fa15015adc0 2023-11-28T18:39:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa14d7fa640 1 -- 172.21.15.7:0/800560687 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c004240 con 0x7fa15015adc0 2023-11-28T18:39:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa150159ad0 con 0x7fa15015adc0 2023-11-28T18:39:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.537+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1501ec000 con 0x7fa15015adc0 2023-11-28T18:39:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.541+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa150071120 con 0x7fa15015adc0 2023-11-28T18:39:01.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.541+0000 7fa14d7fa640 1 -- 172.21.15.7:0/800560687 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa13c0043a0 con 0x7fa15015adc0 2023-11-28T18:39:01.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.541+0000 7fa14d7fa640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa134066900 0x7fa134068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:39:01.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.541+0000 7fa14d7fa640 1 -- 172.21.15.7:0/800560687 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fa13c084e50 con 0x7fa15015adc0 2023-11-28T18:39:01.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.541+0000 7fa154aab640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa134066900 0x7fa134068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:39:01.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.541+0000 7fa154aab640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa134066900 0x7fa134068dc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa144001c00 tx=0x7fa144001de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:39:01.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.545+0000 7fa14d7fa640 1 -- 172.21.15.7:0/800560687 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa13c051e00 con 0x7fa15015adc0 2023-11-28T18:39:01.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.789+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa15004e890 con 0x7fa15015adc0 2023-11-28T18:39:01.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.789+0000 7fa14d7fa640 1 -- 172.21.15.7:0/800560687 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6558 (secure 0 0 0) 0x7fa13c022070 con 0x7fa15015adc0 2023-11-28T18:39:01.791 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:39:01.792 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:39:01.792 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 71 2023-11-28T18:39:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa134066900 msgr2=0x7fa134068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:39:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa134066900 0x7fa134068dc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa144001c00 tx=0x7fa144001de0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15015adc0 msgr2=0x7fa1501427e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:39:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15015adc0 0x7fa1501427e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa13c011cd0 tx=0x7fa13c00f610 comp rx=0 tx=0).stop 2023-11-28T18:39:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 shutdown_connections 2023-11-28T18:39:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa134066900 0x7fa134068dc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa150072dd0 0x7fa150144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15015adc0 0x7fa1501427e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 --2- 172.21.15.7:0/800560687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa140004d10 0x7fa1501eb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:39:01.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 >> 172.21.15.7:0/800560687 conn(0x7fa1500cc490 msgr2=0x7fa1500ca290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:39:01.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 shutdown_connections 2023-11-28T18:39:01.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:39:01.793+0000 7fa156d0f640 1 -- 172.21.15.7:0/800560687 wait complete. 2023-11-28T18:39:01.815 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-11-28T18:39:01.815 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 59 secs before thrashing 2023-11-28T18:39:02.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:39:02.742 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:39:33.119 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:39:33.123 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:40:00.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:00.893 INFO:tasks.workunit.client.0.smithi007.stdout:fs setup took 838 secs 2023-11-28T18:40:00.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6fb1d640 1 Processor -- start 2023-11-28T18:40:00.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6fb1d640 1 -- start start 2023-11-28T18:40:00.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6fb1d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb680c68c0 0x7feb680c2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:00.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6fb1d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7feb6815e300 0x7feb68157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:00.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6fb1d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6815e820 con 0x7feb680c6ca0 2023-11-28T18:40:00.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6fb1d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb6815eb00 con 0x7feb680c68c0 2023-11-28T18:40:00.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6fb1d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb6815ede0 con 0x7feb6815e300 2023-11-28T18:40:00.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6d0b8640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feb680c6ca0 0x7feb68161f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52630/0 (socket says 172.21.15.7:52630) 2023-11-28T18:40:00.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6d0b8640 1 -- 172.21.15.7:0/1214657880 learned_addr learned my addr 172.21.15.7:0/1214657880 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:00.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6d8b9640 1 --2- 172.21.15.7:0/1214657880 >> v2:172.21.15.7:3300/0 conn(0x7feb680c68c0 0x7feb680c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:00.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6e0ba640 1 --2- 172.21.15.7:0/1214657880 >> v2:172.21.15.193:3300/0 conn(0x7feb6815e300 0x7feb68157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:00.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6d8b9640 1 -- 172.21.15.7:0/1214657880 >> v2:172.21.15.193:3300/0 conn(0x7feb6815e300 msgr2=0x7feb68157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:00.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6d8b9640 1 --2- 172.21.15.7:0/1214657880 >> v2:172.21.15.193:3300/0 conn(0x7feb6815e300 0x7feb68157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:00.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6d8b9640 1 -- 172.21.15.7:0/1214657880 >> v1:172.21.15.7:6789/0 conn(0x7feb680c6ca0 legacy=0x7feb68161f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:00.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6d8b9640 1 -- 172.21.15.7:0/1214657880 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6815f0c0 con 0x7feb680c68c0 2023-11-28T18:40:00.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.972+0000 7feb6c8b7640 1 -- 172.21.15.7:0/1214657880 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344164532 0 0) 0x7feb6815e820 con 0x7feb680c6ca0 2023-11-28T18:40:00.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6d8b9640 1 --2- 172.21.15.7:0/1214657880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c68c0 0x7feb680c2260 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7feb680c7160 tx=0x7feb6401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b73149d930ed9f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:00.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6c8b7640 1 -- 172.21.15.7:0/1214657880 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb640a8950 con 0x7feb680c68c0 2023-11-28T18:40:00.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6c8b7640 1 -- 172.21.15.7:0/1214657880 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb640a8ab0 con 0x7feb680c68c0 2023-11-28T18:40:00.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6c8b7640 1 -- 172.21.15.7:0/1214657880 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb640a8e40 con 0x7feb680c68c0 2023-11-28T18:40:00.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6fb1d640 1 -- 172.21.15.7:0/1214657880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c68c0 msgr2=0x7feb680c2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:00.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/1214657880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c68c0 0x7feb680c2260 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7feb680c7160 tx=0x7feb6401ec50 comp rx=0 tx=0).stop 2023-11-28T18:40:00.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6fb1d640 1 -- 172.21.15.7:0/1214657880 shutdown_connections 2023-11-28T18:40:00.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/1214657880 >> v2:172.21.15.193:3300/0 conn(0x7feb6815e300 0x7feb68157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:00.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/1214657880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c68c0 0x7feb680c2260 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:00.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6fb1d640 1 -- 172.21.15.7:0/1214657880 >> 172.21.15.7:0/1214657880 conn(0x7feb680662e0 msgr2=0x7feb680666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:00.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6fb1d640 1 -- 172.21.15.7:0/1214657880 shutdown_connections 2023-11-28T18:40:00.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.976+0000 7feb6fb1d640 1 -- 172.21.15.7:0/1214657880 wait complete. 2023-11-28T18:40:00.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 Processor -- start 2023-11-28T18:40:00.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 -- start start 2023-11-28T18:40:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb680c68c0 0x7feb68161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c6ca0 0x7feb68147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb6815e300 0x7feb68147f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb681f3f10 con 0x7feb680c6ca0 2023-11-28T18:40:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb680ca590 con 0x7feb680c68c0 2023-11-28T18:40:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb680cb080 con 0x7feb6815e300 2023-11-28T18:40:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c6ca0 0x7feb68147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c6ca0 0x7feb68147a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50316/0 (socket says 172.21.15.7:50316) 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 -- 172.21.15.7:0/564627920 learned_addr learned my addr 172.21.15.7:0/564627920 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d8b9640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb680c68c0 0x7feb68161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6e0ba640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb6815e300 0x7feb68147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 -- 172.21.15.7:0/564627920 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb6815e300 msgr2=0x7feb68147f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb6815e300 0x7feb68147f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 -- 172.21.15.7:0/564627920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb680c68c0 msgr2=0x7feb68161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb680c68c0 0x7feb68161120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 -- 172.21.15.7:0/564627920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb64009420 con 0x7feb680c6ca0 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6d0b8640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c6ca0 0x7feb68147a40 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7feb5800c520 tx=0x7feb5800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb5e7fc640 1 -- 172.21.15.7:0/564627920 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb580074e0 con 0x7feb680c6ca0 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb5e7fc640 1 -- 172.21.15.7:0/564627920 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb58007640 con 0x7feb680c6ca0 2023-11-28T18:40:00.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb680ca7e0 con 0x7feb680c6ca0 2023-11-28T18:40:00.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.980+0000 7feb5e7fc640 1 -- 172.21.15.7:0/564627920 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb580049f0 con 0x7feb680c6ca0 2023-11-28T18:40:00.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.984+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb680cada0 con 0x7feb680c6ca0 2023-11-28T18:40:00.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.984+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb681f40c0 con 0x7feb680c6ca0 2023-11-28T18:40:00.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.984+0000 7feb5e7fc640 1 -- 172.21.15.7:0/564627920 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7feb58004b50 con 0x7feb680c6ca0 2023-11-28T18:40:00.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.984+0000 7feb5e7fc640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb4c0668d0 0x7feb4c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:00.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.984+0000 7feb5e7fc640 1 -- 172.21.15.7:0/564627920 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7feb58083d70 con 0x7feb680c6ca0 2023-11-28T18:40:00.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.984+0000 7feb6d8b9640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb4c0668d0 0x7feb4c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:00.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.984+0000 7feb6d8b9640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb4c0668d0 0x7feb4c068d90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb640b2000 tx=0x7feb64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:00.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:00.988+0000 7feb5e7fc640 1 -- 172.21.15.7:0/564627920 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feb681f40c0 con 0x7feb680c6ca0 2023-11-28T18:40:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb681f40c0 con 0x7feb680c6ca0 2023-11-28T18:40:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb5e7fc640 1 -- 172.21.15.7:0/564627920 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6558 (secure 0 0 0) 0x7feb681f40c0 con 0x7feb680c6ca0 2023-11-28T18:40:01.181 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:01.181 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 71 2023-11-28T18:40:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb4c0668d0 msgr2=0x7feb4c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb4c0668d0 0x7feb4c068d90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb640b2000 tx=0x7feb64008040 comp rx=0 tx=0).stop 2023-11-28T18:40:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c6ca0 msgr2=0x7feb68147a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c6ca0 0x7feb68147a40 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7feb5800c520 tx=0x7feb5800c9e0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 shutdown_connections 2023-11-28T18:40:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb4c0668d0 0x7feb4c068d90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb6815e300 0x7feb68147f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb680c6ca0 0x7feb68147a40 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 --2- 172.21.15.7:0/564627920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb680c68c0 0x7feb68161120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 >> 172.21.15.7:0/564627920 conn(0x7feb680662e0 msgr2=0x7feb680c81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 shutdown_connections 2023-11-28T18:40:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.180+0000 7feb6fb1d640 1 -- 172.21.15.7:0/564627920 wait complete. 2023-11-28T18:40:01.202 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=2) 2023-11-28T18:40:01.203 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T18:40:01.203 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:40:01.217 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T18:40:01.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f872019d640 1 Processor -- start 2023-11-28T18:40:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f872019d640 1 -- start start 2023-11-28T18:40:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f872019d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87180cbb70 0x7f87180cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f872019d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f87180cc810 0x7f8718072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f872019d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87180ca920 con 0x7f87181542e0 2023-11-28T18:40:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f872019d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8718071d50 con 0x7f87180cc810 2023-11-28T18:40:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f872019d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8718072030 con 0x7f87180cbb70 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f871e73a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f87180cc810 0x7f8718072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f871df39640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f87181542e0 0x7f871815e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52644/0 (socket says 172.21.15.7:52644) 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f871df39640 1 -- 172.21.15.7:0/3695470668 learned_addr learned my addr 172.21.15.7:0/3695470668 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.408+0000 7f871d738640 1 --2- 172.21.15.7:0/3695470668 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87180cbb70 0x7f87180cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871cf37640 1 -- 172.21.15.7:0/3695470668 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306232307 0 0) 0x7f87180ca920 con 0x7f87181542e0 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871cf37640 1 -- 172.21.15.7:0/3695470668 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8704003530 con 0x7f87181542e0 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871e73a640 1 -- 172.21.15.7:0/3695470668 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87180cbb70 msgr2=0x7f87180cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871e73a640 1 --2- 172.21.15.7:0/3695470668 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87180cbb70 0x7f87180cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871e73a640 1 -- 172.21.15.7:0/3695470668 >> v1:172.21.15.7:6789/0 conn(0x7f87181542e0 legacy=0x7f871815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871e73a640 1 -- 172.21.15.7:0/3695470668 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87180c4cd0 con 0x7f87180cc810 2023-11-28T18:40:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871e73a640 1 --2- 172.21.15.7:0/3695470668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cc810 0x7f8718072450 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8710009d90 tx=0x7f871001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e059d399d1426f22 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871cf37640 1 -- 172.21.15.7:0/3695470668 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87100a5920 con 0x7f87180cc810 2023-11-28T18:40:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871cf37640 1 -- 172.21.15.7:0/3695470668 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87100a5a80 con 0x7f87180cc810 2023-11-28T18:40:01.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.412+0000 7f871cf37640 1 -- 172.21.15.7:0/3695470668 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87100a5df0 con 0x7f87180cc810 2023-11-28T18:40:01.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- 172.21.15.7:0/3695470668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cc810 msgr2=0x7f8718072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:01.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 --2- 172.21.15.7:0/3695470668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cc810 0x7f8718072450 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8710009d90 tx=0x7f871001c950 comp rx=0 tx=0).stop 2023-11-28T18:40:01.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- 172.21.15.7:0/3695470668 shutdown_connections 2023-11-28T18:40:01.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 --2- 172.21.15.7:0/3695470668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cc810 0x7f8718072450 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 --2- 172.21.15.7:0/3695470668 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87180cbb70 0x7f87180cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- 172.21.15.7:0/3695470668 >> 172.21.15.7:0/3695470668 conn(0x7f87180717e0 msgr2=0x7f87180662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:01.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- 172.21.15.7:0/3695470668 shutdown_connections 2023-11-28T18:40:01.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- 172.21.15.7:0/3695470668 wait complete. 2023-11-28T18:40:01.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 Processor -- start 2023-11-28T18:40:01.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- start start 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cbb70 0x7f8718156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87180cc810 0x7f87180cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87181542e0 0x7f87180c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87180c8e80 con 0x7f87180cbb70 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8718157680 con 0x7f87180cc810 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f872019d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180c7fb0 con 0x7f87181542e0 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cbb70 0x7f8718156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cbb70 0x7f8718156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50328/0 (socket says 172.21.15.7:50328) 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 -- 172.21.15.7:0/478465240 learned_addr learned my addr 172.21.15.7:0/478465240 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 -- 172.21.15.7:0/478465240 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87181542e0 msgr2=0x7f87180c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:01.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87181542e0 0x7f87180c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 -- 172.21.15.7:0/478465240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87180cc810 msgr2=0x7f87180cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:01.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871d738640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87180cc810 0x7f87180cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:01.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87180cc810 0x7f87180cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 -- 172.21.15.7:0/478465240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8710009a70 con 0x7f87180cbb70 2023-11-28T18:40:01.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871d738640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87180cc810 0x7f87180cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:40:01.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.416+0000 7f871df39640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cbb70 0x7f8718156000 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f871400c3c0 tx=0x7f871400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:01.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f870e7fc640 1 -- 172.21.15.7:0/478465240 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714005cf0 con 0x7f87180cbb70 2023-11-28T18:40:01.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f870e7fc640 1 -- 172.21.15.7:0/478465240 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8714005e50 con 0x7f87180cbb70 2023-11-28T18:40:01.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f870e7fc640 1 -- 172.21.15.7:0/478465240 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87140072f0 con 0x7f87180cbb70 2023-11-28T18:40:01.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87180c8200 con 0x7f87180cbb70 2023-11-28T18:40:01.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87180c87c0 con 0x7f87180cbb70 2023-11-28T18:40:01.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871806c670 con 0x7f87180cbb70 2023-11-28T18:40:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f870e7fc640 1 -- 172.21.15.7:0/478465240 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8714007450 con 0x7f87180cbb70 2023-11-28T18:40:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f870e7fc640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86fc0668d0 0x7f86fc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f870e7fc640 1 -- 172.21.15.7:0/478465240 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f8714083cc0 con 0x7f87180cbb70 2023-11-28T18:40:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.420+0000 7f871d738640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86fc0668d0 0x7f86fc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.424+0000 7f871d738640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86fc0668d0 0x7f86fc068d90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f87180b4f60 tx=0x7f8708008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.424+0000 7f870e7fc640 1 -- 172.21.15.7:0/478465240 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8714050c60 con 0x7f87180cbb70 2023-11-28T18:40:01.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87180c8fc0 con 0x7f87180cbb70 2023-11-28T18:40:01.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f870e7fc640 1 -- 172.21.15.7:0/478465240 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6558 (secure 0 0 0) 0x7f8714017070 con 0x7f87180cbb70 2023-11-28T18:40:01.622 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:01.622 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:01.622 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 71 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86fc0668d0 msgr2=0x7f86fc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86fc0668d0 0x7f86fc068d90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f87180b4f60 tx=0x7f8708008040 comp rx=0 tx=0).stop 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cbb70 msgr2=0x7f8718156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cbb70 0x7f8718156000 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f871400c3c0 tx=0x7f871400c880 comp rx=0 tx=0).stop 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 shutdown_connections 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86fc0668d0 0x7f86fc068d90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f87181542e0 0x7f87180c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87180cc810 0x7f87180cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 --2- 172.21.15.7:0/478465240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87180cbb70 0x7f8718156000 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:01.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 >> 172.21.15.7:0/478465240 conn(0x7f87180717e0 msgr2=0x7f871815a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:01.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 shutdown_connections 2023-11-28T18:40:01.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:01.620+0000 7f872019d640 1 -- 172.21.15.7:0/478465240 wait complete. 2023-11-28T18:40:01.643 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:40:02.630 INFO:tasks.workunit.client.0.smithi007.stdout:Syncing()...1 sec 2023-11-28T18:40:02.633 INFO:tasks.workunit.client.0.smithi007.stdout:Starting Actual Benchmark At: Tue Nov 28 18:40:02 2023 2023-11-28T18:40:02.633 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:40:03.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:40:03.186 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:40:03.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:03.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.868+0000 7f1040c5a640 1 Processor -- start 2023-11-28T18:40:03.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.868+0000 7f1040c5a640 1 -- start start 2023-11-28T18:40:03.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f1040c5a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c072450 0x7f103c0728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:03.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f1040c5a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c0ca920 con 0x7f103c0cbb70 2023-11-28T18:40:03.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f1040c5a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c0c4cd0 con 0x7f103c1542e0 2023-11-28T18:40:03.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f1040c5a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103c071d50 con 0x7f103c072450 2023-11-28T18:40:03.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103a59c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f103c1542e0 0x7f103c15e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:60130/0 (socket says 172.21.15.7:60130) 2023-11-28T18:40:03.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103a59c640 1 -- 172.21.15.7:0/1324667853 learned_addr learned my addr 172.21.15.7:0/1324667853 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:03.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103ad9d640 1 --2- 172.21.15.7:0/1324667853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c072450 0x7f103c0728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:03.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103959a640 1 -- 172.21.15.7:0/1324667853 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738422041 0 0) 0x7f103c0ca920 con 0x7f103c0cbb70 2023-11-28T18:40:03.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103959a640 1 -- 172.21.15.7:0/1324667853 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1028003530 con 0x7f103c0cbb70 2023-11-28T18:40:03.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103959a640 1 -- 172.21.15.7:0/1324667853 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254620791 0 0) 0x7f103c0c4cd0 con 0x7f103c1542e0 2023-11-28T18:40:03.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103959a640 1 -- 172.21.15.7:0/1324667853 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c0ca920 con 0x7f103c1542e0 2023-11-28T18:40:03.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103ad9d640 1 -- 172.21.15.7:0/1324667853 >> v1:172.21.15.193:6789/0 conn(0x7f103c1542e0 legacy=0x7f103c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:03.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103ad9d640 1 -- 172.21.15.7:0/1324667853 >> v1:172.21.15.7:6789/0 conn(0x7f103c0cbb70 legacy=0x7f103c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:03.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.872+0000 7f103ad9d640 1 -- 172.21.15.7:0/1324667853 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c0cab70 con 0x7f103c072450 2023-11-28T18:40:03.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f103ad9d640 1 --2- 172.21.15.7:0/1324667853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c072450 0x7f103c0728c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f102c0097f0 tx=0x7f102c01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4542dcda3c58b2a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:03.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f103959a640 1 -- 172.21.15.7:0/1324667853 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c0ac630 con 0x7f103c072450 2023-11-28T18:40:03.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f103959a640 1 -- 172.21.15.7:0/1324667853 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c0ac790 con 0x7f103c072450 2023-11-28T18:40:03.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f103959a640 1 -- 172.21.15.7:0/1324667853 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c0acb20 con 0x7f103c072450 2023-11-28T18:40:03.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 -- 172.21.15.7:0/1324667853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c072450 msgr2=0x7f103c0728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:03.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 --2- 172.21.15.7:0/1324667853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c072450 0x7f103c0728c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f102c0097f0 tx=0x7f102c01c4c0 comp rx=0 tx=0).stop 2023-11-28T18:40:03.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 -- 172.21.15.7:0/1324667853 shutdown_connections 2023-11-28T18:40:03.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 --2- 172.21.15.7:0/1324667853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c072450 0x7f103c0728c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:03.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 -- 172.21.15.7:0/1324667853 >> 172.21.15.7:0/1324667853 conn(0x7f103c0717e0 msgr2=0x7f103c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:03.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 -- 172.21.15.7:0/1324667853 shutdown_connections 2023-11-28T18:40:03.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 -- 172.21.15.7:0/1324667853 wait complete. 2023-11-28T18:40:03.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 Processor -- start 2023-11-28T18:40:03.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.880+0000 7f1040c5a640 1 -- start start 2023-11-28T18:40:03.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1040c5a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f103c072450 0x7f103c157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:03.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1040c5a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c0cbb70 0x7f103c1ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:03.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1040c5a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c1542e0 0x7f103c0b1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:03.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1040c5a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c072030 con 0x7f103c1542e0 2023-11-28T18:40:03.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1040c5a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c1579c0 con 0x7f103c072450 2023-11-28T18:40:03.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1040c5a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103c0c8660 con 0x7f103c0cbb70 2023-11-28T18:40:03.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f103c072450 0x7f103c157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:03.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1039d9b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c0cbb70 0x7f103c1ef3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:03.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c1542e0 0x7f103c0b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:03.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f103c072450 0x7f103c157480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:49714/0 (socket says 172.21.15.7:49714) 2023-11-28T18:40:03.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 -- 172.21.15.7:0/716907319 learned_addr learned my addr 172.21.15.7:0/716907319 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:03.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c1542e0 0x7f103c0b1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33192/0 (socket says 172.21.15.7:33192) 2023-11-28T18:40:03.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1039d9b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c0cbb70 0x7f103c1ef3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:49284/0 (socket says 172.21.15.7:49284) 2023-11-28T18:40:03.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 -- 172.21.15.7:0/716907319 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c0cbb70 msgr2=0x7f103c1ef3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:03.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c0cbb70 0x7f103c1ef3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:03.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 -- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c1542e0 msgr2=0x7f103c0b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:03.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c1542e0 0x7f103c0b1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:03.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 -- 172.21.15.7:0/716907319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c009420 con 0x7f103c072450 2023-11-28T18:40:03.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1039d9b640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c0cbb70 0x7f103c1ef3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:40:03.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f103a59c640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f103c072450 0x7f103c157480 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f103400cbc0 tx=0x7f103400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:03.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1032ffd640 1 -- 172.21.15.7:0/716907319 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103400ad40 con 0x7f103c072450 2023-11-28T18:40:03.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1032ffd640 1 -- 172.21.15.7:0/716907319 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1034012400 con 0x7f103c072450 2023-11-28T18:40:03.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1032ffd640 1 -- 172.21.15.7:0/716907319 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103400aea0 con 0x7f103c072450 2023-11-28T18:40:03.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103c159fc0 con 0x7f103c072450 2023-11-28T18:40:03.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.884+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c0c7a50 con 0x7f103c072450 2023-11-28T18:40:03.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.888+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c06c670 con 0x7f103c072450 2023-11-28T18:40:03.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.888+0000 7f1032ffd640 1 -- 172.21.15.7:0/716907319 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1034007070 con 0x7f103c072450 2023-11-28T18:40:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.892+0000 7f1032ffd640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f101c066920 0x7f101c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.892+0000 7f1032ffd640 1 -- 172.21.15.7:0/716907319 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f10340846d0 con 0x7f103c072450 2023-11-28T18:40:03.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.892+0000 7f1039d9b640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f101c066920 0x7f101c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:03.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.892+0000 7f1039d9b640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f101c066920 0x7f101c068de0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f103c1523a0 tx=0x7f1024009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:03.896+0000 7f1032ffd640 1 -- 172.21.15.7:0/716907319 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f10340516e0 con 0x7f103c072450 2023-11-28T18:40:04.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.244+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f103c04e890 con 0x7f103c072450 2023-11-28T18:40:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:04.247 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:04.247 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 71 2023-11-28T18:40:04.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.244+0000 7f1032ffd640 1 -- 172.21.15.7:0/716907319 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6558 (secure 0 0 0) 0x7f1034054d00 con 0x7f103c072450 2023-11-28T18:40:04.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f101c066920 msgr2=0x7f101c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:04.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f101c066920 0x7f101c068de0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f103c1523a0 tx=0x7f1024009040 comp rx=0 tx=0).stop 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f103c072450 msgr2=0x7f103c157480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f103c072450 0x7f103c157480 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f103400cbc0 tx=0x7f103400a430 comp rx=0 tx=0).stop 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 shutdown_connections 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f101c066920 0x7f101c068de0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c1542e0 0x7f103c0b1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f103c0cbb70 0x7f103c1ef3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 --2- 172.21.15.7:0/716907319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f103c072450 0x7f103c157480 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 >> 172.21.15.7:0/716907319 conn(0x7f103c0717e0 msgr2=0x7f103c0b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 shutdown_connections 2023-11-28T18:40:04.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:04.248+0000 7f1040c5a640 1 -- 172.21.15.7:0/716907319 wait complete. 2023-11-28T18:40:04.282 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:40:06.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.491+0000 7f7ada372640 1 Processor -- start 2023-11-28T18:40:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.491+0000 7f7ada372640 1 -- start start 2023-11-28T18:40:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ada372640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ad41542e0 0x7f7ad415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ada372640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7ad4072450 0x7f7ad4072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ada372640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad40ca920 con 0x7f7ad40cbb70 2023-11-28T18:40:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ada372640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad4071d50 con 0x7f7ad41542e0 2023-11-28T18:40:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ada372640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad4072030 con 0x7f7ad4072450 2023-11-28T18:40:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7ad40cbb70 0x7f7ad40cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:60132/0 (socket says 172.21.15.7:60132) 2023-11-28T18:40:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad37fe640 1 -- 172.21.15.7:0/2438849014 learned_addr learned my addr 172.21.15.7:0/2438849014 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad3fff640 1 --2- 172.21.15.7:0/2438849014 >> v2:172.21.15.7:3300/0 conn(0x7f7ad41542e0 0x7f7ad415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847951466 0 0) 0x7f7ad40ca920 con 0x7f7ad40cbb70 2023-11-28T18:40:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7abc003530 con 0x7f7ad40cbb70 2023-11-28T18:40:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad890f640 1 --2- 172.21.15.7:0/2438849014 >> v2:172.21.15.193:3300/0 conn(0x7f7ad4072450 0x7f7ad4072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3979422489 0 0) 0x7f7abc003530 con 0x7f7ad40cbb70 2023-11-28T18:40:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 >> v2:172.21.15.193:3300/0 conn(0x7f7ad4072450 msgr2=0x7f7ad4072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 --2- 172.21.15.7:0/2438849014 >> v2:172.21.15.193:3300/0 conn(0x7f7ad4072450 0x7f7ad4072870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 >> v2:172.21.15.7:3300/0 conn(0x7f7ad41542e0 msgr2=0x7f7ad415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 --2- 172.21.15.7:0/2438849014 >> v2:172.21.15.7:3300/0 conn(0x7f7ad41542e0 0x7f7ad415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad415e550 con 0x7f7ad40cbb70 2023-11-28T18:40:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f7ac8001680 con 0x7f7ad40cbb70 2023-11-28T18:40:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 >> v1:172.21.15.193:6789/0 conn(0x7f7ad40cbb70 legacy=0x7f7ad40cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7abc003980 0x7f7abc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7abc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7abc005190 0x7f7abc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad4071d50 con 0x7f7abc003980 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad4072030 con 0x7f7abc0047d0 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.495+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc00c090 con 0x7f7abc005190 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.499+0000 7f7ad37fe640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7abc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.499+0000 7f7ad890f640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7abc005190 0x7f7abc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.499+0000 7f7ad37fe640 1 -- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7abc005190 msgr2=0x7f7abc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.499+0000 7f7ad37fe640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7abc005190 0x7f7abc00b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.499+0000 7f7ad37fe640 1 -- 172.21.15.7:0/2438849014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7abc003980 msgr2=0x7f7abc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.499+0000 7f7ad37fe640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7abc003980 0x7f7abc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.499+0000 7f7ad37fe640 1 -- 172.21.15.7:0/2438849014 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc003530 con 0x7f7abc0047d0 2023-11-28T18:40:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.499+0000 7f7ad890f640 1 -- 172.21.15.7:0/2438849014 reap_dead start 2023-11-28T18:40:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ad37fe640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7abc004c50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800c1c0 tx=0x7f7ac801cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7656bd8718109fa7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac80058e0 con 0x7f7abc0047d0 2023-11-28T18:40:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac8005a60 con 0x7f7abc0047d0 2023-11-28T18:40:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ad2ffd640 1 -- 172.21.15.7:0/2438849014 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8005e50 con 0x7f7abc0047d0 2023-11-28T18:40:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ada372640 1 -- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 msgr2=0x7f7abc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ada372640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7abc004c50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800c1c0 tx=0x7f7ac801cc00 comp rx=0 tx=0).stop 2023-11-28T18:40:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ada372640 1 -- 172.21.15.7:0/2438849014 shutdown_connections 2023-11-28T18:40:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ada372640 1 --2- 172.21.15.7:0/2438849014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7abc004c50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.503+0000 7f7ada372640 1 -- 172.21.15.7:0/2438849014 >> 172.21.15.7:0/2438849014 conn(0x7f7ad40717e0 msgr2=0x7f7ad40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.507+0000 7f7ada372640 1 -- 172.21.15.7:0/2438849014 shutdown_connections 2023-11-28T18:40:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.507+0000 7f7ada372640 1 -- 172.21.15.7:0/2438849014 wait complete. 2023-11-28T18:40:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.507+0000 7f7ada372640 1 Processor -- start 2023-11-28T18:40:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.507+0000 7f7ada372640 1 -- start start 2023-11-28T18:40:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.507+0000 7f7ada372640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7ad4159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.507+0000 7f7ada372640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ad4159550 0x7f7ad40cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ada372640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ad40cdf00 0x7f7ad41e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ada372640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40ce880 con 0x7f7ad4159550 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ada372640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40cf7f0 con 0x7f7abc0047d0 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ada372640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad4158830 con 0x7f7ad40cdf00 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ad40cdf00 0x7f7ad41e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ad40cdf00 0x7f7ad41e78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:49310/0 (socket says 172.21.15.7:49310) 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 -- 172.21.15.7:0/3167274616 learned_addr learned my addr 172.21.15.7:0/3167274616 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad3fff640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7ad4159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 -- 172.21.15.7:0/3167274616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 msgr2=0x7f7ad4159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7ad4159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 -- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ad4159550 msgr2=0x7f7ad40cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad37fe640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ad4159550 0x7f7ad40cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ad4159550 0x7f7ad40cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 -- 172.21.15.7:0/3167274616 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad4159990 con 0x7f7ad40cdf00 2023-11-28T18:40:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad37fe640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ad4159550 0x7f7ad40cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:40:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad890f640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ad40cdf00 0x7f7ad41e78b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4071cf0 tx=0x7f7acc009d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad0ff9640 1 -- 172.21.15.7:0/3167274616 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc007cf0 con 0x7f7ad40cdf00 2023-11-28T18:40:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad0ff9640 1 -- 172.21.15.7:0/3167274616 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7acc007e70 con 0x7f7ad40cdf00 2023-11-28T18:40:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ad0ff9640 1 -- 172.21.15.7:0/3167274616 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc014620 con 0x7f7ad40cdf00 2023-11-28T18:40:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad40cffd0 con 0x7f7ad40cdf00 2023-11-28T18:40:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.511+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad40d0590 con 0x7f7ad40cdf00 2023-11-28T18:40:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.515+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad406c670 con 0x7f7ad40cdf00 2023-11-28T18:40:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.515+0000 7f7ad0ff9640 1 -- 172.21.15.7:0/3167274616 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7acc01d3b0 con 0x7f7ad40cdf00 2023-11-28T18:40:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.519+0000 7f7ad0ff9640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7ac00669a0 0x7f7ac0068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.519+0000 7f7ad0ff9640 1 -- 172.21.15.7:0/3167274616 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f7acc084400 con 0x7f7ad40cdf00 2023-11-28T18:40:06.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.519+0000 7f7ad3fff640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7ac00669a0 0x7f7ac0068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:06.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.523+0000 7f7ad3fff640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7ac00669a0 0x7f7ac0068e60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4006e90 tx=0x7f7ac4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:06.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.527+0000 7f7ad0ff9640 1 -- 172.21.15.7:0/3167274616 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7acc051310 con 0x7f7ad40cdf00 2023-11-28T18:40:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.867+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ad404e890 con 0x7f7ad40cdf00 2023-11-28T18:40:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.867+0000 7f7ad0ff9640 1 -- 172.21.15.7:0/3167274616 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6558 (secure 0 0 0) 0x7f7acc054930 con 0x7f7ad40cdf00 2023-11-28T18:40:06.872 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:06.872 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:06.872 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 71 2023-11-28T18:40:06.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7ac00669a0 msgr2=0x7f7ac0068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7ac00669a0 0x7f7ac0068e60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4006e90 tx=0x7f7ac4008040 comp rx=0 tx=0).stop 2023-11-28T18:40:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ad40cdf00 msgr2=0x7f7ad41e78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ad40cdf00 0x7f7ad41e78b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4071cf0 tx=0x7f7acc009d40 comp rx=0 tx=0).stop 2023-11-28T18:40:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 shutdown_connections 2023-11-28T18:40:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7ac00669a0 0x7f7ac0068e60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7ad40cdf00 0x7f7ad41e78b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ad4159550 0x7f7ad40cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 --2- 172.21.15.7:0/3167274616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7abc0047d0 0x7f7ad4159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.871+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 >> 172.21.15.7:0/3167274616 conn(0x7f7ad40717e0 msgr2=0x7f7ad415be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.875+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 shutdown_connections 2023-11-28T18:40:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:06.875+0000 7f7ada372640 1 -- 172.21.15.7:0/3167274616 wait complete. 2023-11-28T18:40:06.912 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:40:08.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.143+0000 7f8e00632640 1 Processor -- start 2023-11-28T18:40:09.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.143+0000 7f8e00632640 1 -- start start 2023-11-28T18:40:09.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8e00632640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8df80cbb70 0x7f8df80cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:09.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8e00632640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df80ca920 con 0x7f8df81542e0 2023-11-28T18:40:09.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8e00632640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df80c4cd0 con 0x7f8df80cc810 2023-11-28T18:40:09.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8e00632640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df8071d50 con 0x7f8df80cbb70 2023-11-28T18:40:09.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8dfe3ce640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8df81542e0 0x7f8df815e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60160/0 (socket says 172.21.15.7:60160) 2023-11-28T18:40:09.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8dfe3ce640 1 -- 172.21.15.7:0/3588493236 learned_addr learned my addr 172.21.15.7:0/3588493236 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:09.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8dfdbcd640 1 --2- 172.21.15.7:0/3588493236 >> v2:172.21.15.193:3300/0 conn(0x7f8df80cbb70 0x7f8df80cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:09.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8dfdbcd640 1 -- 172.21.15.7:0/3588493236 >> v1:172.21.15.193:6789/0 conn(0x7f8df80cc810 legacy=0x7f8df8072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:09.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8dfdbcd640 1 -- 172.21.15.7:0/3588493236 >> v1:172.21.15.7:6789/0 conn(0x7f8df81542e0 legacy=0x7f8df815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:09.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.147+0000 7f8dfdbcd640 1 -- 172.21.15.7:0/3588493236 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80cab70 con 0x7f8df80cbb70 2023-11-28T18:40:09.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8dfdbcd640 1 --2- 172.21.15.7:0/3588493236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df80cc2d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8de80097c0 tx=0x7f8de801c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=890c5588ed582cb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:09.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8dfd3cc640 1 -- 172.21.15.7:0/3588493236 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80ab600 con 0x7f8df80cbb70 2023-11-28T18:40:09.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8dfd3cc640 1 -- 172.21.15.7:0/3588493236 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de80a7050 con 0x7f8df80cbb70 2023-11-28T18:40:09.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8dfd3cc640 1 -- 172.21.15.7:0/3588493236 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80ab980 con 0x7f8df80cbb70 2023-11-28T18:40:09.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8e00632640 1 -- 172.21.15.7:0/3588493236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 msgr2=0x7f8df80cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:09.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8e00632640 1 --2- 172.21.15.7:0/3588493236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df80cc2d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8de80097c0 tx=0x7f8de801c2d0 comp rx=0 tx=0).stop 2023-11-28T18:40:09.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8e00632640 1 -- 172.21.15.7:0/3588493236 shutdown_connections 2023-11-28T18:40:09.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8e00632640 1 --2- 172.21.15.7:0/3588493236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df80cc2d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:09.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.151+0000 7f8e00632640 1 -- 172.21.15.7:0/3588493236 >> 172.21.15.7:0/3588493236 conn(0x7f8df80717e0 msgr2=0x7f8df80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:09.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.155+0000 7f8e00632640 1 -- 172.21.15.7:0/3588493236 shutdown_connections 2023-11-28T18:40:09.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.159+0000 7f8e00632640 1 -- 172.21.15.7:0/3588493236 wait complete. 2023-11-28T18:40:09.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.159+0000 7f8e00632640 1 Processor -- start 2023-11-28T18:40:09.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 -- start start 2023-11-28T18:40:09.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df81f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:09.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df80cc810 0x7f8df81fd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:09.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8df81542e0 0x7f8df80c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:09.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8072030 con 0x7f8df80cc810 2023-11-28T18:40:09.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80c70d0 con 0x7f8df80cbb70 2023-11-28T18:40:09.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80c6bc0 con 0x7f8df81542e0 2023-11-28T18:40:09.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df81f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:09.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df81f02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:49762/0 (socket says 172.21.15.7:49762) 2023-11-28T18:40:09.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 -- 172.21.15.7:0/3731542399 learned_addr learned my addr 172.21.15.7:0/3731542399 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:09.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 -- 172.21.15.7:0/3731542399 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8df81542e0 msgr2=0x7f8df80c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:09.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8df81542e0 0x7f8df80c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:09.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 -- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df80cc810 msgr2=0x7f8df81fd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:09.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df80cc810 0x7f8df81fd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:09.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 -- 172.21.15.7:0/3731542399 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de8009420 con 0x7f8df80cbb70 2023-11-28T18:40:09.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8dfe3ce640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df81f02b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009e80 tx=0x7f8df400c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:09.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8deeffd640 1 -- 172.21.15.7:0/3731542399 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df40078f0 con 0x7f8df80cbb70 2023-11-28T18:40:09.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8deeffd640 1 -- 172.21.15.7:0/3731542399 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df4007a50 con 0x7f8df80cbb70 2023-11-28T18:40:09.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8deeffd640 1 -- 172.21.15.7:0/3731542399 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4007de0 con 0x7f8df80cbb70 2023-11-28T18:40:09.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df81fcb80 con 0x7f8df80cbb70 2023-11-28T18:40:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.167+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df80c6df0 con 0x7f8df80cbb70 2023-11-28T18:40:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.171+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df806c670 con 0x7f8df80cbb70 2023-11-28T18:40:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.171+0000 7f8deeffd640 1 -- 172.21.15.7:0/3731542399 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8df4009070 con 0x7f8df80cbb70 2023-11-28T18:40:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.171+0000 7f8deeffd640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ddc0668d0 0x7f8ddc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.171+0000 7f8deeffd640 1 -- 172.21.15.7:0/3731542399 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f8df4086620 con 0x7f8df80cbb70 2023-11-28T18:40:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.175+0000 7f8dfdbcd640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ddc0668d0 0x7f8ddc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.175+0000 7f8dfdbcd640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ddc0668d0 0x7f8ddc068d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8de80ad000 tx=0x7f8de8004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:09.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.175+0000 7f8deeffd640 1 -- 172.21.15.7:0/3731542399 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8df40535c0 con 0x7f8df80cbb70 2023-11-28T18:40:09.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.683+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8df804e890 con 0x7f8df80cbb70 2023-11-28T18:40:09.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.683+0000 7f8deeffd640 1 -- 172.21.15.7:0/3731542399 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6558 (secure 0 0 0) 0x7f8df4056be0 con 0x7f8df80cbb70 2023-11-28T18:40:09.687 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:09.687 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:09.688 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 71 2023-11-28T18:40:09.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ddc0668d0 msgr2=0x7f8ddc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:09.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ddc0668d0 0x7f8ddc068d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8de80ad000 tx=0x7f8de8004740 comp rx=0 tx=0).stop 2023-11-28T18:40:09.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 msgr2=0x7f8df81f02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:09.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df81f02b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009e80 tx=0x7f8df400c430 comp rx=0 tx=0).stop 2023-11-28T18:40:09.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 shutdown_connections 2023-11-28T18:40:09.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ddc0668d0 0x7f8ddc068d90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:09.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8df81542e0 0x7f8df80c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:09.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df80cc810 0x7f8df81fd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:09.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 --2- 172.21.15.7:0/3731542399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df80cbb70 0x7f8df81f02b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:09.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.687+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 >> 172.21.15.7:0/3731542399 conn(0x7f8df80717e0 msgr2=0x7f8df80cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:09.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.691+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 shutdown_connections 2023-11-28T18:40:09.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:09.691+0000 7f8e00632640 1 -- 172.21.15.7:0/3731542399 wait complete. 2023-11-28T18:40:09.723 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:40:11.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:11.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.971+0000 7f0e3ea32640 1 Processor -- start 2023-11-28T18:40:11.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.971+0000 7f0e3ea32640 1 -- start start 2023-11-28T18:40:11.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e3ea32640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0e381542e0 0x7f0e3815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e3ea32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 0x7f0e380cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e3ea32640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e380ca920 con 0x7f0e380cc810 2023-11-28T18:40:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e3ea32640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e38071d50 con 0x7f0e381542e0 2023-11-28T18:40:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e3ea32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e38072030 con 0x7f0e380cbb70 2023-11-28T18:40:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e37fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0e381542e0 0x7f0e3815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 0x7f0e380cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e37fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0e381542e0 0x7f0e3815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33262/0 (socket says 172.21.15.7:33262) 2023-11-28T18:40:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e37fff640 1 -- 172.21.15.7:0/1903907538 learned_addr learned my addr 172.21.15.7:0/1903907538 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e37fff640 1 -- 172.21.15.7:0/1903907538 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 msgr2=0x7f0e380cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e37fff640 1 --2- 172.21.15.7:0/1903907538 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 0x7f0e380cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e37fff640 1 -- 172.21.15.7:0/1903907538 >> v1:172.21.15.7:6789/0 conn(0x7f0e380cc810 legacy=0x7f0e38072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.975+0000 7f0e37fff640 1 -- 172.21.15.7:0/1903907538 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3815e550 con 0x7f0e381542e0 2023-11-28T18:40:11.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.995+0000 7f0e37fff640 1 --2- 172.21.15.7:0/1903907538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 0x7f0e3815e010 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009870 tx=0x7f0e2801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ea3b7ed42a988c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:12.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.995+0000 7f0e36ffd640 1 -- 172.21.15.7:0/1903907538 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280aa620 con 0x7f0e381542e0 2023-11-28T18:40:12.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.995+0000 7f0e36ffd640 1 -- 172.21.15.7:0/1903907538 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e280aa780 con 0x7f0e381542e0 2023-11-28T18:40:12.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.995+0000 7f0e36ffd640 1 -- 172.21.15.7:0/1903907538 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280aab10 con 0x7f0e381542e0 2023-11-28T18:40:12.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.995+0000 7f0e3ea32640 1 -- 172.21.15.7:0/1903907538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 msgr2=0x7f0e3815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:12.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.995+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/1903907538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 0x7f0e3815e010 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009870 tx=0x7f0e2801c520 comp rx=0 tx=0).stop 2023-11-28T18:40:12.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.999+0000 7f0e3ea32640 1 -- 172.21.15.7:0/1903907538 shutdown_connections 2023-11-28T18:40:12.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.999+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/1903907538 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 0x7f0e380cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:12.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.999+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/1903907538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 0x7f0e3815e010 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:12.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.999+0000 7f0e3ea32640 1 -- 172.21.15.7:0/1903907538 >> 172.21.15.7:0/1903907538 conn(0x7f0e380717e0 msgr2=0x7f0e380662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:12.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.999+0000 7f0e3ea32640 1 -- 172.21.15.7:0/1903907538 shutdown_connections 2023-11-28T18:40:12.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.999+0000 7f0e3ea32640 1 -- 172.21.15.7:0/1903907538 wait complete. 2023-11-28T18:40:12.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.999+0000 7f0e3ea32640 1 Processor -- start 2023-11-28T18:40:12.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:11.999+0000 7f0e3ea32640 1 -- start start 2023-11-28T18:40:12.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e3ea32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 0x7f0e3815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e3ea32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e380cc810 0x7f0e380cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e3ea32640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 0x7f0e38155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e3ea32640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e38154a80 con 0x7f0e381542e0 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e3ea32640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e38154d60 con 0x7f0e380cc810 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e3ea32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e38155040 con 0x7f0e380cbb70 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e3cfcf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 0x7f0e38155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e380cc810 0x7f0e380cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e380cc810 0x7f0e380cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:49764/0 (socket says 172.21.15.7:49764) 2023-11-28T18:40:12.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e377fe640 1 -- 172.21.15.7:0/791098457 learned_addr learned my addr 172.21.15.7:0/791098457 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:12.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e37fff640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 0x7f0e3815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:12.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e377fe640 1 -- 172.21.15.7:0/791098457 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 msgr2=0x7f0e3815ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:12.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e377fe640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 0x7f0e3815ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:12.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e377fe640 1 -- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 msgr2=0x7f0e38155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:12.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e377fe640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 0x7f0e38155e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:12.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.003+0000 7f0e377fe640 1 -- 172.21.15.7:0/791098457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280094d0 con 0x7f0e380cc810 2023-11-28T18:40:12.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.007+0000 7f0e377fe640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e380cc810 0x7f0e380cfad0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00e870 tx=0x7f0e2c00ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:12.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.007+0000 7f0e34ff9640 1 -- 172.21.15.7:0/791098457 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c009510 con 0x7f0e380cc810 2023-11-28T18:40:12.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.007+0000 7f0e34ff9640 1 -- 172.21.15.7:0/791098457 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c00eec0 con 0x7f0e380cc810 2023-11-28T18:40:12.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.007+0000 7f0e34ff9640 1 -- 172.21.15.7:0/791098457 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00cb40 con 0x7f0e380cc810 2023-11-28T18:40:12.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.007+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e38155320 con 0x7f0e380cc810 2023-11-28T18:40:12.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.007+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e381558e0 con 0x7f0e380cc810 2023-11-28T18:40:12.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.007+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3806c670 con 0x7f0e380cc810 2023-11-28T18:40:12.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.011+0000 7f0e34ff9640 1 -- 172.21.15.7:0/791098457 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0e2c00cca0 con 0x7f0e380cc810 2023-11-28T18:40:12.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.011+0000 7f0e34ff9640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0e24066900 0x7f0e24068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:12.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.011+0000 7f0e34ff9640 1 -- 172.21.15.7:0/791098457 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f0e2c0848a0 con 0x7f0e380cc810 2023-11-28T18:40:12.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.015+0000 7f0e37fff640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0e24066900 0x7f0e24068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:12.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.015+0000 7f0e34ff9640 1 -- 172.21.15.7:0/791098457 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0e2c0517d0 con 0x7f0e380cc810 2023-11-28T18:40:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.019+0000 7f0e37fff640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0e24066900 0x7f0e24068dc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0e280ae000 tx=0x7f0e280b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:12.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.471+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e38155a70 con 0x7f0e380cc810 2023-11-28T18:40:12.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.475+0000 7f0e34ff9640 1 -- 172.21.15.7:0/791098457 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6558 (secure 0 0 0) 0x7f0e2c017090 con 0x7f0e380cc810 2023-11-28T18:40:12.480 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:12.480 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:12.480 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 71 2023-11-28T18:40:12.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0e24066900 msgr2=0x7f0e24068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:12.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0e24066900 0x7f0e24068dc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0e280ae000 tx=0x7f0e280b5040 comp rx=0 tx=0).stop 2023-11-28T18:40:12.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e380cc810 msgr2=0x7f0e380cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:12.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e380cc810 0x7f0e380cfad0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00e870 tx=0x7f0e2c00ec70 comp rx=0 tx=0).stop 2023-11-28T18:40:12.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 shutdown_connections 2023-11-28T18:40:12.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0e24066900 0x7f0e24068dc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:12.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e381542e0 0x7f0e38155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:12.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e380cc810 0x7f0e380cfad0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:12.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 --2- 172.21.15.7:0/791098457 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e380cbb70 0x7f0e3815ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:12.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.479+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 >> 172.21.15.7:0/791098457 conn(0x7f0e380717e0 msgr2=0x7f0e3815b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:12.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.483+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 shutdown_connections 2023-11-28T18:40:12.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:12.483+0000 7f0e3ea32640 1 -- 172.21.15.7:0/791098457 wait complete. 2023-11-28T18:40:12.522 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:40:14.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:14.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.827+0000 7f3e7249a640 1 Processor -- start 2023-11-28T18:40:14.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.831+0000 7f3e7249a640 1 -- start start 2023-11-28T18:40:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.831+0000 7f3e7249a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3e6c154300 0x7f3e6c159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.831+0000 7f3e7249a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c15dc50 0x7f3e6c0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.831+0000 7f3e6bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3e6c154300 0x7f3e6c159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.831+0000 7f3e6bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3e6c154300 0x7f3e6c159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:40512/0 (socket says 172.21.15.7:40512) 2023-11-28T18:40:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6b7fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c15dc50 0x7f3e6c0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6b7fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c15dc50 0x7f3e6c0cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:34062/0 (socket says 172.21.15.7:34062) 2023-11-28T18:40:14.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e70a37640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3e6c15e090 0x7f3e6c0ca410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58454/0 (socket says 172.21.15.7:58454) 2023-11-28T18:40:14.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6bfff640 1 -- 172.21.15.7:0/2250979216 learned_addr learned my addr 172.21.15.7:0/2250979216 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:14.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e7249a640 1 -- 172.21.15.7:0/2250979216 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c15d490 con 0x7f3e6c15e090 2023-11-28T18:40:14.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e7249a640 1 -- 172.21.15.7:0/2250979216 --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6c15a480 con 0x7f3e6c154300 2023-11-28T18:40:14.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e7249a640 1 -- 172.21.15.7:0/2250979216 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c15a760 con 0x7f3e6c15dc50 2023-11-28T18:40:14.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6affd640 1 -- 172.21.15.7:0/2250979216 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017115097 0 0) 0x7f3e6c15d490 con 0x7f3e6c15e090 2023-11-28T18:40:14.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6affd640 1 -- 172.21.15.7:0/2250979216 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e54003530 con 0x7f3e6c15e090 2023-11-28T18:40:14.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6bfff640 1 -- 172.21.15.7:0/2250979216 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c15dc50 msgr2=0x7f3e6c0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:14.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6bfff640 1 --2- 172.21.15.7:0/2250979216 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c15dc50 0x7f3e6c0cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:14.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6bfff640 1 -- 172.21.15.7:0/2250979216 >> v1:172.21.15.7:6789/0 conn(0x7f3e6c15e090 legacy=0x7f3e6c0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:14.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.835+0000 7f3e6bfff640 1 -- 172.21.15.7:0/2250979216 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c15aa40 con 0x7f3e6c154300 2023-11-28T18:40:14.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.843+0000 7f3e6bfff640 1 --2- 172.21.15.7:0/2250979216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c154300 0x7f3e6c159e40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c009870 tx=0x7f3e5c01ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82442e508b6ae5ef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:14.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e6affd640 1 -- 172.21.15.7:0/2250979216 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c0ad640 con 0x7f3e6c154300 2023-11-28T18:40:14.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e6affd640 1 -- 172.21.15.7:0/2250979216 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c0ad7a0 con 0x7f3e6c154300 2023-11-28T18:40:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e7249a640 1 -- 172.21.15.7:0/2250979216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c154300 msgr2=0x7f3e6c159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e6affd640 1 -- 172.21.15.7:0/2250979216 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c0ad640 con 0x7f3e6c154300 2023-11-28T18:40:14.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e7249a640 1 --2- 172.21.15.7:0/2250979216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c154300 0x7f3e6c159e40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c009870 tx=0x7f3e5c01ceb0 comp rx=0 tx=0).stop 2023-11-28T18:40:14.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e7249a640 1 -- 172.21.15.7:0/2250979216 shutdown_connections 2023-11-28T18:40:14.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e7249a640 1 --2- 172.21.15.7:0/2250979216 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c15dc50 0x7f3e6c0cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:14.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e7249a640 1 --2- 172.21.15.7:0/2250979216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c154300 0x7f3e6c159e40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:14.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e7249a640 1 -- 172.21.15.7:0/2250979216 >> 172.21.15.7:0/2250979216 conn(0x7f3e6c0662e0 msgr2=0x7f3e6c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:14.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e7249a640 1 -- 172.21.15.7:0/2250979216 shutdown_connections 2023-11-28T18:40:14.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.847+0000 7f3e7249a640 1 -- 172.21.15.7:0/2250979216 wait complete. 2023-11-28T18:40:14.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e7249a640 1 Processor -- start 2023-11-28T18:40:14.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e7249a640 1 -- start start 2023-11-28T18:40:14.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e7249a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c154300 0x7f3e6c13d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:14.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e7249a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c15dc50 0x7f3e6c13bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:14.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e7249a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e6c15e090 0x7f3e6c1f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:14.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e6b7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c15dc50 0x7f3e6c13bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:14.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e6b7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c15dc50 0x7f3e6c13bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:40526/0 (socket says 172.21.15.7:40526) 2023-11-28T18:40:14.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e6bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c154300 0x7f3e6c13d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:14.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e6bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c154300 0x7f3e6c13d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:34066/0 (socket says 172.21.15.7:34066) 2023-11-28T18:40:14.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e70a37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e6c15e090 0x7f3e6c1f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:14.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e70a37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e6c15e090 0x7f3e6c1f6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38792/0 (socket says 172.21.15.7:38792) 2023-11-28T18:40:14.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e7249a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c142ce0 con 0x7f3e6c15e090 2023-11-28T18:40:14.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e70a37640 1 -- 172.21.15.7:0/389830105 learned_addr learned my addr 172.21.15.7:0/389830105 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:14.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c142270 con 0x7f3e6c15dc50 2023-11-28T18:40:14.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.851+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c142a00 con 0x7f3e6c154300 2023-11-28T18:40:14.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.855+0000 7f3e6b7fe640 1 -- 172.21.15.7:0/389830105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c154300 msgr2=0x7f3e6c13d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:14.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.855+0000 7f3e6b7fe640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c154300 0x7f3e6c13d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:14.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.855+0000 7f3e6b7fe640 1 -- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e6c15e090 msgr2=0x7f3e6c1f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:14.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.855+0000 7f3e6b7fe640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e6c15e090 0x7f3e6c1f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.855+0000 7f3e6b7fe640 1 -- 172.21.15.7:0/389830105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0094d0 con 0x7f3e6c15dc50 2023-11-28T18:40:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.859+0000 7f3e6b7fe640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c15dc50 0x7f3e6c13bd40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000e870 tx=0x7f3e6000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.859+0000 7f3e68ff9640 1 -- 172.21.15.7:0/389830105 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6000c940 con 0x7f3e6c15dc50 2023-11-28T18:40:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.859+0000 7f3e68ff9640 1 -- 172.21.15.7:0/389830105 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6000caa0 con 0x7f3e6c15dc50 2023-11-28T18:40:14.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.859+0000 7f3e68ff9640 1 -- 172.21.15.7:0/389830105 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6000ce10 con 0x7f3e6c15dc50 2023-11-28T18:40:14.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.859+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c0b5420 con 0x7f3e6c15dc50 2023-11-28T18:40:14.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.859+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c0b3b90 con 0x7f3e6c15dc50 2023-11-28T18:40:14.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.859+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6c0b5260 con 0x7f3e6c15dc50 2023-11-28T18:40:14.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.863+0000 7f3e68ff9640 1 -- 172.21.15.7:0/389830105 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3e60013470 con 0x7f3e6c15dc50 2023-11-28T18:40:14.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.863+0000 7f3e68ff9640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3e580669a0 0x7f3e58068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:14.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.863+0000 7f3e68ff9640 1 -- 172.21.15.7:0/389830105 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f3e60084950 con 0x7f3e6c15dc50 2023-11-28T18:40:14.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.863+0000 7f3e6bfff640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3e580669a0 0x7f3e58068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:14.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.867+0000 7f3e6bfff640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3e580669a0 0x7f3e58068e60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c0af000 tx=0x7f3e5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:14.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:14.867+0000 7f3e68ff9640 1 -- 172.21.15.7:0/389830105 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3e60051860 con 0x7f3e6c15dc50 2023-11-28T18:40:15.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.311+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e6c0cf020 con 0x7f3e6c15dc50 2023-11-28T18:40:15.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.315+0000 7f3e68ff9640 1 -- 172.21.15.7:0/389830105 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6558 (secure 0 0 0) 0x7f3e60017090 con 0x7f3e6c15dc50 2023-11-28T18:40:15.317 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:15.317 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:38:58.455452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5307,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5307":{"gid":5307,"name":"a","rank":2,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.7:6835/1029915019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1029915019},{"type":"v1","addr":"172.21.15.7:6835","nonce":1029915019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 71 2023-11-28T18:40:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3e580669a0 msgr2=0x7f3e58068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3e580669a0 0x7f3e58068e60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c0af000 tx=0x7f3e5c009040 comp rx=0 tx=0).stop 2023-11-28T18:40:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c15dc50 msgr2=0x7f3e6c13bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c15dc50 0x7f3e6c13bd40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000e870 tx=0x7f3e6000ed30 comp rx=0 tx=0).stop 2023-11-28T18:40:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 shutdown_connections 2023-11-28T18:40:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3e580669a0 0x7f3e58068e60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e6c15e090 0x7f3e6c1f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e6c15dc50 0x7f3e6c13bd40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 --2- 172.21.15.7:0/389830105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e6c154300 0x7f3e6c13d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 >> 172.21.15.7:0/389830105 conn(0x7f3e6c0662e0 msgr2=0x7f3e6c146e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 shutdown_connections 2023-11-28T18:40:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:15.319+0000 7f3e7249a640 1 -- 172.21.15.7:0/389830105 wait complete. 2023-11-28T18:40:15.369 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-11-28T18:40:17.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:17.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.655+0000 7fcd6ffff640 1 Processor -- start 2023-11-28T18:40:17.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.655+0000 7fcd6ffff640 1 -- start start 2023-11-28T18:40:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.655+0000 7fcd6ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcd68037500 0x7fcd6804f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd6ffff640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd680dfac0 con 0x7fcd680d91e0 2023-11-28T18:40:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd6ffff640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd680065b0 con 0x7fcd680d8e00 2023-11-28T18:40:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd6ffff640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd680063a0 con 0x7fcd68037500 2023-11-28T18:40:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd6e7fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcd68037500 0x7fcd6804f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd75972640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcd680d91e0 0x7fcd680e95f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58470/0 (socket says 172.21.15.7:58470) 2023-11-28T18:40:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd75972640 1 -- 172.21.15.7:0/2905287050 learned_addr learned my addr 172.21.15.7:0/2905287050 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd6effd640 1 --1- 172.21.15.7:0/2905287050 >> v1:172.21.15.193:6789/0 conn(0x7fcd680d8e00 0x7fcd68007370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:57788/0 (socket says 172.21.15.7:57788) 2023-11-28T18:40:17.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd6e7fc640 1 -- 172.21.15.7:0/2905287050 >> v1:172.21.15.193:6789/0 conn(0x7fcd680d8e00 legacy=0x7fcd68007370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:17.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd6e7fc640 1 -- 172.21.15.7:0/2905287050 >> v1:172.21.15.7:6789/0 conn(0x7fcd680d91e0 legacy=0x7fcd680e95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:17.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.659+0000 7fcd6e7fc640 1 -- 172.21.15.7:0/2905287050 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68006890 con 0x7fcd68037500 2023-11-28T18:40:17.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.663+0000 7fcd6e7fc640 1 --2- 172.21.15.7:0/2905287050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd68037500 0x7fcd6804f360 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fcd640097c0 tx=0x7fcd6401c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415b5c39249b1d1d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:17.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.663+0000 7fcd6dffb640 1 -- 172.21.15.7:0/2905287050 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640ab600 con 0x7fcd68037500 2023-11-28T18:40:17.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.663+0000 7fcd6dffb640 1 -- 172.21.15.7:0/2905287050 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd640a7050 con 0x7fcd68037500 2023-11-28T18:40:17.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.663+0000 7fcd6dffb640 1 -- 172.21.15.7:0/2905287050 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640ab980 con 0x7fcd68037500 2023-11-28T18:40:17.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.667+0000 7fcd6ffff640 1 -- 172.21.15.7:0/2905287050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd68037500 msgr2=0x7fcd6804f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:17.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.667+0000 7fcd6ffff640 1 --2- 172.21.15.7:0/2905287050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd68037500 0x7fcd6804f360 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fcd640097c0 tx=0x7fcd6401c240 comp rx=0 tx=0).stop 2023-11-28T18:40:17.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.667+0000 7fcd6ffff640 1 -- 172.21.15.7:0/2905287050 shutdown_connections 2023-11-28T18:40:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.667+0000 7fcd6ffff640 1 --2- 172.21.15.7:0/2905287050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd68037500 0x7fcd6804f360 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.667+0000 7fcd6ffff640 1 -- 172.21.15.7:0/2905287050 >> 172.21.15.7:0/2905287050 conn(0x7fcd68054730 msgr2=0x7fcd68054b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:17.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.667+0000 7fcd6ffff640 1 -- 172.21.15.7:0/2905287050 shutdown_connections 2023-11-28T18:40:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.671+0000 7fcd6ffff640 1 -- 172.21.15.7:0/2905287050 wait complete. 2023-11-28T18:40:17.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.671+0000 7fcd6ffff640 1 Processor -- start 2023-11-28T18:40:17.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.671+0000 7fcd6ffff640 1 -- start start 2023-11-28T18:40:17.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd68037500 0x7fcd680cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:17.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcd680d8e00 0x7fcd680c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:17.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680d91e0 0x7fcd680c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:17.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68053960 con 0x7fcd680d91e0 2023-11-28T18:40:17.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680cc0a0 con 0x7fcd68037500 2023-11-28T18:40:17.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680cbdc0 con 0x7fcd680d8e00 2023-11-28T18:40:17.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd68037500 0x7fcd680cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:17.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd68037500 0x7fcd680cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:40530/0 (socket says 172.21.15.7:40530) 2023-11-28T18:40:17.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6effd640 1 -- 172.21.15.7:0/4146477935 learned_addr learned my addr 172.21.15.7:0/4146477935 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:17.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd75972640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680d91e0 0x7fcd680c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:17.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6e7fc640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcd680d8e00 0x7fcd680c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:17.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd75972640 1 -- 172.21.15.7:0/4146477935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcd680d8e00 msgr2=0x7fcd680c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:17.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd75972640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcd680d8e00 0x7fcd680c8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:17.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd75972640 1 -- 172.21.15.7:0/4146477935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd68037500 msgr2=0x7fcd680cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:17.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd75972640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd68037500 0x7fcd680cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:17.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd75972640 1 -- 172.21.15.7:0/4146477935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd64009420 con 0x7fcd680d91e0 2023-11-28T18:40:17.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd75972640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680d91e0 0x7fcd680c8eb0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fcd700501d0 tx=0x7fcd7006e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:17.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd57fff640 1 -- 172.21.15.7:0/4146477935 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7006c600 con 0x7fcd680d91e0 2023-11-28T18:40:17.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd57fff640 1 -- 172.21.15.7:0/4146477935 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7006c7a0 con 0x7fcd680d91e0 2023-11-28T18:40:17.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd57fff640 1 -- 172.21.15.7:0/4146477935 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7006cbd0 con 0x7fcd680d91e0 2023-11-28T18:40:17.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd680df890 con 0x7fcd680d91e0 2023-11-28T18:40:17.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd68053b50 con 0x7fcd680d91e0 2023-11-28T18:40:17.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.675+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd680d7f80 con 0x7fcd680d91e0 2023-11-28T18:40:17.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.679+0000 7fcd57fff640 1 -- 172.21.15.7:0/4146477935 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fcd7006cd30 con 0x7fcd680d91e0 2023-11-28T18:40:17.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.679+0000 7fcd57fff640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcd5c0669a0 0x7fcd5c068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:17.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.679+0000 7fcd57fff640 1 -- 172.21.15.7:0/4146477935 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fcd70102280 con 0x7fcd680d91e0 2023-11-28T18:40:17.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.679+0000 7fcd6effd640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcd5c0669a0 0x7fcd5c068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:17.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.683+0000 7fcd6effd640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcd5c0669a0 0x7fcd5c068e60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcd60004e40 tx=0x7fcd60007f30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:17.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:17.687+0000 7fcd57fff640 1 -- 172.21.15.7:0/4146477935 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcd700cf130 con 0x7fcd680d91e0 2023-11-28T18:40:18.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.019+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd680058c0 con 0x7fcd680d91e0 2023-11-28T18:40:18.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.019+0000 7fcd57fff640 1 -- 172.21.15.7:0/4146477935 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5770 (secure 0 0 0) 0x7fcd70067cf0 con 0x7fcd680d91e0 2023-11-28T18:40:18.024 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:18.024 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:17.364246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:18.024 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 72 2023-11-28T18:40:18.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcd5c0669a0 msgr2=0x7fcd5c068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcd5c0669a0 0x7fcd5c068e60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcd60004e40 tx=0x7fcd60007f30 comp rx=0 tx=0).stop 2023-11-28T18:40:18.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680d91e0 msgr2=0x7fcd680c8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680d91e0 0x7fcd680c8eb0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fcd700501d0 tx=0x7fcd7006e9f0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 shutdown_connections 2023-11-28T18:40:18.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fcd5c0669a0 0x7fcd5c068e60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680d91e0 0x7fcd680c8eb0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcd680d8e00 0x7fcd680c8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 --2- 172.21.15.7:0/4146477935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd68037500 0x7fcd680cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 >> 172.21.15.7:0/4146477935 conn(0x7fcd68054730 msgr2=0x7fcd6804f300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:18.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 shutdown_connections 2023-11-28T18:40:18.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.023+0000 7fcd6ffff640 1 -- 172.21.15.7:0/4146477935 wait complete. 2023-11-28T18:40:18.061 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-11-28T18:40:18.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:40:18.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.327+0000 7f0b971d1640 1 Processor -- start 2023-11-28T18:40:18.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.331+0000 7f0b971d1640 1 -- start start 2023-11-28T18:40:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.331+0000 7f0b971d1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b901542e0 0x7f0b9015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.331+0000 7f0b971d1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b900ca920 con 0x7f0b90072450 2023-11-28T18:40:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.331+0000 7f0b971d1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9015e550 con 0x7f0b900cbb70 2023-11-28T18:40:18.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.331+0000 7f0b971d1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b90071d50 con 0x7f0b901542e0 2023-11-28T18:40:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.331+0000 7f0b8ffff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f0b900cbb70 0x7f0b900cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:57794/0 (socket says 172.21.15.7:57794) 2023-11-28T18:40:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.331+0000 7f0b8ffff640 1 -- 172.21.15.7:0/3039160797 learned_addr learned my addr 172.21.15.7:0/3039160797 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.331+0000 7f0b94f6d640 1 --2- 172.21.15.7:0/3039160797 >> v2:172.21.15.7:3300/0 conn(0x7f0b901542e0 0x7f0b9015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.335+0000 7f0b94f6d640 1 -- 172.21.15.7:0/3039160797 >> v1:172.21.15.193:6789/0 conn(0x7f0b900cbb70 legacy=0x7f0b900cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.335+0000 7f0b8f7fe640 1 -- 172.21.15.7:0/3039160797 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020004156 0 0) 0x7f0b9015e550 con 0x7f0b900cbb70 2023-11-28T18:40:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.335+0000 7f0b94f6d640 1 -- 172.21.15.7:0/3039160797 >> v1:172.21.15.7:6789/0 conn(0x7f0b90072450 legacy=0x7f0b90164990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.335+0000 7f0b94f6d640 1 -- 172.21.15.7:0/3039160797 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b900c4cd0 con 0x7f0b901542e0 2023-11-28T18:40:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.339+0000 7f0b94f6d640 1 --2- 172.21.15.7:0/3039160797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b901542e0 0x7f0b9015e010 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009d90 tx=0x7f0b8801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac5093730ac7e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:18.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.339+0000 7f0b8f7fe640 1 -- 172.21.15.7:0/3039160797 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b880a5a10 con 0x7f0b901542e0 2023-11-28T18:40:18.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.339+0000 7f0b8f7fe640 1 -- 172.21.15.7:0/3039160797 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b880a5b70 con 0x7f0b901542e0 2023-11-28T18:40:18.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.339+0000 7f0b8f7fe640 1 -- 172.21.15.7:0/3039160797 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b880ad440 con 0x7f0b901542e0 2023-11-28T18:40:18.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.343+0000 7f0b971d1640 1 -- 172.21.15.7:0/3039160797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b901542e0 msgr2=0x7f0b9015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.343+0000 7f0b971d1640 1 --2- 172.21.15.7:0/3039160797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b901542e0 0x7f0b9015e010 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009d90 tx=0x7f0b8801c950 comp rx=0 tx=0).stop 2023-11-28T18:40:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.343+0000 7f0b971d1640 1 -- 172.21.15.7:0/3039160797 shutdown_connections 2023-11-28T18:40:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.343+0000 7f0b971d1640 1 --2- 172.21.15.7:0/3039160797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b901542e0 0x7f0b9015e010 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.343+0000 7f0b971d1640 1 -- 172.21.15.7:0/3039160797 >> 172.21.15.7:0/3039160797 conn(0x7f0b900717e0 msgr2=0x7f0b900662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.343+0000 7f0b971d1640 1 -- 172.21.15.7:0/3039160797 shutdown_connections 2023-11-28T18:40:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.343+0000 7f0b971d1640 1 -- 172.21.15.7:0/3039160797 wait complete. 2023-11-28T18:40:18.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.347+0000 7f0b971d1640 1 Processor -- start 2023-11-28T18:40:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b971d1640 1 -- start start 2023-11-28T18:40:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b971d1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b90072450 0x7f0b900b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b971d1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b900cbb70 0x7f0b90151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:18.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b971d1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b901542e0 0x7f0b901f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:18.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b94f6d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b90072450 0x7f0b900b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b94f6d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b90072450 0x7f0b900b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38810/0 (socket says 172.21.15.7:38810) 2023-11-28T18:40:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b8ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b900cbb70 0x7f0b90151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b8ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b900cbb70 0x7f0b90151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:40534/0 (socket says 172.21.15.7:40534) 2023-11-28T18:40:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b8ffff640 1 -- 172.21.15.7:0/327351902 learned_addr learned my addr 172.21.15.7:0/327351902 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90072030 con 0x7f0b90072450 2023-11-28T18:40:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90151bc0 con 0x7f0b900cbb70 2023-11-28T18:40:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b9576e640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b901542e0 0x7f0b901f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:18.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.351+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9014e8c0 con 0x7f0b901542e0 2023-11-28T18:40:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8ffff640 1 -- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b901542e0 msgr2=0x7f0b901f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8ffff640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b901542e0 0x7f0b901f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8ffff640 1 -- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b90072450 msgr2=0x7f0b900b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8ffff640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b90072450 0x7f0b900b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8ffff640 1 -- 172.21.15.7:0/327351902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88009a70 con 0x7f0b900cbb70 2023-11-28T18:40:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8ffff640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b900cbb70 0x7f0b90151680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b800011f0 tx=0x7f0b8000c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b9576e640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b901542e0 0x7f0b901f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:40:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8d7fa640 1 -- 172.21.15.7:0/327351902 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b800074e0 con 0x7f0b900cbb70 2023-11-28T18:40:18.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8d7fa640 1 -- 172.21.15.7:0/327351902 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80007640 con 0x7f0b900cbb70 2023-11-28T18:40:18.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b8d7fa640 1 -- 172.21.15.7:0/327351902 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80004a10 con 0x7f0b900cbb70 2023-11-28T18:40:18.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b901f2e10 con 0x7f0b900cbb70 2023-11-28T18:40:18.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b901f2ae0 con 0x7f0b900cbb70 2023-11-28T18:40:18.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.355+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9006c670 con 0x7f0b900cbb70 2023-11-28T18:40:18.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.359+0000 7f0b8d7fa640 1 -- 172.21.15.7:0/327351902 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0b80004b70 con 0x7f0b900cbb70 2023-11-28T18:40:18.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.359+0000 7f0b8d7fa640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b700669a0 0x7f0b70068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:18.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.359+0000 7f0b8d7fa640 1 -- 172.21.15.7:0/327351902 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f0b80083e80 con 0x7f0b900cbb70 2023-11-28T18:40:18.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.359+0000 7f0b94f6d640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b700669a0 0x7f0b70068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:18.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.363+0000 7f0b94f6d640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b700669a0 0x7f0b70068e60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b880af000 tx=0x7f0b880a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:18.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.367+0000 7f0b8d7fa640 1 -- 172.21.15.7:0/327351902 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0b80050d40 con 0x7f0b900cbb70 2023-11-28T18:40:18.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.735+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b901f2940 con 0x7f0b900cbb70 2023-11-28T18:40:18.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.735+0000 7f0b8d7fa640 1 -- 172.21.15.7:0/327351902 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5770 (secure 0 0 0) 0x7f0b901f2940 con 0x7f0b900cbb70 2023-11-28T18:40:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:18.741 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:17.364246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:18.741 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 72 2023-11-28T18:40:18.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b700669a0 msgr2=0x7f0b70068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b700669a0 0x7f0b70068e60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b880af000 tx=0x7f0b880a7070 comp rx=0 tx=0).stop 2023-11-28T18:40:18.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b900cbb70 msgr2=0x7f0b90151680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:18.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b900cbb70 0x7f0b90151680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b800011f0 tx=0x7f0b8000c9b0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 shutdown_connections 2023-11-28T18:40:18.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b700669a0 0x7f0b70068e60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0b901542e0 0x7f0b901f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b900cbb70 0x7f0b90151680 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 --2- 172.21.15.7:0/327351902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b90072450 0x7f0b900b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:18.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.739+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 >> 172.21.15.7:0/327351902 conn(0x7f0b900717e0 msgr2=0x7f0b900b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:18.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.743+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 shutdown_connections 2023-11-28T18:40:18.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:18.743+0000 7f0b971d1640 1 -- 172.21.15.7:0/327351902 wait complete. 2023-11-28T18:40:18.778 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 72, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 72, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 33 }, "gid_5450": { "addr": "172.21.15.193:6835/2303207996", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 2303207996, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 2303207996, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5450, "incarnation": 72, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:40:17.364246+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5359, "mds_2": 5450, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:40:20.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:21.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.015+0000 7f936ab1d640 1 Processor -- start 2023-11-28T18:40:21.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.015+0000 7f936ab1d640 1 -- start start 2023-11-28T18:40:21.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f936ab1d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f93641542e0 0x7f936415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f936ab1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cbb70 0x7f93640cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f936ab1d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93640ca920 con 0x7f93640cc810 2023-11-28T18:40:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f936ab1d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9364071d50 con 0x7f93641542e0 2023-11-28T18:40:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f936ab1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9364072030 con 0x7f93640cbb70 2023-11-28T18:40:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f9363fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cbb70 0x7f93640cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f9363fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cbb70 0x7f93640cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:34100/0 (socket says 172.21.15.7:34100) 2023-11-28T18:40:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f9363fff640 1 -- 172.21.15.7:0/160722410 learned_addr learned my addr 172.21.15.7:0/160722410 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f9363fff640 1 -- 172.21.15.7:0/160722410 >> v2:172.21.15.193:3300/0 conn(0x7f93641542e0 msgr2=0x7f936415e010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:40:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f9363fff640 1 --2- 172.21.15.7:0/160722410 >> v2:172.21.15.193:3300/0 conn(0x7f93641542e0 0x7f936415e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f9363fff640 1 -- 172.21.15.7:0/160722410 >> v1:172.21.15.193:6789/0 conn(0x7f93640cc810 legacy=0x7f9364072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.019+0000 7f9363fff640 1 -- 172.21.15.7:0/160722410 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936415e550 con 0x7f93640cbb70 2023-11-28T18:40:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f9363fff640 1 --2- 172.21.15.7:0/160722410 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cbb70 0x7f93640cc2d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f935c009e50 tx=0x7f935c01c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8823e3cb37f310de server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f93637fe640 1 -- 172.21.15.7:0/160722410 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c0a5870 con 0x7f93640cbb70 2023-11-28T18:40:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f93637fe640 1 -- 172.21.15.7:0/160722410 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935c0a59d0 con 0x7f93640cbb70 2023-11-28T18:40:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f93637fe640 1 -- 172.21.15.7:0/160722410 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c0a5d60 con 0x7f93640cbb70 2023-11-28T18:40:21.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f936ab1d640 1 -- 172.21.15.7:0/160722410 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cbb70 msgr2=0x7f93640cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:21.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f936ab1d640 1 --2- 172.21.15.7:0/160722410 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cbb70 0x7f93640cc2d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f935c009e50 tx=0x7f935c01c9f0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f936ab1d640 1 -- 172.21.15.7:0/160722410 shutdown_connections 2023-11-28T18:40:21.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f936ab1d640 1 --2- 172.21.15.7:0/160722410 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cbb70 0x7f93640cc2d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f936ab1d640 1 --2- 172.21.15.7:0/160722410 >> v2:172.21.15.193:3300/0 conn(0x7f93641542e0 0x7f936415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f936ab1d640 1 -- 172.21.15.7:0/160722410 >> 172.21.15.7:0/160722410 conn(0x7f93640717e0 msgr2=0x7f93640662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:21.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f936ab1d640 1 -- 172.21.15.7:0/160722410 shutdown_connections 2023-11-28T18:40:21.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.023+0000 7f936ab1d640 1 -- 172.21.15.7:0/160722410 wait complete. 2023-11-28T18:40:21.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f936ab1d640 1 Processor -- start 2023-11-28T18:40:21.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f936ab1d640 1 -- start start 2023-11-28T18:40:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f936ab1d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93640cbb70 0x7f9364150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f936ab1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cc810 0x7f93640cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f936ab1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93641542e0 0x7f936414f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f936ab1d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936414f210 con 0x7f93641542e0 2023-11-28T18:40:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f936ab1d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936414e6a0 con 0x7f93640cbb70 2023-11-28T18:40:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f936ab1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93641578a0 con 0x7f93640cc810 2023-11-28T18:40:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93688b9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93640cbb70 0x7f9364150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93690ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93641542e0 0x7f936414f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93688b9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93640cbb70 0x7f9364150a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:40540/0 (socket says 172.21.15.7:40540) 2023-11-28T18:40:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93688b9640 1 -- 172.21.15.7:0/3514157078 learned_addr learned my addr 172.21.15.7:0/3514157078 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f9363fff640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cc810 0x7f93640cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93688b9640 1 -- 172.21.15.7:0/3514157078 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cc810 msgr2=0x7f93640cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93688b9640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cc810 0x7f93640cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93688b9640 1 -- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93641542e0 msgr2=0x7f936414f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93688b9640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93641542e0 0x7f936414f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.027+0000 7f93688b9640 1 -- 172.21.15.7:0/3514157078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c009b30 con 0x7f93640cbb70 2023-11-28T18:40:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.031+0000 7f93690ba640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93641542e0 0x7f936414f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:40:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.031+0000 7f93688b9640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93640cbb70 0x7f9364150a20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f934c00b4a0 tx=0x7f934c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.031+0000 7f93617fa640 1 -- 172.21.15.7:0/3514157078 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c010610 con 0x7f93640cbb70 2023-11-28T18:40:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.031+0000 7f93617fa640 1 -- 172.21.15.7:0/3514157078 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c010770 con 0x7f93640cbb70 2023-11-28T18:40:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.031+0000 7f93617fa640 1 -- 172.21.15.7:0/3514157078 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c010b60 con 0x7f93640cbb70 2023-11-28T18:40:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.031+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936414e900 con 0x7f93640cbb70 2023-11-28T18:40:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.031+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936414eea0 con 0x7f93640cbb70 2023-11-28T18:40:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.031+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936406c670 con 0x7f93640cbb70 2023-11-28T18:40:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.035+0000 7f93617fa640 1 -- 172.21.15.7:0/3514157078 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f934c010cc0 con 0x7f93640cbb70 2023-11-28T18:40:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.035+0000 7f93617fa640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9344066920 0x7f9344068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.035+0000 7f93617fa640 1 -- 172.21.15.7:0/3514157078 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f934c0843a0 con 0x7f93640cbb70 2023-11-28T18:40:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.035+0000 7f9363fff640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9344066920 0x7f9344068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.039+0000 7f9363fff640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9344066920 0x7f9344068de0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f935c0ae000 tx=0x7f935c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:21.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.039+0000 7f93617fa640 1 -- 172.21.15.7:0/3514157078 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f934c051380 con 0x7f93640cbb70 2023-11-28T18:40:21.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.459+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f936404e890 con 0x7f93640cbb70 2023-11-28T18:40:21.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.459+0000 7f93617fa640 1 -- 172.21.15.7:0/3514157078 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5770 (secure 0 0 0) 0x7f934c0549a0 con 0x7f93640cbb70 2023-11-28T18:40:21.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:21.461 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:17.364246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:21.462 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 72 2023-11-28T18:40:21.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.459+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9344066920 msgr2=0x7f9344068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:21.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.459+0000 7f936ab1d640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9344066920 0x7f9344068de0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f935c0ae000 tx=0x7f935c0a7070 comp rx=0 tx=0).stop 2023-11-28T18:40:21.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.459+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93640cbb70 msgr2=0x7f9364150a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:21.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.459+0000 7f936ab1d640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93640cbb70 0x7f9364150a20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f934c00b4a0 tx=0x7f934c00b960 comp rx=0 tx=0).stop 2023-11-28T18:40:21.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.463+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 shutdown_connections 2023-11-28T18:40:21.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.463+0000 7f936ab1d640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9344066920 0x7f9344068de0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.463+0000 7f936ab1d640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93641542e0 0x7f936414f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.463+0000 7f936ab1d640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93640cc810 0x7f93640cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.463+0000 7f936ab1d640 1 --2- 172.21.15.7:0/3514157078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93640cbb70 0x7f9364150a20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:21.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.463+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 >> 172.21.15.7:0/3514157078 conn(0x7f93640717e0 msgr2=0x7f93640b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:21.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.463+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 shutdown_connections 2023-11-28T18:40:21.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:21.463+0000 7f936ab1d640 1 -- 172.21.15.7:0/3514157078 wait complete. 2023-11-28T18:40:21.501 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:40:23.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:23.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.767+0000 7f9fe54bb640 1 Processor -- start 2023-11-28T18:40:23.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.767+0000 7f9fe54bb640 1 -- start start 2023-11-28T18:40:23.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fe54bb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9fd80d9250 0x7f9fd80df9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:23.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fe54bb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9fd8037a00 0x7f9fd8038630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:23.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fe54bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd8038b70 0x7f9fd8050190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:23.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fe54bb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd80381a0 con 0x7f9fd80d9250 2023-11-28T18:40:23.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fe54bb640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd80dff20 con 0x7f9fd8037a00 2023-11-28T18:40:23.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fe54bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8037340 con 0x7f9fd8038b70 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9fd8037a00 0x7f9fd8038630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9fd8037a00 0x7f9fd8038630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:49990/0 (socket says 172.21.15.7:49990) 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fde7fc640 1 -- 172.21.15.7:0/2581398818 learned_addr learned my addr 172.21.15.7:0/2581398818 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fdeffd640 1 --2- 172.21.15.7:0/2581398818 >> v2:172.21.15.7:3300/0 conn(0x7f9fd80d9250 0x7f9fd80df9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fde7fc640 1 -- 172.21.15.7:0/2581398818 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd8038b70 msgr2=0x7f9fd8050190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fde7fc640 1 --2- 172.21.15.7:0/2581398818 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd8038b70 0x7f9fd8050190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fde7fc640 1 -- 172.21.15.7:0/2581398818 >> v2:172.21.15.7:3300/0 conn(0x7f9fd80d9250 msgr2=0x7f9fd80df9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fde7fc640 1 --2- 172.21.15.7:0/2581398818 >> v2:172.21.15.7:3300/0 conn(0x7f9fd80d9250 0x7f9fd80df9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fde7fc640 1 -- 172.21.15.7:0/2581398818 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd8038fc0 con 0x7f9fd8037a00 2023-11-28T18:40:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.771+0000 7f9fdeffd640 1 --2- 172.21.15.7:0/2581398818 >> v2:172.21.15.7:3300/0 conn(0x7f9fd80d9250 0x7f9fd80df9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:40:23.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.775+0000 7f9fde7fc640 1 --2- 172.21.15.7:0/2581398818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8037a00 0x7f9fd8038630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9fd804b680 tx=0x7f9fd401c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=515cdd3e0a1007fa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:23.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.775+0000 7f9fddffb640 1 -- 172.21.15.7:0/2581398818 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40ab650 con 0x7f9fd8037a00 2023-11-28T18:40:23.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.775+0000 7f9fddffb640 1 -- 172.21.15.7:0/2581398818 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd40ab7b0 con 0x7f9fd8037a00 2023-11-28T18:40:23.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.775+0000 7f9fddffb640 1 -- 172.21.15.7:0/2581398818 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40abb40 con 0x7f9fd8037a00 2023-11-28T18:40:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 -- 172.21.15.7:0/2581398818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8037a00 msgr2=0x7f9fd8038630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/2581398818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8037a00 0x7f9fd8038630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9fd804b680 tx=0x7f9fd401c490 comp rx=0 tx=0).stop 2023-11-28T18:40:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 -- 172.21.15.7:0/2581398818 shutdown_connections 2023-11-28T18:40:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/2581398818 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd8038b70 0x7f9fd8050190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/2581398818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8037a00 0x7f9fd8038630 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/2581398818 >> v2:172.21.15.7:3300/0 conn(0x7f9fd80d9250 0x7f9fd80df9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 -- 172.21.15.7:0/2581398818 >> 172.21.15.7:0/2581398818 conn(0x7f9fd8054620 msgr2=0x7f9fd8054a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 -- 172.21.15.7:0/2581398818 shutdown_connections 2023-11-28T18:40:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.779+0000 7f9fe54bb640 1 -- 172.21.15.7:0/2581398818 wait complete. 2023-11-28T18:40:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.787+0000 7f9fe54bb640 1 Processor -- start 2023-11-28T18:40:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.787+0000 7f9fe54bb640 1 -- start start 2023-11-28T18:40:23.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fe54bb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fd8037a00 0x7f9fd80cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:23.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fe54bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8038b70 0x7f9fd804e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:23.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fe54bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd80d9250 0x7f9fd817f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:23.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fe54bb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8048940 con 0x7f9fd8037a00 2023-11-28T18:40:23.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fe54bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8048cd0 con 0x7f9fd8038b70 2023-11-28T18:40:23.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fe54bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd80d02a0 con 0x7f9fd80d9250 2023-11-28T18:40:23.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fdf7fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd80d9250 0x7f9fd817f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:23.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8038b70 0x7f9fd804e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:23.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8038b70 0x7f9fd804e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:49998/0 (socket says 172.21.15.7:49998) 2023-11-28T18:40:23.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fde7fc640 1 -- 172.21.15.7:0/691386604 learned_addr learned my addr 172.21.15.7:0/691386604 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:23.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.791+0000 7f9fdeffd640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fd8037a00 0x7f9fd80cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:23.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.795+0000 7f9fde7fc640 1 -- 172.21.15.7:0/691386604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd80d9250 msgr2=0x7f9fd817f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:23.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.795+0000 7f9fde7fc640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd80d9250 0x7f9fd817f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:23.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.795+0000 7f9fde7fc640 1 -- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fd8037a00 msgr2=0x7f9fd80cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:23.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.795+0000 7f9fde7fc640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fd8037a00 0x7f9fd80cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:23.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.795+0000 7f9fde7fc640 1 -- 172.21.15.7:0/691386604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd40094d0 con 0x7f9fd8038b70 2023-11-28T18:40:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.795+0000 7f9fde7fc640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8038b70 0x7f9fd804e1f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9fd401ab40 tx=0x7f9fd401acd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.799+0000 7f9fc77fe640 1 -- 172.21.15.7:0/691386604 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40b4800 con 0x7f9fd8038b70 2023-11-28T18:40:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.799+0000 7f9fc77fe640 1 -- 172.21.15.7:0/691386604 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd40b4960 con 0x7f9fd8038b70 2023-11-28T18:40:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.799+0000 7f9fc77fe640 1 -- 172.21.15.7:0/691386604 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40b4d10 con 0x7f9fd8038b70 2023-11-28T18:40:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.799+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd804aef0 con 0x7f9fd8038b70 2023-11-28T18:40:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.799+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd80d04f0 con 0x7f9fd8038b70 2023-11-28T18:40:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.799+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd80052d0 con 0x7f9fd8038b70 2023-11-28T18:40:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.803+0000 7f9fc77fe640 1 -- 172.21.15.7:0/691386604 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9fd40c8420 con 0x7f9fd8038b70 2023-11-28T18:40:23.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.807+0000 7f9fc77fe640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9fcc0669a0 0x7f9fcc068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:23.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.807+0000 7f9fc77fe640 1 -- 172.21.15.7:0/691386604 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f9fd411f510 con 0x7f9fd8038b70 2023-11-28T18:40:23.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.807+0000 7f9fc77fe640 1 -- 172.21.15.7:0/691386604 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9fd411f8f0 con 0x7f9fd8038b70 2023-11-28T18:40:23.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.811+0000 7f9fdeffd640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9fcc0669a0 0x7f9fcc068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:23.811+0000 7f9fdeffd640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9fcc0669a0 0x7f9fcc068e60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000a950 tx=0x7f9fd0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:24.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.143+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fd80e9380 con 0x7f9fd8038b70 2023-11-28T18:40:24.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.147+0000 7f9fc77fe640 1 -- 172.21.15.7:0/691386604 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f9fd40ec580 con 0x7f9fd8038b70 2023-11-28T18:40:24.149 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:24.149 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:22.401308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:resolve","state_seq":22,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:24.149 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 73 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.147+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9fcc0669a0 msgr2=0x7f9fcc068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.147+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9fcc0669a0 0x7f9fcc068e60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000a950 tx=0x7f9fd0008040 comp rx=0 tx=0).stop 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.147+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8038b70 msgr2=0x7f9fd804e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.147+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8038b70 0x7f9fd804e1f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9fd401ab40 tx=0x7f9fd401acd0 comp rx=0 tx=0).stop 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.151+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 shutdown_connections 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.151+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9fcc0669a0 0x7f9fcc068e60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.151+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9fd80d9250 0x7f9fd817f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.151+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fd8038b70 0x7f9fd804e1f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.151+0000 7f9fe54bb640 1 --2- 172.21.15.7:0/691386604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fd8037a00 0x7f9fd80cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:24.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.151+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 >> 172.21.15.7:0/691386604 conn(0x7f9fd8054620 msgr2=0x7f9fd8006280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:24.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.151+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 shutdown_connections 2023-11-28T18:40:24.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:24.151+0000 7f9fe54bb640 1 -- 172.21.15.7:0/691386604 wait complete. 2023-11-28T18:40:24.185 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:40:26.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:26.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.423+0000 7f4cc73b8640 1 Processor -- start 2023-11-28T18:40:26.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.423+0000 7f4cc73b8640 1 -- start start 2023-11-28T18:40:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.427+0000 7f4cc73b8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4cc00cbb70 0x7f4cc00cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.427+0000 7f4cc73b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cc810 0x7f4cc0072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.427+0000 7f4cc73b8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc00ca920 con 0x7f4cc01542e0 2023-11-28T18:40:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.427+0000 7f4cc73b8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc0071d50 con 0x7f4cc00cbb70 2023-11-28T18:40:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.427+0000 7f4cc73b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc0072030 con 0x7f4cc00cc810 2023-11-28T18:40:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.427+0000 7f4cc5955640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cc810 0x7f4cc0072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.427+0000 7f4cc5955640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cc810 0x7f4cc0072450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:37312/0 (socket says 172.21.15.7:37312) 2023-11-28T18:40:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.427+0000 7f4cc5955640 1 -- 172.21.15.7:0/138517296 learned_addr learned my addr 172.21.15.7:0/138517296 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.431+0000 7f4cc4953640 1 --2- 172.21.15.7:0/138517296 >> v2:172.21.15.7:3300/0 conn(0x7f4cc00cbb70 0x7f4cc00cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.431+0000 7f4cc5955640 1 -- 172.21.15.7:0/138517296 >> v2:172.21.15.7:3300/0 conn(0x7f4cc00cbb70 msgr2=0x7f4cc00cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.431+0000 7f4cb7fff640 1 -- 172.21.15.7:0/138517296 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794269047 0 0) 0x7f4cc00ca920 con 0x7f4cc01542e0 2023-11-28T18:40:26.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.431+0000 7f4cc5955640 1 --2- 172.21.15.7:0/138517296 >> v2:172.21.15.7:3300/0 conn(0x7f4cc00cbb70 0x7f4cc00cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.431+0000 7f4cc5955640 1 -- 172.21.15.7:0/138517296 >> v1:172.21.15.7:6789/0 conn(0x7f4cc01542e0 legacy=0x7f4cc015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:26.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.431+0000 7f4cc5955640 1 -- 172.21.15.7:0/138517296 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc00c4cd0 con 0x7f4cc00cc810 2023-11-28T18:40:26.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cc5955640 1 --2- 172.21.15.7:0/138517296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cc810 0x7f4cc0072450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009e50 tx=0x7f4cb801ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94fd4767acc3406f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:26.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cb7fff640 1 -- 172.21.15.7:0/138517296 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a57f0 con 0x7f4cc00cc810 2023-11-28T18:40:26.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cb7fff640 1 -- 172.21.15.7:0/138517296 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb80a5950 con 0x7f4cc00cc810 2023-11-28T18:40:26.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cb7fff640 1 -- 172.21.15.7:0/138517296 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a5ce0 con 0x7f4cc00cc810 2023-11-28T18:40:26.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cc73b8640 1 -- 172.21.15.7:0/138517296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cc810 msgr2=0x7f4cc0072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:26.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/138517296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cc810 0x7f4cc0072450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009e50 tx=0x7f4cb801ca30 comp rx=0 tx=0).stop 2023-11-28T18:40:26.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cc73b8640 1 -- 172.21.15.7:0/138517296 shutdown_connections 2023-11-28T18:40:26.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/138517296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cc810 0x7f4cc0072450 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/138517296 >> v2:172.21.15.7:3300/0 conn(0x7f4cc00cbb70 0x7f4cc00cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.447+0000 7f4cc73b8640 1 -- 172.21.15.7:0/138517296 >> 172.21.15.7:0/138517296 conn(0x7f4cc00717e0 msgr2=0x7f4cc00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:26.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.451+0000 7f4cc73b8640 1 -- 172.21.15.7:0/138517296 shutdown_connections 2023-11-28T18:40:26.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.451+0000 7f4cc73b8640 1 -- 172.21.15.7:0/138517296 wait complete. 2023-11-28T18:40:26.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.451+0000 7f4cc73b8640 1 Processor -- start 2023-11-28T18:40:26.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.451+0000 7f4cc73b8640 1 -- start start 2023-11-28T18:40:26.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc73b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cbb70 0x7f4cc00c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:26.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc73b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cc810 0x7f4cc00ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:26.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc73b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc01542e0 0x7f4cc014a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:26.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc73b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc00cd6a0 con 0x7f4cc00cc810 2023-11-28T18:40:26.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc73b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc00cce90 con 0x7f4cc01542e0 2023-11-28T18:40:26.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc73b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc00cee60 con 0x7f4cc00cbb70 2023-11-28T18:40:26.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5154640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cbb70 0x7f4cc00c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:26.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc01542e0 0x7f4cc014a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:26.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc01542e0 0x7f4cc014a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50000/0 (socket says 172.21.15.7:50000) 2023-11-28T18:40:26.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5154640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cbb70 0x7f4cc00c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:37326/0 (socket says 172.21.15.7:37326) 2023-11-28T18:40:26.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 -- 172.21.15.7:0/586728800 learned_addr learned my addr 172.21.15.7:0/586728800 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:26.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc4953640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cc810 0x7f4cc00ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 -- 172.21.15.7:0/586728800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cbb70 msgr2=0x7f4cc00c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cbb70 0x7f4cc00c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 -- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cc810 msgr2=0x7f4cc00ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cc810 0x7f4cc00ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 -- 172.21.15.7:0/586728800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb8009b30 con 0x7f4cc01542e0 2023-11-28T18:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.455+0000 7f4cc5955640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc01542e0 0x7f4cc014a120 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4cb801a660 tx=0x7f4cb80197d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:26.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.459+0000 7f4cb5ffb640 1 -- 172.21.15.7:0/586728800 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80b8670 con 0x7f4cc01542e0 2023-11-28T18:40:26.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.459+0000 7f4cb5ffb640 1 -- 172.21.15.7:0/586728800 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb80b87d0 con 0x7f4cc01542e0 2023-11-28T18:40:26.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.459+0000 7f4cb5ffb640 1 -- 172.21.15.7:0/586728800 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80b8b40 con 0x7f4cc01542e0 2023-11-28T18:40:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.459+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc00cf0d0 con 0x7f4cc01542e0 2023-11-28T18:40:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.459+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc00cd8b0 con 0x7f4cc01542e0 2023-11-28T18:40:26.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.463+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca0005260 con 0x7f4cc01542e0 2023-11-28T18:40:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.467+0000 7f4cb5ffb640 1 -- 172.21.15.7:0/586728800 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4cb80b8ca0 con 0x7f4cc01542e0 2023-11-28T18:40:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.467+0000 7f4cb5ffb640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ca40669d0 0x7f4ca4068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.467+0000 7f4cb5ffb640 1 -- 172.21.15.7:0/586728800 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f4cb8120630 con 0x7f4cc01542e0 2023-11-28T18:40:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.467+0000 7f4cc5154640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ca40669d0 0x7f4ca4068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:26.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.471+0000 7f4cc5154640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ca40669d0 0x7f4ca4068e90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc007fd0 tx=0x7f4cbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:26.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.475+0000 7f4cb5ffb640 1 -- 172.21.15.7:0/586728800 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4cb80ed4f0 con 0x7f4cc01542e0 2023-11-28T18:40:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.903+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ca00057e0 con 0x7f4cc01542e0 2023-11-28T18:40:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.903+0000 7f4cb5ffb640 1 -- 172.21.15.7:0/586728800 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f4cb80f0b10 con 0x7f4cc01542e0 2023-11-28T18:40:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:26.908 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:22.401308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:resolve","state_seq":22,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 73 2023-11-28T18:40:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ca40669d0 msgr2=0x7f4ca4068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ca40669d0 0x7f4ca4068e90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc007fd0 tx=0x7f4cbc009040 comp rx=0 tx=0).stop 2023-11-28T18:40:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc01542e0 msgr2=0x7f4cc014a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc01542e0 0x7f4cc014a120 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4cb801a660 tx=0x7f4cb80197d0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 shutdown_connections 2023-11-28T18:40:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ca40669d0 0x7f4ca4068e90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc01542e0 0x7f4cc014a120 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cc810 0x7f4cc00ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 --2- 172.21.15.7:0/586728800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cbb70 0x7f4cc00c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:26.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 >> 172.21.15.7:0/586728800 conn(0x7f4cc00717e0 msgr2=0x7f4cc0144dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:26.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 shutdown_connections 2023-11-28T18:40:26.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:26.907+0000 7f4cc73b8640 1 -- 172.21.15.7:0/586728800 wait complete. 2023-11-28T18:40:26.950 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:40:28.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.179+0000 7f0bb57b5640 1 Processor -- start 2023-11-28T18:40:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.179+0000 7f0bb57b5640 1 -- start start 2023-11-28T18:40:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.179+0000 7f0bb57b5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0bb015ca30 0x7f0bb0159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.179+0000 7f0bb57b5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0bb015ce10 0x7f0bb00ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.179+0000 7f0bb57b5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb015a620 con 0x7f0bb00cafe0 2023-11-28T18:40:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.179+0000 7f0bb57b5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb015a900 con 0x7f0bb015ca30 2023-11-28T18:40:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.179+0000 7f0bb57b5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb015abe0 con 0x7f0bb015ce10 2023-11-28T18:40:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.183+0000 7f0baf7fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0bb015ce10 0x7f0bb00ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.183+0000 7f0baf7fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0bb015ce10 0x7f0bb00ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50016/0 (socket says 172.21.15.7:50016) 2023-11-28T18:40:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.183+0000 7f0baf7fe640 1 -- 172.21.15.7:0/1133189868 learned_addr learned my addr 172.21.15.7:0/1133189868 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.183+0000 7f0baf7fe640 1 -- 172.21.15.7:0/1133189868 >> v2:172.21.15.7:3300/0 conn(0x7f0bb015ca30 msgr2=0x7f0bb0159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.183+0000 7f0baf7fe640 1 --2- 172.21.15.7:0/1133189868 >> v2:172.21.15.7:3300/0 conn(0x7f0bb015ca30 0x7f0bb0159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.183+0000 7f0baf7fe640 1 -- 172.21.15.7:0/1133189868 >> v1:172.21.15.193:6789/0 conn(0x7f0bb00cafe0 legacy=0x7f0bb00cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.183+0000 7f0baeffd640 1 -- 172.21.15.7:0/1133189868 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050252116 0 0) 0x7f0bb015a620 con 0x7f0bb00cafe0 2023-11-28T18:40:29.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.183+0000 7f0baf7fe640 1 -- 172.21.15.7:0/1133189868 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00cbb10 con 0x7f0bb015ce10 2023-11-28T18:40:29.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.187+0000 7f0baf7fe640 1 --2- 172.21.15.7:0/1133189868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 0x7f0bb00ca1d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00d0200 tx=0x7f0b9801c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6146831b337bab2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:29.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.187+0000 7f0baeffd640 1 -- 172.21.15.7:0/1133189868 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b980ac640 con 0x7f0bb015ce10 2023-11-28T18:40:29.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.187+0000 7f0baeffd640 1 -- 172.21.15.7:0/1133189868 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b980ac7a0 con 0x7f0bb015ce10 2023-11-28T18:40:29.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.187+0000 7f0baeffd640 1 -- 172.21.15.7:0/1133189868 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b980acb30 con 0x7f0bb015ce10 2023-11-28T18:40:29.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.187+0000 7f0bb57b5640 1 -- 172.21.15.7:0/1133189868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 msgr2=0x7f0bb00ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:29.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.187+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/1133189868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 0x7f0bb00ca1d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00d0200 tx=0x7f0b9801c4f0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.191+0000 7f0bb57b5640 1 -- 172.21.15.7:0/1133189868 shutdown_connections 2023-11-28T18:40:29.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.191+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/1133189868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 0x7f0bb00ca1d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.191+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/1133189868 >> v2:172.21.15.7:3300/0 conn(0x7f0bb015ca30 0x7f0bb0159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.191+0000 7f0bb57b5640 1 -- 172.21.15.7:0/1133189868 >> 172.21.15.7:0/1133189868 conn(0x7f0bb006c060 msgr2=0x7f0bb00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:29.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.191+0000 7f0bb57b5640 1 -- 172.21.15.7:0/1133189868 shutdown_connections 2023-11-28T18:40:29.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.191+0000 7f0bb57b5640 1 -- 172.21.15.7:0/1133189868 wait complete. 2023-11-28T18:40:29.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.191+0000 7f0bb57b5640 1 Processor -- start 2023-11-28T18:40:29.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.195+0000 7f0bb57b5640 1 -- start start 2023-11-28T18:40:29.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb57b5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bb00cafe0 0x7f0bb00c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:29.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb57b5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bb015ca30 0x7f0bb01ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:29.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb57b5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 0x7f0bb00724e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:29.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb57b5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb01ecd70 con 0x7f0bb00cafe0 2023-11-28T18:40:29.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb57b5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb00c5e80 con 0x7f0bb015ce10 2023-11-28T18:40:29.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb57b5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb01ec1f0 con 0x7f0bb015ca30 2023-11-28T18:40:29.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bb00cafe0 0x7f0bb00c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:29.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baf7fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bb015ca30 0x7f0bb01ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:29.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bb00cafe0 0x7f0bb00c6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55510/0 (socket says 172.21.15.7:55510) 2023-11-28T18:40:29.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baffff640 1 -- 172.21.15.7:0/3880667301 learned_addr learned my addr 172.21.15.7:0/3880667301 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:29.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb4fb4640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 0x7f0bb00724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:29.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baf7fe640 1 -- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 msgr2=0x7f0bb00724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:29.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baf7fe640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 0x7f0bb00724e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baf7fe640 1 -- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bb00cafe0 msgr2=0x7f0bb00c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:29.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baf7fe640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bb00cafe0 0x7f0bb00c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baf7fe640 1 -- 172.21.15.7:0/3880667301 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b980094d0 con 0x7f0bb015ca30 2023-11-28T18:40:29.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb4fb4640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 0x7f0bb00724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:40:29.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0baf7fe640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bb015ca30 0x7f0bb01ebb40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0b980ac000 tx=0x7f0b98019290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:29.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bacff9640 1 -- 172.21.15.7:0/3880667301 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98004220 con 0x7f0bb015ca30 2023-11-28T18:40:29.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bacff9640 1 -- 172.21.15.7:0/3880667301 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b98004380 con 0x7f0bb015ca30 2023-11-28T18:40:29.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bacff9640 1 -- 172.21.15.7:0/3880667301 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98003820 con 0x7f0bb015ca30 2023-11-28T18:40:29.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb01ec4d0 con 0x7f0bb015ca30 2023-11-28T18:40:29.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.199+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb01eca90 con 0x7f0bb015ca30 2023-11-28T18:40:29.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.203+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb004e890 con 0x7f0bb015ca30 2023-11-28T18:40:29.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.207+0000 7f0bacff9640 1 -- 172.21.15.7:0/3880667301 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0b980b3050 con 0x7f0bb015ca30 2023-11-28T18:40:29.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.207+0000 7f0bacff9640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b900668d0 0x7f0b90068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:29.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.207+0000 7f0bacff9640 1 -- 172.21.15.7:0/3880667301 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f0b9811e950 con 0x7f0bb015ca30 2023-11-28T18:40:29.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.207+0000 7f0baffff640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b900668d0 0x7f0b90068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:29.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.211+0000 7f0baffff640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b900668d0 0x7f0b90068d90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009ad0 tx=0x7f0ba8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:29.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.211+0000 7f0bacff9640 1 -- 172.21.15.7:0/3880667301 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0b980eb810 con 0x7f0bb015ca30 2023-11-28T18:40:29.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.603+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bb00cad10 con 0x7f0bb015ca30 2023-11-28T18:40:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.603+0000 7f0bacff9640 1 -- 172.21.15.7:0/3880667301 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f0b980044f0 con 0x7f0bb015ca30 2023-11-28T18:40:29.608 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:29.608 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:28.384875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 74 2023-11-28T18:40:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.607+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b900668d0 msgr2=0x7f0b90068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.607+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b900668d0 0x7f0b90068d90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009ad0 tx=0x7f0ba8009210 comp rx=0 tx=0).stop 2023-11-28T18:40:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.607+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bb015ca30 msgr2=0x7f0bb01ebb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.607+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bb015ca30 0x7f0bb01ebb40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0b980ac000 tx=0x7f0b98019290 comp rx=0 tx=0).stop 2023-11-28T18:40:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.611+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 shutdown_connections 2023-11-28T18:40:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.611+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0b900668d0 0x7f0b90068d90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.611+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb015ce10 0x7f0bb00724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.611+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bb015ca30 0x7f0bb01ebb40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.611+0000 7f0bb57b5640 1 --2- 172.21.15.7:0/3880667301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bb00cafe0 0x7f0bb00c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:29.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.611+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 >> 172.21.15.7:0/3880667301 conn(0x7f0bb006c060 msgr2=0x7f0bb00c9200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:29.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.611+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 shutdown_connections 2023-11-28T18:40:29.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:29.611+0000 7f0bb57b5640 1 -- 172.21.15.7:0/3880667301 wait complete. 2023-11-28T18:40:29.648 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:40:31.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:31.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.875+0000 7fbe4d679640 1 Processor -- start 2023-11-28T18:40:31.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe4d679640 1 -- start start 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe4d679640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbe481542e0 0x7fbe4815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe4d679640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbe48072450 0x7fbe48072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe4d679640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe480ca920 con 0x7fbe480cbb70 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe4d679640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe48071d50 con 0x7fbe481542e0 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe4d679640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe48072030 con 0x7fbe48072450 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe467fc640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fbe480cbb70 0x7fbe480cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:51948/0 (socket says 172.21.15.7:51948) 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe467fc640 1 -- 172.21.15.7:0/2701921786 learned_addr learned my addr 172.21.15.7:0/2701921786 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838180098 0 0) 0x7fbe480ca920 con 0x7fbe480cbb70 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe30003530 con 0x7fbe480cbb70 2023-11-28T18:40:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.879+0000 7fbe477fe640 1 --2- 172.21.15.7:0/2701921786 >> v2:172.21.15.193:3300/0 conn(0x7fbe48072450 0x7fbe48072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1914180473 0 0) 0x7fbe30003530 con 0x7fbe480cbb70 2023-11-28T18:40:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 >> v2:172.21.15.193:3300/0 conn(0x7fbe48072450 msgr2=0x7fbe48072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 --2- 172.21.15.7:0/2701921786 >> v2:172.21.15.193:3300/0 conn(0x7fbe48072450 0x7fbe48072870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 >> v2:172.21.15.7:3300/0 conn(0x7fbe481542e0 msgr2=0x7fbe4815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 --2- 172.21.15.7:0/2701921786 >> v2:172.21.15.7:3300/0 conn(0x7fbe481542e0 0x7fbe4815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4815e550 con 0x7fbe480cbb70 2023-11-28T18:40:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fbe3c001680 con 0x7fbe480cbb70 2023-11-28T18:40:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 >> v1:172.21.15.193:6789/0 conn(0x7fbe480cbb70 legacy=0x7fbe480cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300039d0 0x7fbe30003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe300047d0 0x7fbe30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe300051e0 0x7fbe3000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48071d50 con 0x7fbe300047d0 2023-11-28T18:40:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48072030 con 0x7fbe300039d0 2023-11-28T18:40:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3000c1b0 con 0x7fbe300051e0 2023-11-28T18:40:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe477fe640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe300051e0 0x7fbe3000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe477fe640 1 -- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300039d0 msgr2=0x7fbe30003df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:40:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe477fe640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300039d0 0x7fbe30003df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe477fe640 1 -- 172.21.15.7:0/2701921786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe300047d0 msgr2=0x7fbe30004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe477fe640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe300047d0 0x7fbe30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.883+0000 7fbe477fe640 1 -- 172.21.15.7:0/2701921786 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30003530 con 0x7fbe300051e0 2023-11-28T18:40:31.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe477fe640 1 -- 172.21.15.7:0/2701921786 reap_dead start 2023-11-28T18:40:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe477fe640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe300051e0 0x7fbe3000b990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbe40007360 tx=0x7fbe4001ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac629481c67feff2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe400a5d60 con 0x7fbe300051e0 2023-11-28T18:40:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe400039a0 con 0x7fbe300051e0 2023-11-28T18:40:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe45ffb640 1 -- 172.21.15.7:0/2701921786 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40003d50 con 0x7fbe300051e0 2023-11-28T18:40:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe4d679640 1 -- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe300051e0 msgr2=0x7fbe3000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe4d679640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe300051e0 0x7fbe3000b990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbe40007360 tx=0x7fbe4001ce60 comp rx=0 tx=0).stop 2023-11-28T18:40:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe4d679640 1 -- 172.21.15.7:0/2701921786 shutdown_connections 2023-11-28T18:40:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe4d679640 1 --2- 172.21.15.7:0/2701921786 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe300051e0 0x7fbe3000b990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe4d679640 1 -- 172.21.15.7:0/2701921786 >> 172.21.15.7:0/2701921786 conn(0x7fbe480717e0 msgr2=0x7fbe480662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe4d679640 1 -- 172.21.15.7:0/2701921786 shutdown_connections 2023-11-28T18:40:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.887+0000 7fbe4d679640 1 -- 172.21.15.7:0/2701921786 wait complete. 2023-11-28T18:40:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.891+0000 7fbe4d679640 1 Processor -- start 2023-11-28T18:40:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.891+0000 7fbe4d679640 1 -- start start 2023-11-28T18:40:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe4d679640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300051e0 0x7fbe4815d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe4d679640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe48159690 0x7fbe4815a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe4d679640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe4815aba0 0x7fbe48145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe4d679640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4815a240 con 0x7fbe48159690 2023-11-28T18:40:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe4d679640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe480ce7e0 con 0x7fbe300051e0 2023-11-28T18:40:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe4d679640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48154c10 con 0x7fbe4815aba0 2023-11-28T18:40:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe477fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe4815aba0 0x7fbe48145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe477fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe4815aba0 0x7fbe48145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:37360/0 (socket says 172.21.15.7:37360) 2023-11-28T18:40:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe477fe640 1 -- 172.21.15.7:0/2887616198 learned_addr learned my addr 172.21.15.7:0/2887616198 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe467fc640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe48159690 0x7fbe4815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe46ffd640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300051e0 0x7fbe4815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe46ffd640 1 -- 172.21.15.7:0/2887616198 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe4815aba0 msgr2=0x7fbe48145e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe46ffd640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe4815aba0 0x7fbe48145e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe46ffd640 1 -- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe48159690 msgr2=0x7fbe4815a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe46ffd640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe48159690 0x7fbe4815a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe46ffd640 1 -- 172.21.15.7:0/2887616198 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe481582e0 con 0x7fbe300051e0 2023-11-28T18:40:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe46ffd640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300051e0 0x7fbe4815d870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800b410 tx=0x7fbe38009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe2f7fe640 1 -- 172.21.15.7:0/2887616198 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38011830 con 0x7fbe300051e0 2023-11-28T18:40:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe2f7fe640 1 -- 172.21.15.7:0/2887616198 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe38011990 con 0x7fbe300051e0 2023-11-28T18:40:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe2f7fe640 1 -- 172.21.15.7:0/2887616198 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38011d20 con 0x7fbe300051e0 2023-11-28T18:40:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe481585c0 con 0x7fbe300051e0 2023-11-28T18:40:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.895+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe48158b80 con 0x7fbe300051e0 2023-11-28T18:40:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.899+0000 7fbe2f7fe640 1 -- 172.21.15.7:0/2887616198 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbe3801a400 con 0x7fbe300051e0 2023-11-28T18:40:31.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.899+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4806c670 con 0x7fbe300051e0 2023-11-28T18:40:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.903+0000 7fbe2f7fe640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe340669d0 0x7fbe34068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.903+0000 7fbe2f7fe640 1 -- 172.21.15.7:0/2887616198 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fbe380847d0 con 0x7fbe300051e0 2023-11-28T18:40:31.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.903+0000 7fbe467fc640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe340669d0 0x7fbe34068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:31.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.903+0000 7fbe467fc640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe340669d0 0x7fbe34068e90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00afd0 tx=0x7fbe3c00c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:31.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:31.907+0000 7fbe2f7fe640 1 -- 172.21.15.7:0/2887616198 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbe38051690 con 0x7fbe300051e0 2023-11-28T18:40:32.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.419+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe4804e890 con 0x7fbe300051e0 2023-11-28T18:40:32.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.423+0000 7fbe2f7fe640 1 -- 172.21.15.7:0/2887616198 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fbe38015090 con 0x7fbe300051e0 2023-11-28T18:40:32.428 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:32.428 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:28.384875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:32.428 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 74 2023-11-28T18:40:32.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe340669d0 msgr2=0x7fbe34068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:32.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe340669d0 0x7fbe34068e90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00afd0 tx=0x7fbe3c00c040 comp rx=0 tx=0).stop 2023-11-28T18:40:32.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300051e0 msgr2=0x7fbe4815d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:32.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300051e0 0x7fbe4815d870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800b410 tx=0x7fbe38009f90 comp rx=0 tx=0).stop 2023-11-28T18:40:32.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 shutdown_connections 2023-11-28T18:40:32.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbe340669d0 0x7fbe34068e90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:32.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbe4815aba0 0x7fbe48145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:32.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe48159690 0x7fbe4815a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:32.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 --2- 172.21.15.7:0/2887616198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe300051e0 0x7fbe4815d870 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800b410 tx=0x7fbe38009f90 comp rx=0 tx=0).stop 2023-11-28T18:40:32.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 >> 172.21.15.7:0/2887616198 conn(0x7fbe480717e0 msgr2=0x7fbe480b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:32.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 shutdown_connections 2023-11-28T18:40:32.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:32.427+0000 7fbe4d679640 1 -- 172.21.15.7:0/2887616198 wait complete. 2023-11-28T18:40:32.481 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 74, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 74, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5359": { "addr": "172.21.15.193:6837/1742899523", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1742899523, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1742899523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5359, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 33 }, "gid_5450": { "addr": "172.21.15.193:6835/2303207996", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 2303207996, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 2303207996, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5450, "incarnation": 72, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:reconnect", "state_seq": 24 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:40:28.384875+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5359, "mds_2": 5450, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:40:33.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:40:33.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:40:34.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:34.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.715+0000 7f6267cc9640 1 Processor -- start 2023-11-28T18:40:34.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.715+0000 7f6267cc9640 1 -- start start 2023-11-28T18:40:34.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.715+0000 7f6267cc9640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f62600c24f0 0x7f62601575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:34.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.715+0000 7f6267cc9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c7980 0x7f62600c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:34.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.715+0000 7f6267cc9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62600c6140 con 0x7f6260150b30 2023-11-28T18:40:34.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.715+0000 7f6267cc9640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62600c72b0 con 0x7f62600c24f0 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.715+0000 7f6267cc9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6260070c00 con 0x7f62600c7980 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.719+0000 7f6266266640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c7980 0x7f62600c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.719+0000 7f6265264640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f62600c24f0 0x7f62601575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.719+0000 7f6266266640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c7980 0x7f62600c7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:52850/0 (socket says 172.21.15.7:52850) 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.719+0000 7f6266266640 1 -- 172.21.15.7:0/626702128 learned_addr learned my addr 172.21.15.7:0/626702128 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.719+0000 7f6265264640 1 -- 172.21.15.7:0/626702128 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c7980 msgr2=0x7f62600c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.719+0000 7f6265264640 1 --2- 172.21.15.7:0/626702128 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c7980 0x7f62600c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.719+0000 7f6265264640 1 -- 172.21.15.7:0/626702128 >> v1:172.21.15.7:6789/0 conn(0x7f6260150b30 legacy=0x7f62600c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.719+0000 7f6265264640 1 -- 172.21.15.7:0/626702128 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6260157b00 con 0x7f62600c24f0 2023-11-28T18:40:34.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.723+0000 7f6265264640 1 --2- 172.21.15.7:0/626702128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62600c24f0 0x7f62601575c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f625c013050 tx=0x7f625c02fba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9a23e3deec327a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:34.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.723+0000 7f6264a63640 1 -- 172.21.15.7:0/626702128 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c0112d0 con 0x7f62600c24f0 2023-11-28T18:40:34.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.723+0000 7f6264a63640 1 -- 172.21.15.7:0/626702128 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c011430 con 0x7f62600c24f0 2023-11-28T18:40:34.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.723+0000 7f6264a63640 1 -- 172.21.15.7:0/626702128 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c00f850 con 0x7f62600c24f0 2023-11-28T18:40:34.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.731+0000 7f6267cc9640 1 -- 172.21.15.7:0/626702128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62600c24f0 msgr2=0x7f62601575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:34.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.731+0000 7f6267cc9640 1 --2- 172.21.15.7:0/626702128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62600c24f0 0x7f62601575c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f625c013050 tx=0x7f625c02fba0 comp rx=0 tx=0).stop 2023-11-28T18:40:34.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.735+0000 7f6267cc9640 1 -- 172.21.15.7:0/626702128 shutdown_connections 2023-11-28T18:40:34.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.735+0000 7f6267cc9640 1 --2- 172.21.15.7:0/626702128 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c7980 0x7f62600c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:34.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.735+0000 7f6267cc9640 1 --2- 172.21.15.7:0/626702128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62600c24f0 0x7f62601575c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:34.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.735+0000 7f6267cc9640 1 -- 172.21.15.7:0/626702128 >> 172.21.15.7:0/626702128 conn(0x7f62600662e0 msgr2=0x7f62600666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:34.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.735+0000 7f6267cc9640 1 -- 172.21.15.7:0/626702128 shutdown_connections 2023-11-28T18:40:34.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.735+0000 7f6267cc9640 1 -- 172.21.15.7:0/626702128 wait complete. 2023-11-28T18:40:34.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.735+0000 7f6267cc9640 1 Processor -- start 2023-11-28T18:40:34.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.735+0000 7f6267cc9640 1 -- start start 2023-11-28T18:40:34.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6267cc9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c24f0 0x7f6260155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:34.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6267cc9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62600c7980 0x7f6260156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6267cc9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6260150b30 0x7f626013bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6267cc9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62601533a0 con 0x7f62600c7980 2023-11-28T18:40:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265264640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62600c7980 0x7f6260156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265264640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62600c7980 0x7f6260156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42644/0 (socket says 172.21.15.7:42644) 2023-11-28T18:40:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265264640 1 -- 172.21.15.7:0/1639065715 learned_addr learned my addr 172.21.15.7:0/1639065715 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626014cc20 con 0x7f6260150b30 2023-11-28T18:40:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6260153970 con 0x7f62600c24f0 2023-11-28T18:40:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6266266640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6260150b30 0x7f626013bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:34.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265a65640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c24f0 0x7f6260155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:34.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265a65640 1 -- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6260150b30 msgr2=0x7f626013bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:34.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265a65640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6260150b30 0x7f626013bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:34.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265a65640 1 -- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62600c7980 msgr2=0x7f6260156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:34.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265a65640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62600c7980 0x7f6260156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:34.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6265a65640 1 -- 172.21.15.7:0/1639065715 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c011cc0 con 0x7f62600c24f0 2023-11-28T18:40:34.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.739+0000 7f6266266640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6260150b30 0x7f626013bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:40:34.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.743+0000 7f6265a65640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c24f0 0x7f6260155ae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6250000f70 tx=0x7f625000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:34.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.743+0000 7f62567fc640 1 -- 172.21.15.7:0/1639065715 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62500052d0 con 0x7f62600c24f0 2023-11-28T18:40:34.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.743+0000 7f62567fc640 1 -- 172.21.15.7:0/1639065715 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6250005430 con 0x7f62600c24f0 2023-11-28T18:40:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.743+0000 7f62567fc640 1 -- 172.21.15.7:0/1639065715 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250003840 con 0x7f62600c24f0 2023-11-28T18:40:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.743+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6260153c50 con 0x7f62600c24f0 2023-11-28T18:40:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.743+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6260154210 con 0x7f62600c24f0 2023-11-28T18:40:34.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.743+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6260154400 con 0x7f62600c24f0 2023-11-28T18:40:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.747+0000 7f62567fc640 1 -- 172.21.15.7:0/1639065715 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f62500039a0 con 0x7f62600c24f0 2023-11-28T18:40:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.747+0000 7f62567fc640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440669d0 0x7f6244068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.747+0000 7f62567fc640 1 -- 172.21.15.7:0/1639065715 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f6250083d40 con 0x7f62600c24f0 2023-11-28T18:40:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.747+0000 7f6265264640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440669d0 0x7f6244068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:34.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.751+0000 7f6265264640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440669d0 0x7f6244068e90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f625c02e250 tx=0x7f625c0bb070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:34.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:34.751+0000 7f62567fc640 1 -- 172.21.15.7:0/1639065715 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6250050c90 con 0x7f62600c24f0 2023-11-28T18:40:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.075+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f626004e890 con 0x7f62600c24f0 2023-11-28T18:40:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.075+0000 7f62567fc640 1 -- 172.21.15.7:0/1639065715 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f62500542b0 con 0x7f62600c24f0 2023-11-28T18:40:35.080 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:35.080 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:33.481653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 75 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440669d0 msgr2=0x7f6244068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440669d0 0x7f6244068e90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f625c02e250 tx=0x7f625c0bb070 comp rx=0 tx=0).stop 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c24f0 msgr2=0x7f6260155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c24f0 0x7f6260155ae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6250000f70 tx=0x7f625000c820 comp rx=0 tx=0).stop 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 shutdown_connections 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f62440669d0 0x7f6244068e90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6260150b30 0x7f626013bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62600c7980 0x7f6260156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 --2- 172.21.15.7:0/1639065715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f62600c24f0 0x7f6260155ae0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 >> 172.21.15.7:0/1639065715 conn(0x7f62600662e0 msgr2=0x7f626014e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 shutdown_connections 2023-11-28T18:40:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:35.079+0000 7f6267cc9640 1 -- 172.21.15.7:0/1639065715 wait complete. 2023-11-28T18:40:35.116 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:40:37.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:37.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.351+0000 7fdac251d640 1 Processor -- start 2023-11-28T18:40:37.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.351+0000 7fdac251d640 1 -- start start 2023-11-28T18:40:37.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdac251d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc1542e0 0x7fdabc15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdac251d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdabc0cbb70 0x7fdabc0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdac251d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc0ca920 con 0x7fdabc0cc810 2023-11-28T18:40:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdac251d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdabc071d50 con 0x7fdabc0cbb70 2023-11-28T18:40:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdac251d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc072030 con 0x7fdabc1542e0 2023-11-28T18:40:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdabbfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc1542e0 0x7fdabc15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdac0aba640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdabc0cc810 0x7fdabc072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33690/0 (socket says 172.21.15.7:33690) 2023-11-28T18:40:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdac0aba640 1 -- 172.21.15.7:0/655104650 learned_addr learned my addr 172.21.15.7:0/655104650 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.355+0000 7fdabbfff640 1 -- 172.21.15.7:0/655104650 >> v2:172.21.15.7:3300/0 conn(0x7fdabc0cbb70 msgr2=0x7fdabc0cc2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:40:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.359+0000 7fdabbfff640 1 --2- 172.21.15.7:0/655104650 >> v2:172.21.15.7:3300/0 conn(0x7fdabc0cbb70 0x7fdabc0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.359+0000 7fdabbfff640 1 -- 172.21.15.7:0/655104650 >> v1:172.21.15.7:6789/0 conn(0x7fdabc0cc810 legacy=0x7fdabc072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.359+0000 7fdabaffd640 1 -- 172.21.15.7:0/655104650 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77334095 0 0) 0x7fdabc0ca920 con 0x7fdabc0cc810 2023-11-28T18:40:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.359+0000 7fdabbfff640 1 -- 172.21.15.7:0/655104650 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc15e550 con 0x7fdabc1542e0 2023-11-28T18:40:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdabbfff640 1 --2- 172.21.15.7:0/655104650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc1542e0 0x7fdabc15e010 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009d90 tx=0x7fdab401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6989916fe7320ee0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdabaffd640 1 -- 172.21.15.7:0/655104650 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab40a5940 con 0x7fdabc1542e0 2023-11-28T18:40:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdabaffd640 1 -- 172.21.15.7:0/655104650 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab40a5aa0 con 0x7fdabc1542e0 2023-11-28T18:40:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdabaffd640 1 -- 172.21.15.7:0/655104650 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab40a5e30 con 0x7fdabc1542e0 2023-11-28T18:40:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdac251d640 1 -- 172.21.15.7:0/655104650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc1542e0 msgr2=0x7fdabc15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdac251d640 1 --2- 172.21.15.7:0/655104650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc1542e0 0x7fdabc15e010 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009d90 tx=0x7fdab401c970 comp rx=0 tx=0).stop 2023-11-28T18:40:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdac251d640 1 -- 172.21.15.7:0/655104650 shutdown_connections 2023-11-28T18:40:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdac251d640 1 --2- 172.21.15.7:0/655104650 >> v2:172.21.15.7:3300/0 conn(0x7fdabc0cbb70 0x7fdabc0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdac251d640 1 --2- 172.21.15.7:0/655104650 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc1542e0 0x7fdabc15e010 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdac251d640 1 -- 172.21.15.7:0/655104650 >> 172.21.15.7:0/655104650 conn(0x7fdabc0717e0 msgr2=0x7fdabc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdac251d640 1 -- 172.21.15.7:0/655104650 shutdown_connections 2023-11-28T18:40:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.363+0000 7fdac251d640 1 -- 172.21.15.7:0/655104650 wait complete. 2023-11-28T18:40:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.367+0000 7fdac251d640 1 Processor -- start 2023-11-28T18:40:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac251d640 1 -- start start 2023-11-28T18:40:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac251d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc0cbb70 0x7fdabc154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac251d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdabc0cc810 0x7fdabc1589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac251d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdabc1542e0 0x7fdabc158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac251d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc159e50 con 0x7fdabc1542e0 2023-11-28T18:40:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac251d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc15af80 con 0x7fdabc0cc810 2023-11-28T18:40:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac251d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc159b70 con 0x7fdabc0cbb70 2023-11-28T18:40:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac0aba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdabc1542e0 0x7fdabc158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac0aba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdabc1542e0 0x7fdabc158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42658/0 (socket says 172.21.15.7:42658) 2023-11-28T18:40:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdac0aba640 1 -- 172.21.15.7:0/1767499159 learned_addr learned my addr 172.21.15.7:0/1767499159 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdabbfff640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc0cbb70 0x7fdabc154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdabb7fe640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdabc0cc810 0x7fdabc1589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdabbfff640 1 -- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdabc0cc810 msgr2=0x7fdabc1589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdabbfff640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdabc0cc810 0x7fdabc1589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdabbfff640 1 -- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdabc1542e0 msgr2=0x7fdabc158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdabbfff640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdabc1542e0 0x7fdabc158ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdabbfff640 1 -- 172.21.15.7:0/1767499159 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab4009a70 con 0x7fdabc0cbb70 2023-11-28T18:40:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.371+0000 7fdabbfff640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc0cbb70 0x7fdabc154c40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdab40ac000 tx=0x7fdab4019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.375+0000 7fdac0aba640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdabc1542e0 0x7fdabc158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:40:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.375+0000 7fdab8ff9640 1 -- 172.21.15.7:0/1767499159 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab40bf070 con 0x7fdabc0cbb70 2023-11-28T18:40:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.375+0000 7fdab8ff9640 1 -- 172.21.15.7:0/1767499159 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab40aec90 con 0x7fdabc0cbb70 2023-11-28T18:40:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.375+0000 7fdab8ff9640 1 -- 172.21.15.7:0/1767499159 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab40b7410 con 0x7fdabc0cbb70 2023-11-28T18:40:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.375+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdabc157ed0 con 0x7fdabc0cbb70 2023-11-28T18:40:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.375+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdabc158490 con 0x7fdabc0cbb70 2023-11-28T18:40:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.375+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdabc06c670 con 0x7fdabc0cbb70 2023-11-28T18:40:37.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.379+0000 7fdab8ff9640 1 -- 172.21.15.7:0/1767499159 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fdab40b4050 con 0x7fdabc0cbb70 2023-11-28T18:40:37.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.379+0000 7fdab8ff9640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdaa8066900 0x7fdaa8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:37.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.379+0000 7fdab8ff9640 1 -- 172.21.15.7:0/1767499159 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fdab411f320 con 0x7fdabc0cbb70 2023-11-28T18:40:37.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.379+0000 7fdabb7fe640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdaa8066900 0x7fdaa8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.383+0000 7fdabb7fe640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdaa8066900 0x7fdaa8068dc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdabc15e340 tx=0x7fdab0006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.383+0000 7fdab8ff9640 1 -- 172.21.15.7:0/1767499159 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdab40ec2b0 con 0x7fdabc0cbb70 2023-11-28T18:40:37.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.839+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdabc04e890 con 0x7fdabc0cbb70 2023-11-28T18:40:37.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.843+0000 7fdab8ff9640 1 -- 172.21.15.7:0/1767499159 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5771 (secure 0 0 0) 0x7fdab40ef8d0 con 0x7fdabc0cbb70 2023-11-28T18:40:37.848 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:37.848 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:33.481653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:37.848 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 75 2023-11-28T18:40:37.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.851+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdaa8066900 msgr2=0x7fdaa8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:37.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.851+0000 7fdac251d640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdaa8066900 0x7fdaa8068dc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdabc15e340 tx=0x7fdab0006f90 comp rx=0 tx=0).stop 2023-11-28T18:40:37.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.851+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc0cbb70 msgr2=0x7fdabc154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:37.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.851+0000 7fdac251d640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc0cbb70 0x7fdabc154c40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdab40ac000 tx=0x7fdab4019710 comp rx=0 tx=0).stop 2023-11-28T18:40:37.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.855+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 shutdown_connections 2023-11-28T18:40:37.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.855+0000 7fdac251d640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fdaa8066900 0x7fdaa8068dc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.855+0000 7fdac251d640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdabc1542e0 0x7fdabc158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.855+0000 7fdac251d640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdabc0cc810 0x7fdabc1589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.855+0000 7fdac251d640 1 --2- 172.21.15.7:0/1767499159 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fdabc0cbb70 0x7fdabc154c40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:37.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.855+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 >> 172.21.15.7:0/1767499159 conn(0x7fdabc0717e0 msgr2=0x7fdabc15b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:37.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.859+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 shutdown_connections 2023-11-28T18:40:37.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:37.859+0000 7fdac251d640 1 -- 172.21.15.7:0/1767499159 wait complete. 2023-11-28T18:40:37.915 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:40:39.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.191+0000 7faa58246640 1 Processor -- start 2023-11-28T18:40:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.191+0000 7faa58246640 1 -- start start 2023-11-28T18:40:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.191+0000 7faa58246640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7faa500cbb70 0x7faa500cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.191+0000 7faa58246640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500ca920 con 0x7faa501542e0 2023-11-28T18:40:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.191+0000 7faa58246640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500c4cd0 con 0x7faa500cc810 2023-11-28T18:40:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.191+0000 7faa58246640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa50071d50 con 0x7faa500cbb70 2023-11-28T18:40:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.195+0000 7faa567e3640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7faa500cc810 0x7faa50072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:41088/0 (socket says 172.21.15.7:41088) 2023-11-28T18:40:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.195+0000 7faa567e3640 1 -- 172.21.15.7:0/2716659175 learned_addr learned my addr 172.21.15.7:0/2716659175 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.195+0000 7faa557e1640 1 --2- 172.21.15.7:0/2716659175 >> v2:172.21.15.193:3300/0 conn(0x7faa500cbb70 0x7faa500cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.195+0000 7faa557e1640 1 -- 172.21.15.7:0/2716659175 >> v1:172.21.15.193:6789/0 conn(0x7faa500cc810 legacy=0x7faa50072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.195+0000 7faa557e1640 1 -- 172.21.15.7:0/2716659175 >> v1:172.21.15.7:6789/0 conn(0x7faa501542e0 legacy=0x7faa5015e010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:40:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.195+0000 7faa557e1640 1 -- 172.21.15.7:0/2716659175 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500cab70 con 0x7faa500cbb70 2023-11-28T18:40:40.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa557e1640 1 --2- 172.21.15.7:0/2716659175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa500cbb70 0x7faa500cc2d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faa400097c0 tx=0x7faa4001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ad8087ececddc16 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:40.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa54fe0640 1 -- 172.21.15.7:0/2716659175 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40004230 con 0x7faa500cbb70 2023-11-28T18:40:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa54fe0640 1 -- 172.21.15.7:0/2716659175 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa400a7050 con 0x7faa500cbb70 2023-11-28T18:40:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa54fe0640 1 -- 172.21.15.7:0/2716659175 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40003620 con 0x7faa500cbb70 2023-11-28T18:40:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa58246640 1 -- 172.21.15.7:0/2716659175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa500cbb70 msgr2=0x7faa500cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa58246640 1 --2- 172.21.15.7:0/2716659175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa500cbb70 0x7faa500cc2d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faa400097c0 tx=0x7faa4001c460 comp rx=0 tx=0).stop 2023-11-28T18:40:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa58246640 1 -- 172.21.15.7:0/2716659175 shutdown_connections 2023-11-28T18:40:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa58246640 1 --2- 172.21.15.7:0/2716659175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa500cbb70 0x7faa500cc2d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa58246640 1 -- 172.21.15.7:0/2716659175 >> 172.21.15.7:0/2716659175 conn(0x7faa500717e0 msgr2=0x7faa500662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa58246640 1 -- 172.21.15.7:0/2716659175 shutdown_connections 2023-11-28T18:40:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.199+0000 7faa58246640 1 -- 172.21.15.7:0/2716659175 wait complete. 2023-11-28T18:40:40.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.207+0000 7faa58246640 1 Processor -- start 2023-11-28T18:40:40.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.207+0000 7faa58246640 1 -- start start 2023-11-28T18:40:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa58246640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa500cbb70 0x7faa500c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa58246640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa500cc810 0x7faa500c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa58246640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa501542e0 0x7faa500c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa58246640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa50072030 con 0x7faa500cc810 2023-11-28T18:40:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa58246640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa50145160 con 0x7faa501542e0 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa58246640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa50164d30 con 0x7faa500cbb70 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa500cbb70 0x7faa500c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa567e3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa501542e0 0x7faa500c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa500cbb70 0x7faa500c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:52892/0 (socket says 172.21.15.7:52892) 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 -- 172.21.15.7:0/3525080771 learned_addr learned my addr 172.21.15.7:0/3525080771 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 -- 172.21.15.7:0/3525080771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa501542e0 msgr2=0x7faa500c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa501542e0 0x7faa500c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 -- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa500cc810 msgr2=0x7faa500c7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa500cc810 0x7faa500c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 -- 172.21.15.7:0/3525080771 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40009420 con 0x7faa500cbb70 2023-11-28T18:40:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa55fe2640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa500cbb70 0x7faa500c8f10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa4c00b420 tx=0x7faa4c00b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa467fc640 1 -- 172.21.15.7:0/3525080771 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c011830 con 0x7faa500cbb70 2023-11-28T18:40:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa467fc640 1 -- 172.21.15.7:0/3525080771 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4c0119b0 con 0x7faa500cbb70 2023-11-28T18:40:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa467fc640 1 -- 172.21.15.7:0/3525080771 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c011d60 con 0x7faa500cbb70 2023-11-28T18:40:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.211+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa50165010 con 0x7faa500cbb70 2023-11-28T18:40:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.215+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa501655d0 con 0x7faa500cbb70 2023-11-28T18:40:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.215+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5006c670 con 0x7faa500cbb70 2023-11-28T18:40:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.215+0000 7faa467fc640 1 -- 172.21.15.7:0/3525080771 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7faa4c01b460 con 0x7faa500cbb70 2023-11-28T18:40:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.219+0000 7faa467fc640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa340668d0 0x7faa34068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.219+0000 7faa467fc640 1 -- 172.21.15.7:0/3525080771 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7faa4c084670 con 0x7faa500cbb70 2023-11-28T18:40:40.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.219+0000 7faa557e1640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa340668d0 0x7faa34068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:40.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.219+0000 7faa557e1640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa340668d0 0x7faa34068d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faa400ad000 tx=0x7faa40004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:40.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.223+0000 7faa467fc640 1 -- 172.21.15.7:0/3525080771 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faa4c051490 con 0x7faa500cbb70 2023-11-28T18:40:40.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.651+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa500cffa0 con 0x7faa500cbb70 2023-11-28T18:40:40.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.651+0000 7faa467fc640 1 -- 172.21.15.7:0/3525080771 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5771 (secure 0 0 0) 0x7faa4c054ab0 con 0x7faa500cbb70 2023-11-28T18:40:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:40.659 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:39.402383+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:40.659 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 76 2023-11-28T18:40:40.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa340668d0 msgr2=0x7faa34068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:40.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa340668d0 0x7faa34068d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faa400ad000 tx=0x7faa40004740 comp rx=0 tx=0).stop 2023-11-28T18:40:40.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa500cbb70 msgr2=0x7faa500c8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:40.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa500cbb70 0x7faa500c8f10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa4c00b420 tx=0x7faa4c00b8e0 comp rx=0 tx=0).stop 2023-11-28T18:40:40.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 shutdown_connections 2023-11-28T18:40:40.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faa340668d0 0x7faa34068d90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa501542e0 0x7faa500c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa500cc810 0x7faa500c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 --2- 172.21.15.7:0/3525080771 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faa500cbb70 0x7faa500c8f10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 >> 172.21.15.7:0/3525080771 conn(0x7faa500717e0 msgr2=0x7faa50142db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 shutdown_connections 2023-11-28T18:40:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:40.659+0000 7faa58246640 1 -- 172.21.15.7:0/3525080771 wait complete. 2023-11-28T18:40:40.716 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=5307 2023-11-28T18:40:40.716 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2023-11-28T18:40:41.716 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-11-28T18:40:41.717 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T18:40:41.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T18:40:41.723 INFO:tasks.ceph.mds.a:Started 2023-11-28T18:40:41.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:41.875 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2023-11-28T18:40:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.963+0000 7fd485bf8640 1 Processor -- start 2023-11-28T18:40:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.963+0000 7fd485bf8640 1 -- start start 2023-11-28T18:40:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.963+0000 7fd485bf8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:41.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd485bf8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4800ca920 con 0x7fd4800cc810 2023-11-28T18:40:41.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd485bf8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4800c4cd0 con 0x7fd4801542e0 2023-11-28T18:40:41.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd485bf8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd480071d50 con 0x7fd4800cbb70 2023-11-28T18:40:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd47ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd4800cc810 0x7fd480072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33698/0 (socket says 172.21.15.7:33698) 2023-11-28T18:40:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd47ffff640 1 -- 172.21.15.7:0/1342819556 learned_addr learned my addr 172.21.15.7:0/1342819556 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:40:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd47f7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd4801542e0 0x7fd48015e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:41104/0 (socket says 172.21.15.7:41104) 2023-11-28T18:40:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.963+0000 7fd47effd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd47effd640 1 -- 172.21.15.7:0/1342819556 >> v1:172.21.15.193:6789/0 conn(0x7fd4801542e0 legacy=0x7fd48015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd47effd640 1 -- 172.21.15.7:0/1342819556 >> v1:172.21.15.7:6789/0 conn(0x7fd4800cc810 legacy=0x7fd480072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.967+0000 7fd47effd640 1 -- 172.21.15.7:0/1342819556 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4800cab70 con 0x7fd4800cbb70 2023-11-28T18:40:41.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd47effd640 1 --2- 172.21.15.7:0/1342819556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800cc2d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd4740097f0 tx=0x7fd47401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3394835168216244 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:41.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd47e7fc640 1 -- 172.21.15.7:0/1342819556 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4740ac630 con 0x7fd4800cbb70 2023-11-28T18:40:41.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd47e7fc640 1 -- 172.21.15.7:0/1342819556 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4740ac790 con 0x7fd4800cbb70 2023-11-28T18:40:41.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd47e7fc640 1 -- 172.21.15.7:0/1342819556 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4740acb20 con 0x7fd4800cbb70 2023-11-28T18:40:41.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd485bf8640 1 -- 172.21.15.7:0/1342819556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 msgr2=0x7fd4800cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:41.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd485bf8640 1 --2- 172.21.15.7:0/1342819556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800cc2d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd4740097f0 tx=0x7fd47401c4c0 comp rx=0 tx=0).stop 2023-11-28T18:40:41.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd485bf8640 1 -- 172.21.15.7:0/1342819556 shutdown_connections 2023-11-28T18:40:41.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd485bf8640 1 --2- 172.21.15.7:0/1342819556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800cc2d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:41.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd485bf8640 1 -- 172.21.15.7:0/1342819556 >> 172.21.15.7:0/1342819556 conn(0x7fd4800717e0 msgr2=0x7fd4800662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd485bf8640 1 -- 172.21.15.7:0/1342819556 shutdown_connections 2023-11-28T18:40:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.975+0000 7fd485bf8640 1 -- 172.21.15.7:0/1342819556 wait complete. 2023-11-28T18:40:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd485bf8640 1 Processor -- start 2023-11-28T18:40:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd485bf8640 1 -- start start 2023-11-28T18:40:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd485bf8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd485bf8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4800cc810 0x7fd4800c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd485bf8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4801542e0 0x7fd4801f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:41.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd485bf8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd480072030 con 0x7fd4801542e0 2023-11-28T18:40:41.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd485bf8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd480144f00 con 0x7fd4800cc810 2023-11-28T18:40:41.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd485bf8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4801f64e0 con 0x7fd4800cbb70 2023-11-28T18:40:41.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd47f7fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:41.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd47f7fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:52946/0 (socket says 172.21.15.7:52946) 2023-11-28T18:40:41.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.979+0000 7fd47f7fe640 1 -- 172.21.15.7:0/1514229904 learned_addr learned my addr 172.21.15.7:0/1514229904 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:41.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd47effd640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4800cc810 0x7fd4800c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd47f7fe640 1 -- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4800cc810 msgr2=0x7fd4800c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd47f7fe640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4800cc810 0x7fd4800c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd47f7fe640 1 -- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4801542e0 msgr2=0x7fd4801f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd47f7fe640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4801542e0 0x7fd4801f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd47f7fe640 1 -- 172.21.15.7:0/1514229904 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd474009420 con 0x7fd4800cbb70 2023-11-28T18:40:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd47effd640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4800cc810 0x7fd4800c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:40:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd47f7fe640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800c7850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd47000c460 tx=0x7fd47000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd467fff640 1 -- 172.21.15.7:0/1514229904 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470007480 con 0x7fd4800cbb70 2023-11-28T18:40:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd467fff640 1 -- 172.21.15.7:0/1514229904 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4700075e0 con 0x7fd4800cbb70 2023-11-28T18:40:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd467fff640 1 -- 172.21.15.7:0/1514229904 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470004a10 con 0x7fd4800cbb70 2023-11-28T18:40:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4800c7f70 con 0x7fd4800cbb70 2023-11-28T18:40:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.983+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4800d0310 con 0x7fd4800cbb70 2023-11-28T18:40:41.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.987+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48006c670 con 0x7fd4800cbb70 2023-11-28T18:40:41.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.987+0000 7fd467fff640 1 -- 172.21.15.7:0/1514229904 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd470004b70 con 0x7fd4800cbb70 2023-11-28T18:40:41.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.987+0000 7fd467fff640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd46c0668d0 0x7fd46c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:41.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.987+0000 7fd467fff640 1 -- 172.21.15.7:0/1514229904 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fd470083a30 con 0x7fd4800cbb70 2023-11-28T18:40:41.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.991+0000 7fd47effd640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd46c0668d0 0x7fd46c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:41.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.991+0000 7fd47effd640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd46c0668d0 0x7fd46c068d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd4740ae000 tx=0x7fd4740a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:41.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:41.991+0000 7fd467fff640 1 -- 172.21.15.7:0/1514229904 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd4700507c0 con 0x7fd4800cbb70 2023-11-28T18:40:42.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.407+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4800cb710 con 0x7fd4800cbb70 2023-11-28T18:40:42.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.407+0000 7fd467fff640 1 -- 172.21.15.7:0/1514229904 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5771 (secure 0 0 0) 0x7fd470018070 con 0x7fd4800cbb70 2023-11-28T18:40:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:42.412 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:39.402383+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:42.413 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 76 2023-11-28T18:40:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd46c0668d0 msgr2=0x7fd46c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd46c0668d0 0x7fd46c068d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd4740ae000 tx=0x7fd4740a7000 comp rx=0 tx=0).stop 2023-11-28T18:40:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 msgr2=0x7fd4800c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800c7850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd47000c460 tx=0x7fd47000c920 comp rx=0 tx=0).stop 2023-11-28T18:40:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 shutdown_connections 2023-11-28T18:40:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd46c0668d0 0x7fd46c068d90 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd4740ae000 tx=0x7fd4740a7000 comp rx=0 tx=0).stop 2023-11-28T18:40:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4801542e0 0x7fd4801f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:42.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4800cc810 0x7fd4800c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:42.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 --2- 172.21.15.7:0/1514229904 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4800cbb70 0x7fd4800c7850 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:42.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.411+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 >> 172.21.15.7:0/1514229904 conn(0x7fd4800717e0 msgr2=0x7fd4801597e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:42.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.415+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 shutdown_connections 2023-11-28T18:40:42.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:42.415+0000 7fd485bf8640 1 -- 172.21.15.7:0/1514229904 wait complete. 2023-11-28T18:40:42.453 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-11-28T18:40:44.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:40:44.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.691+0000 7f93f6d65640 1 Processor -- start 2023-11-28T18:40:44.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.691+0000 7f93f6d65640 1 -- start start 2023-11-28T18:40:44.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.691+0000 7f93f6d65640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f93f00cbb70 0x7f93f00cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:44.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.691+0000 7f93f6d65640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f00cc810 0x7f93f0072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:44.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.691+0000 7f93f6d65640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f00ca920 con 0x7f93f01542e0 2023-11-28T18:40:44.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.691+0000 7f93f6d65640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f0071d50 con 0x7f93f00cbb70 2023-11-28T18:40:44.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.691+0000 7f93f6d65640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0072030 con 0x7f93f00cc810 2023-11-28T18:40:44.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.695+0000 7f93f5302640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f00cc810 0x7f93f0072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:44.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.695+0000 7f93f5302640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f00cc810 0x7f93f0072450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:60182/0 (socket says 172.21.15.7:60182) 2023-11-28T18:40:44.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.695+0000 7f93f5302640 1 -- 172.21.15.7:0/1020682496 learned_addr learned my addr 172.21.15.7:0/1020682496 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:44.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.695+0000 7f93f5302640 1 -- 172.21.15.7:0/1020682496 >> v2:172.21.15.193:3300/0 conn(0x7f93f00cbb70 msgr2=0x7f93f00cc2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T18:40:44.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.695+0000 7f93f5302640 1 --2- 172.21.15.7:0/1020682496 >> v2:172.21.15.193:3300/0 conn(0x7f93f00cbb70 0x7f93f00cc2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:44.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.695+0000 7f93f5302640 1 -- 172.21.15.7:0/1020682496 >> v1:172.21.15.193:6789/0 conn(0x7f93f01542e0 legacy=0x7f93f015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:40:44.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.695+0000 7f93f5302640 1 -- 172.21.15.7:0/1020682496 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f00c4cd0 con 0x7f93f00cc810 2023-11-28T18:40:44.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f5302640 1 --2- 172.21.15.7:0/1020682496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f00cc810 0x7f93f0072450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009e50 tx=0x7f93e801c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20fdfad780c76def server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93e77fe640 1 -- 172.21.15.7:0/1020682496 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e80a5780 con 0x7f93f00cc810 2023-11-28T18:40:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93e77fe640 1 -- 172.21.15.7:0/1020682496 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e80a58e0 con 0x7f93f00cc810 2023-11-28T18:40:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93e77fe640 1 -- 172.21.15.7:0/1020682496 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e80a5c70 con 0x7f93f00cc810 2023-11-28T18:40:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f6d65640 1 -- 172.21.15.7:0/1020682496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f00cc810 msgr2=0x7f93f0072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f6d65640 1 --2- 172.21.15.7:0/1020682496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f00cc810 0x7f93f0072450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009e50 tx=0x7f93e801c9f0 comp rx=0 tx=0).stop 2023-11-28T18:40:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f6d65640 1 -- 172.21.15.7:0/1020682496 shutdown_connections 2023-11-28T18:40:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f6d65640 1 --2- 172.21.15.7:0/1020682496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f00cc810 0x7f93f0072450 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f6d65640 1 --2- 172.21.15.7:0/1020682496 >> v2:172.21.15.193:3300/0 conn(0x7f93f00cbb70 0x7f93f00cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f6d65640 1 -- 172.21.15.7:0/1020682496 >> 172.21.15.7:0/1020682496 conn(0x7f93f00717e0 msgr2=0x7f93f00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:44.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f6d65640 1 -- 172.21.15.7:0/1020682496 shutdown_connections 2023-11-28T18:40:44.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.699+0000 7f93f6d65640 1 -- 172.21.15.7:0/1020682496 wait complete. 2023-11-28T18:40:44.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f6d65640 1 Processor -- start 2023-11-28T18:40:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f6d65640 1 -- start start 2023-11-28T18:40:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f6d65640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f00cbb70 0x7f93f0200080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f6d65640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93f00cc810 0x7f93f0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f6d65640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f01542e0 0x7f93f0150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f6d65640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f01511d0 con 0x7f93f00cbb70 2023-11-28T18:40:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f6d65640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f02005c0 con 0x7f93f00cc810 2023-11-28T18:40:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f6d65640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f014e6c0 con 0x7f93f01542e0 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93e7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93f00cc810 0x7f93f0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93e7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93f00cc810 0x7f93f0155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:50576/0 (socket says 172.21.15.7:50576) 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93e7fff640 1 -- 172.21.15.7:0/3587234443 learned_addr learned my addr 172.21.15.7:0/3587234443 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.703+0000 7f93f5302640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f01542e0 0x7f93f0150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93f4b01640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f00cbb70 0x7f93f0200080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e7fff640 1 -- 172.21.15.7:0/3587234443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f01542e0 msgr2=0x7f93f0150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e7fff640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f01542e0 0x7f93f0150540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e7fff640 1 -- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f00cbb70 msgr2=0x7f93f0200080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e7fff640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f00cbb70 0x7f93f0200080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e7fff640 1 -- 172.21.15.7:0/3587234443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e8009b30 con 0x7f93f00cc810 2023-11-28T18:40:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e7fff640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93f00cc810 0x7f93f0155fb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f93e000b4d0 tx=0x7f93e000b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e57fa640 1 -- 172.21.15.7:0/3587234443 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0010870 con 0x7f93f00cc810 2023-11-28T18:40:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e57fa640 1 -- 172.21.15.7:0/3587234443 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e00109f0 con 0x7f93f00cc810 2023-11-28T18:40:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93e57fa640 1 -- 172.21.15.7:0/3587234443 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0010da0 con 0x7f93f00cc810 2023-11-28T18:40:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f00c9610 con 0x7f93f00cc810 2023-11-28T18:40:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f00c98b0 con 0x7f93f00cc810 2023-11-28T18:40:44.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.707+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f006c670 con 0x7f93f00cc810 2023-11-28T18:40:44.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.711+0000 7f93e57fa640 1 -- 172.21.15.7:0/3587234443 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f93e0019450 con 0x7f93f00cc810 2023-11-28T18:40:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.711+0000 7f93e57fa640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93dc066900 0x7f93dc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:40:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.711+0000 7f93e57fa640 1 -- 172.21.15.7:0/3587234443 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f93e0084730 con 0x7f93f00cc810 2023-11-28T18:40:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.711+0000 7f93f4b01640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93dc066900 0x7f93dc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:40:44.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.715+0000 7f93f4b01640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93dc066900 0x7f93dc068dc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00b810 tx=0x7f93ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:40:44.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:44.719+0000 7f93e57fa640 1 -- 172.21.15.7:0/3587234443 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f93e00515f0 con 0x7f93f00cc810 2023-11-28T18:40:45.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.103+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93f004e890 con 0x7f93f00cc810 2023-11-28T18:40:45.108 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:40:45.108 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:40:39.402383+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:40:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 77 2023-11-28T18:40:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.103+0000 7f93e57fa640 1 -- 172.21.15.7:0/3587234443 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f93e0054c10 con 0x7f93f00cc810 2023-11-28T18:40:45.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93dc066900 msgr2=0x7f93dc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:45.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93dc066900 0x7f93dc068dc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00b810 tx=0x7f93ec008040 comp rx=0 tx=0).stop 2023-11-28T18:40:45.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93f00cc810 msgr2=0x7f93f0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:40:45.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93f00cc810 0x7f93f0155fb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f93e000b4d0 tx=0x7f93e000b990 comp rx=0 tx=0).stop 2023-11-28T18:40:45.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 shutdown_connections 2023-11-28T18:40:45.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93dc066900 0x7f93dc068dc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:45.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93f01542e0 0x7f93f0150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:45.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93f00cc810 0x7f93f0155fb0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:45.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 --2- 172.21.15.7:0/3587234443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f00cbb70 0x7f93f0200080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:40:45.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 >> 172.21.15.7:0/3587234443 conn(0x7f93f00717e0 msgr2=0x7f93f0157c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:40:45.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.107+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 shutdown_connections 2023-11-28T18:40:45.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:40:45.111+0000 7f93f6d65640 1 -- 172.21.15.7:0/3587234443 wait complete. 2023-11-28T18:40:45.148 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-11-28T18:40:45.148 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-11-28T18:41:03.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:41:03.622 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:41:33.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:41:33.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:41:52.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:41:52.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.625+0000 7f3b6e348640 1 Processor -- start 2023-11-28T18:41:52.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.625+0000 7f3b6e348640 1 -- start start 2023-11-28T18:41:52.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.625+0000 7f3b6e348640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600dd5b0 0x7f3b600e3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:52.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.625+0000 7f3b6e348640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b60007d80 con 0x7f3b6004d600 2023-11-28T18:41:52.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.625+0000 7f3b6e348640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b600e6840 con 0x7f3b60008210 2023-11-28T18:41:52.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.625+0000 7f3b6e348640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600e6b20 con 0x7f3b600dd5b0 2023-11-28T18:41:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.625+0000 7f3b6cb45640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f3b60008210 0x7f3b60008650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:51784/0 (socket says 172.21.15.7:51784) 2023-11-28T18:41:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.625+0000 7f3b6cb45640 1 -- 172.21.15.7:0/2899619624 learned_addr learned my addr 172.21.15.7:0/2899619624 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577583285 0 0) 0x7f3b60007d80 con 0x7f3b6004d600 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b54003530 con 0x7f3b6004d600 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727884790 0 0) 0x7f3b54003530 con 0x7f3b6004d600 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600dd5b0 msgr2=0x7f3b600e3b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600dd5b0 0x7f3b600e3b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 >> v1:172.21.15.193:6789/0 conn(0x7f3b60008210 legacy=0x7f3b60008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600e6e00 con 0x7f3b6004d600 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f3b64001680 con 0x7f3b6004d600 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 >> v1:172.21.15.7:6789/0 conn(0x7f3b6004d600 legacy=0x7f3b6004da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b540039b0 0x7f3b54003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b540047d0 0x7f3b54004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b54005390 0x7f3b5400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600e6b20 con 0x7f3b540039b0 2023-11-28T18:41:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5401d1c0 con 0x7f3b540047d0 2023-11-28T18:41:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5401d380 con 0x7f3b54005390 2023-11-28T18:41:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b6db47640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b54005390 0x7f3b5400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b6db47640 1 -- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b540047d0 msgr2=0x7f3b54004c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:41:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b6db47640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b540047d0 0x7f3b54004c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b6db47640 1 -- 172.21.15.7:0/2899619624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b540039b0 msgr2=0x7f3b54003db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:41:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b6db47640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b540039b0 0x7f3b54003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.629+0000 7f3b6db47640 1 -- 172.21.15.7:0/2899619624 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600e6e00 con 0x7f3b54005390 2023-11-28T18:41:52.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.633+0000 7f3b6db47640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b54005390 0x7f3b5400b990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400c160 tx=0x7f3b6401cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6f118bc415d6b6d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:52.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.633+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b640a5a60 con 0x7f3b54005390 2023-11-28T18:41:52.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.633+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b640a5be0 con 0x7f3b54005390 2023-11-28T18:41:52.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.633+0000 7f3b5ffff640 1 -- 172.21.15.7:0/2899619624 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b640067e0 con 0x7f3b54005390 2023-11-28T18:41:52.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.633+0000 7f3b6e348640 1 -- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b54005390 msgr2=0x7f3b5400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:52.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.633+0000 7f3b6e348640 1 --2- 172.21.15.7:0/2899619624 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b54005390 0x7f3b5400b990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400c160 tx=0x7f3b6401cbb0 comp rx=0 tx=0).stop 2023-11-28T18:41:52.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.633+0000 7f3b6db47640 1 -- 172.21.15.7:0/2899619624 reap_dead start 2023-11-28T18:41:52.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.637+0000 7f3b6e348640 1 -- 172.21.15.7:0/2899619624 shutdown_connections 2023-11-28T18:41:52.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.637+0000 7f3b6e348640 1 -- 172.21.15.7:0/2899619624 >> 172.21.15.7:0/2899619624 conn(0x7f3b6000e650 msgr2=0x7f3b60009d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:41:52.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.637+0000 7f3b6e348640 1 -- 172.21.15.7:0/2899619624 shutdown_connections 2023-11-28T18:41:52.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.637+0000 7f3b6e348640 1 -- 172.21.15.7:0/2899619624 wait complete. 2023-11-28T18:41:52.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.641+0000 7f3b6e348640 1 Processor -- start 2023-11-28T18:41:52.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.649+0000 7f3b6e348640 1 -- start start 2023-11-28T18:41:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.649+0000 7f3b6e348640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b54005390 0x7f3b600da700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.649+0000 7f3b6e348640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600e2f00 0x7f3b600d3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.649+0000 7f3b6e348640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b600e1320 0x7f3b600e17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.649+0000 7f3b6e348640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6016d540 con 0x7f3b54005390 2023-11-28T18:41:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.649+0000 7f3b6e348640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600e2600 con 0x7f3b600e1320 2023-11-28T18:41:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.649+0000 7f3b6e348640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b60050e40 con 0x7f3b600e2f00 2023-11-28T18:41:52.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6db47640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b600e1320 0x7f3b600e17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:52.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6db47640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b600e1320 0x7f3b600e17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:39210/0 (socket says 172.21.15.7:39210) 2023-11-28T18:41:52.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6db47640 1 -- 172.21.15.7:0/4179698645 learned_addr learned my addr 172.21.15.7:0/4179698645 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:41:52.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6cb45640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600e2f00 0x7f3b600d3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:52.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6cb45640 1 -- 172.21.15.7:0/4179698645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b600e1320 msgr2=0x7f3b600e17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:52.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6cb45640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b600e1320 0x7f3b600e17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:52.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6cb45640 1 -- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b54005390 msgr2=0x7f3b600da700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:52.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6cb45640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b54005390 0x7f3b600da700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:52.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.653+0000 7f3b6cb45640 1 -- 172.21.15.7:0/4179698645 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600511b0 con 0x7f3b600e2f00 2023-11-28T18:41:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.657+0000 7f3b6cb45640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600e2f00 0x7f3b600d3960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800c410 tx=0x7f3b5800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.657+0000 7f3b5dffb640 1 -- 172.21.15.7:0/4179698645 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b58005cf0 con 0x7f3b600e2f00 2023-11-28T18:41:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.657+0000 7f3b5dffb640 1 -- 172.21.15.7:0/4179698645 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b58005e50 con 0x7f3b600e2f00 2023-11-28T18:41:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.657+0000 7f3b5dffb640 1 -- 172.21.15.7:0/4179698645 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b58007310 con 0x7f3b600e2f00 2023-11-28T18:41:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.657+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b600e2af0 con 0x7f3b600e2f00 2023-11-28T18:41:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.657+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b600e2810 con 0x7f3b600e2f00 2023-11-28T18:41:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.661+0000 7f3b5dffb640 1 -- 172.21.15.7:0/4179698645 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3b58007470 con 0x7f3b600e2f00 2023-11-28T18:41:52.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.661+0000 7f3b5dffb640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:52.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.661+0000 7f3b5dffb640 1 -- 172.21.15.7:0/4179698645 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f3b58083c20 con 0x7f3b600e2f00 2023-11-28T18:41:52.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.665+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b600511b0 con 0x7f3b600e2f00 2023-11-28T18:41:52.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.665+0000 7f3b6d346640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.665+0000 7f3b6d346640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3b680501d0 tx=0x7f3b6807b800 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:52.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:52.673+0000 7f3b5dffb640 1 -- 172.21.15.7:0/4179698645 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3b600511b0 con 0x7f3b600e2f00 2023-11-28T18:41:53.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.065+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b600511b0 con 0x7f3b600e2f00 2023-11-28T18:41:53.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.065+0000 7f3b5dffb640 1 -- 172.21.15.7:0/4179698645 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f3b600511b0 con 0x7f3b600e2f00 2023-11-28T18:41:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:41:53.068 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:41:38.479534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:41:53.068 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 83 2023-11-28T18:41:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 msgr2=0x7f3b4c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3b680501d0 tx=0x7f3b6807b800 comp rx=0 tx=0).stop 2023-11-28T18:41:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600e2f00 msgr2=0x7f3b600d3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600e2f00 0x7f3b600d3960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800c410 tx=0x7f3b5800c8d0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 shutdown_connections 2023-11-28T18:41:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b600e1320 0x7f3b600e17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b600e2f00 0x7f3b600d3960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 --2- 172.21.15.7:0/4179698645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b54005390 0x7f3b600da700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 >> 172.21.15.7:0/4179698645 conn(0x7f3b6000e650 msgr2=0x7f3b600e5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:41:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 shutdown_connections 2023-11-28T18:41:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.069+0000 7f3b6e348640 1 -- 172.21.15.7:0/4179698645 wait complete. 2023-11-28T18:41:53.110 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=1) 2023-11-28T18:41:53.110 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T18:41:53.110 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:41:53.130 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T18:41:53.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:41:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.385+0000 7f742f99d640 1 Processor -- start 2023-11-28T18:41:53.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.385+0000 7f742f99d640 1 -- start start 2023-11-28T18:41:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742f99d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7428066e00 0x7f74280c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742f99d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f74280c85f0 0x7f74280c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742f99d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74280c7880 con 0x7f74280c32d0 2023-11-28T18:41:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742f99d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74280c5600 con 0x7f74280c85f0 2023-11-28T18:41:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742f99d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74280c7bd0 con 0x7f7428066e00 2023-11-28T18:41:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742df3a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f74280c85f0 0x7f74280c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742df3a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f74280c85f0 0x7f74280c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46464/0 (socket says 172.21.15.7:46464) 2023-11-28T18:41:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742df3a640 1 -- 172.21.15.7:0/473079012 learned_addr learned my addr 172.21.15.7:0/473079012 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:41:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.389+0000 7f742d739640 1 --2- 172.21.15.7:0/473079012 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7428066e00 0x7f74280c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.393+0000 7f742df3a640 1 -- 172.21.15.7:0/473079012 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7428066e00 msgr2=0x7f74280c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:41:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.393+0000 7f742df3a640 1 --2- 172.21.15.7:0/473079012 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7428066e00 0x7f74280c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.393+0000 7f742df3a640 1 -- 172.21.15.7:0/473079012 >> v1:172.21.15.193:6789/0 conn(0x7f74280c32d0 legacy=0x7f742814fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.393+0000 7f742df3a640 1 -- 172.21.15.7:0/473079012 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74280c74a0 con 0x7f74280c85f0 2023-11-28T18:41:53.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.397+0000 7f742df3a640 1 --2- 172.21.15.7:0/473079012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74280c85f0 0x7f74280c8a20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7420009870 tx=0x7f742002d4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeab23631096a3f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:53.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.397+0000 7f741ffff640 1 -- 172.21.15.7:0/473079012 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74200bb660 con 0x7f74280c85f0 2023-11-28T18:41:53.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.397+0000 7f741ffff640 1 -- 172.21.15.7:0/473079012 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74200bb7c0 con 0x7f74280c85f0 2023-11-28T18:41:53.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.397+0000 7f741ffff640 1 -- 172.21.15.7:0/473079012 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74200bbb50 con 0x7f74280c85f0 2023-11-28T18:41:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.405+0000 7f742f99d640 1 -- 172.21.15.7:0/473079012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74280c85f0 msgr2=0x7f74280c8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.405+0000 7f742f99d640 1 --2- 172.21.15.7:0/473079012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74280c85f0 0x7f74280c8a20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7420009870 tx=0x7f742002d4e0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.405+0000 7f742f99d640 1 -- 172.21.15.7:0/473079012 shutdown_connections 2023-11-28T18:41:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.405+0000 7f742f99d640 1 --2- 172.21.15.7:0/473079012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74280c85f0 0x7f74280c8a20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.405+0000 7f742f99d640 1 --2- 172.21.15.7:0/473079012 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7428066e00 0x7f74280c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.405+0000 7f742f99d640 1 -- 172.21.15.7:0/473079012 >> 172.21.15.7:0/473079012 conn(0x7f74280662e0 msgr2=0x7f74280666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:41:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.405+0000 7f742f99d640 1 -- 172.21.15.7:0/473079012 shutdown_connections 2023-11-28T18:41:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.409+0000 7f742f99d640 1 -- 172.21.15.7:0/473079012 wait complete. 2023-11-28T18:41:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.409+0000 7f742f99d640 1 Processor -- start 2023-11-28T18:41:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.413+0000 7f742f99d640 1 -- start start 2023-11-28T18:41:53.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742f99d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7428066e00 0x7f74280cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:53.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742f99d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f74280c32d0 0x7f742814ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:53.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742f99d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74280c85f0 0x7f74281e3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:53.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742f99d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74280cb740 con 0x7f7428066e00 2023-11-28T18:41:53.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742f99d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74281e3a00 con 0x7f74280c85f0 2023-11-28T18:41:53.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742f99d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74281e36d0 con 0x7f74280c32d0 2023-11-28T18:41:53.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742df3a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74280c85f0 0x7f74281e3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742df3a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74280c85f0 0x7f74281e3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:39222/0 (socket says 172.21.15.7:39222) 2023-11-28T18:41:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.417+0000 7f742df3a640 1 -- 172.21.15.7:0/3141980753 learned_addr learned my addr 172.21.15.7:0/3141980753 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:41:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742cf38640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f74280c32d0 0x7f742814ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742d739640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7428066e00 0x7f74280cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742df3a640 1 -- 172.21.15.7:0/3141980753 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f74280c32d0 msgr2=0x7f742814ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:53.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742df3a640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f74280c32d0 0x7f742814ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742df3a640 1 -- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7428066e00 msgr2=0x7f74280cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:53.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742df3a640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7428066e00 0x7f74280cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742df3a640 1 -- 172.21.15.7:0/3141980753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74200094d0 con 0x7f74280c85f0 2023-11-28T18:41:53.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742df3a640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74280c85f0 0x7f74281e3e90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f74200bd040 tx=0x7f742002bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:53.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f741dffb640 1 -- 172.21.15.7:0/3141980753 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74200c4800 con 0x7f74280c85f0 2023-11-28T18:41:53.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f741dffb640 1 -- 172.21.15.7:0/3141980753 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74200c4980 con 0x7f74280c85f0 2023-11-28T18:41:53.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f741dffb640 1 -- 172.21.15.7:0/3141980753 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74200c4d30 con 0x7f74280c85f0 2023-11-28T18:41:53.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.421+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7428159fa0 con 0x7f74280c85f0 2023-11-28T18:41:53.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.425+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74280cc080 con 0x7f74280c85f0 2023-11-28T18:41:53.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.425+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74281e3b70 con 0x7f74280c85f0 2023-11-28T18:41:53.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.425+0000 7f741dffb640 1 -- 172.21.15.7:0/3141980753 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f74200c3380 con 0x7f74280c85f0 2023-11-28T18:41:53.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.429+0000 7f741dffb640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f740c066920 0x7f740c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:53.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.429+0000 7f741dffb640 1 -- 172.21.15.7:0/3141980753 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f74200b7040 con 0x7f74280c85f0 2023-11-28T18:41:53.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.429+0000 7f742d739640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f740c066920 0x7f740c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.429+0000 7f742d739640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f740c066920 0x7f740c068de0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f74240113e0 tx=0x7f742400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:53.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-11-28T18:41:53.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.437+0000 7f741dffb640 1 -- 172.21.15.7:0/3141980753 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f74200fc600 con 0x7f74280c85f0 2023-11-28T18:41:53.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.817+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f742804e890 con 0x7f74280c85f0 2023-11-28T18:41:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.817+0000 7f741dffb640 1 -- 172.21.15.7:0/3141980753 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f74200ffc20 con 0x7f74280c85f0 2023-11-28T18:41:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:41:53.822 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:41:38.479534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:41:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 83 2023-11-28T18:41:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.821+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f740c066920 msgr2=0x7f740c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.821+0000 7f742f99d640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f740c066920 0x7f740c068de0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f74240113e0 tx=0x7f742400b040 comp rx=0 tx=0).stop 2023-11-28T18:41:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.821+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74280c85f0 msgr2=0x7f74281e3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.821+0000 7f742f99d640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74280c85f0 0x7f74281e3e90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f74200bd040 tx=0x7f742002bd20 comp rx=0 tx=0).stop 2023-11-28T18:41:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.825+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 shutdown_connections 2023-11-28T18:41:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.825+0000 7f742f99d640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f740c066920 0x7f740c068de0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.825+0000 7f742f99d640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74280c85f0 0x7f74281e3e90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.825+0000 7f742f99d640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f74280c32d0 0x7f742814ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.825+0000 7f742f99d640 1 --2- 172.21.15.7:0/3141980753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7428066e00 0x7f74280cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.825+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 >> 172.21.15.7:0/3141980753 conn(0x7f74280662e0 msgr2=0x7f7428071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:41:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.825+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 shutdown_connections 2023-11-28T18:41:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:53.825+0000 7f742f99d640 1 -- 172.21.15.7:0/3141980753 wait complete. 2023-11-28T18:41:53.865 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:41:55.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:41:56.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff01c1aa640 1 Processor -- start 2023-11-28T18:41:56.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff01c1aa640 1 -- start start 2023-11-28T18:41:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff01c1aa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb3d0 0x7ff01415db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff01c1aa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01415d3f0 con 0x7ff0140caab0 2023-11-28T18:41:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff01c1aa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01415d6d0 con 0x7ff0140cb7b0 2023-11-28T18:41:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff01c1aa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0140c4a80 con 0x7ff0140cb3d0 2023-11-28T18:41:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff019f46640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb3d0 0x7ff01415db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff019f46640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb3d0 0x7ff01415db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:39122/0 (socket says 172.21.15.7:39122) 2023-11-28T18:41:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff019f46640 1 -- 172.21.15.7:0/4051721682 learned_addr learned my addr 172.21.15.7:0/4051721682 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:41:56.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff019f46640 1 -- 172.21.15.7:0/4051721682 >> v1:172.21.15.193:6789/0 conn(0x7ff0140cb7b0 legacy=0x7ff0140ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:41:56.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.185+0000 7ff019f46640 1 -- 172.21.15.7:0/4051721682 >> v1:172.21.15.7:6789/0 conn(0x7ff0140caab0 legacy=0x7ff01414f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:41:56.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.189+0000 7ff019f46640 1 -- 172.21.15.7:0/4051721682 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01415a3a0 con 0x7ff0140cb3d0 2023-11-28T18:41:56.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff019f46640 1 --2- 172.21.15.7:0/4051721682 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb3d0 0x7ff01415db60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff0100098a0 tx=0x7ff01001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ec21bc3b852ba74 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:56.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff018f44640 1 -- 172.21.15.7:0/4051721682 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0100ac640 con 0x7ff0140cb3d0 2023-11-28T18:41:56.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff018f44640 1 -- 172.21.15.7:0/4051721682 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0100ac7a0 con 0x7ff0140cb3d0 2023-11-28T18:41:56.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff018f44640 1 -- 172.21.15.7:0/4051721682 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0100acb30 con 0x7ff0140cb3d0 2023-11-28T18:41:56.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff01c1aa640 1 -- 172.21.15.7:0/4051721682 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb3d0 msgr2=0x7ff01415db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:56.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff01c1aa640 1 --2- 172.21.15.7:0/4051721682 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb3d0 0x7ff01415db60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff0100098a0 tx=0x7ff01001c420 comp rx=0 tx=0).stop 2023-11-28T18:41:56.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff01c1aa640 1 -- 172.21.15.7:0/4051721682 shutdown_connections 2023-11-28T18:41:56.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff01c1aa640 1 --2- 172.21.15.7:0/4051721682 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb3d0 0x7ff01415db60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:56.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.193+0000 7ff01c1aa640 1 -- 172.21.15.7:0/4051721682 >> 172.21.15.7:0/4051721682 conn(0x7ff0140662e0 msgr2=0x7ff0140666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:41:56.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.197+0000 7ff01c1aa640 1 -- 172.21.15.7:0/4051721682 shutdown_connections 2023-11-28T18:41:56.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.197+0000 7ff01c1aa640 1 -- 172.21.15.7:0/4051721682 wait complete. 2023-11-28T18:41:56.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01c1aa640 1 Processor -- start 2023-11-28T18:41:56.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01c1aa640 1 -- start start 2023-11-28T18:41:56.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01c1aa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0140caab0 0x7ff01413e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:56.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01c1aa640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0140cb3d0 0x7ff0141437c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:56.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01c1aa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb7b0 0x7ff014143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:56.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01c1aa640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0141fb680 con 0x7ff0140caab0 2023-11-28T18:41:56.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01c1aa640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff014144240 con 0x7ff0140cb3d0 2023-11-28T18:41:56.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01c1aa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff014142300 con 0x7ff0140cb7b0 2023-11-28T18:41:56.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff019f46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0140caab0 0x7ff01413e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:56.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff019f46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0140caab0 0x7ff01413e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51976/0 (socket says 172.21.15.7:51976) 2023-11-28T18:41:56.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff019f46640 1 -- 172.21.15.7:0/1514826370 learned_addr learned my addr 172.21.15.7:0/1514826370 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:41:56.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff01a747640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb7b0 0x7ff014143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:56.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.201+0000 7ff019745640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0140cb3d0 0x7ff0141437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff019f46640 1 -- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb7b0 msgr2=0x7ff014143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff019f46640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb7b0 0x7ff014143d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff019f46640 1 -- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0140cb3d0 msgr2=0x7ff0141437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff019f46640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0140cb3d0 0x7ff0141437c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff019f46640 1 -- 172.21.15.7:0/1514826370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0100094d0 con 0x7ff0140caab0 2023-11-28T18:41:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff019745640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0140cb3d0 0x7ff0141437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:41:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff019f46640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0140caab0 0x7ff01413e3f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff01001aad0 tx=0x7ff0100191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff00a7fc640 1 -- 172.21.15.7:0/1514826370 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0100b5850 con 0x7ff0140caab0 2023-11-28T18:41:56.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff00a7fc640 1 -- 172.21.15.7:0/1514826370 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0100b59b0 con 0x7ff0140caab0 2023-11-28T18:41:56.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff00a7fc640 1 -- 172.21.15.7:0/1514826370 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0100b5d20 con 0x7ff0140caab0 2023-11-28T18:41:56.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0141596f0 con 0x7ff0140caab0 2023-11-28T18:41:56.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0141faa00 con 0x7ff0140caab0 2023-11-28T18:41:56.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.205+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0141fb7c0 con 0x7ff0140caab0 2023-11-28T18:41:56.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.209+0000 7ff00a7fc640 1 -- 172.21.15.7:0/1514826370 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff0100b3070 con 0x7ff0140caab0 2023-11-28T18:41:56.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.209+0000 7ff00a7fc640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feff80669a0 0x7feff8068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:41:56.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.209+0000 7ff00a7fc640 1 -- 172.21.15.7:0/1514826370 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7ff01011ecb0 con 0x7ff0140caab0 2023-11-28T18:41:56.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.213+0000 7ff019745640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feff80669a0 0x7feff8068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:41:56.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.213+0000 7ff019745640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feff80669a0 0x7feff8068e60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff014144ad0 tx=0x7ff004008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:41:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.221+0000 7ff00a7fc640 1 -- 172.21.15.7:0/1514826370 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff0141fb7c0 con 0x7ff0140caab0 2023-11-28T18:41:56.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.649+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0141fb7c0 con 0x7ff0140caab0 2023-11-28T18:41:56.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.649+0000 7ff00a7fc640 1 -- 172.21.15.7:0/1514826370 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6568 (secure 0 0 0) 0x7ff0141fb7c0 con 0x7ff0140caab0 2023-11-28T18:41:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:41:56.654 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:41:38.479534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:41:56.654 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 83 2023-11-28T18:41:56.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.653+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feff80669a0 msgr2=0x7feff8068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:56.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.653+0000 7ff01c1aa640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feff80669a0 0x7feff8068e60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff014144ad0 tx=0x7ff004008040 comp rx=0 tx=0).stop 2023-11-28T18:41:56.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.653+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0140caab0 msgr2=0x7ff01413e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:41:56.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.653+0000 7ff01c1aa640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0140caab0 0x7ff01413e3f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff01001aad0 tx=0x7ff0100191c0 comp rx=0 tx=0).stop 2023-11-28T18:41:56.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.657+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 shutdown_connections 2023-11-28T18:41:56.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.657+0000 7ff01c1aa640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feff80669a0 0x7feff8068e60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:56.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.657+0000 7ff01c1aa640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff0140cb7b0 0x7ff014143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:56.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.657+0000 7ff01c1aa640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0140cb3d0 0x7ff0141437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:56.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.657+0000 7ff01c1aa640 1 --2- 172.21.15.7:0/1514826370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0140caab0 0x7ff01413e3f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:41:56.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.657+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 >> 172.21.15.7:0/1514826370 conn(0x7ff0140662e0 msgr2=0x7ff0140ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:41:56.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.657+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 shutdown_connections 2023-11-28T18:41:56.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:41:56.657+0000 7ff01c1aa640 1 -- 172.21.15.7:0/1514826370 wait complete. 2023-11-28T18:41:56.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:41:58.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:00.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-11-28T18:42:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.425+0000 7fbabbe17640 1 Processor -- start 2023-11-28T18:42:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.429+0000 7fbabbe17640 1 -- start start 2023-11-28T18:42:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.429+0000 7fbabbe17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbab40c32d0 0x7fbab40c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.429+0000 7fbabbe17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40c7880 con 0x7fbab4150090 2023-11-28T18:42:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.429+0000 7fbabbe17640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40c74a0 con 0x7fbab4066e00 2023-11-28T18:42:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.429+0000 7fbabbe17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab40c5600 con 0x7fbab40c32d0 2023-11-28T18:42:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.429+0000 7fbab9bb3640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fbab4066e00 0x7fbab40c85f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:54882/0 (socket says 172.21.15.7:54882) 2023-11-28T18:42:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.429+0000 7fbab93b2640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbab40c32d0 0x7fbab40c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.429+0000 7fbab9bb3640 1 -- 172.21.15.7:0/460933487 learned_addr learned my addr 172.21.15.7:0/460933487 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:42:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013366601 0 0) 0x7fbab40c74a0 con 0x7fbab4066e00 2023-11-28T18:42:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa4003530 con 0x7fbab4066e00 2023-11-28T18:42:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843029667 0 0) 0x7fbaa4003530 con 0x7fbab4066e00 2023-11-28T18:42:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 >> v2:172.21.15.193:3300/0 conn(0x7fbab40c32d0 msgr2=0x7fbab40c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 --2- 172.21.15.7:0/460933487 >> v2:172.21.15.193:3300/0 conn(0x7fbab40c32d0 0x7fbab40c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 >> v1:172.21.15.7:6789/0 conn(0x7fbab4150090 legacy=0x7fbab4155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab415a1d0 con 0x7fbab4066e00 2023-11-28T18:42:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fbaa80019e0 con 0x7fbab4066e00 2023-11-28T18:42:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 >> v1:172.21.15.193:6789/0 conn(0x7fbab4066e00 legacy=0x7fbab40c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaa40039d0 0x7fbaa4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40047d0 0x7fbaa4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbaa40052b0 0x7fbaa400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40c5600 con 0x7fbaa40047d0 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa400c1b0 con 0x7fbaa40039d0 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa400c370 con 0x7fbaa40052b0 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab9bb3640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaa40039d0 0x7fbaa4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbaba3b4640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbaa40052b0 0x7fbaa400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab93b2640 1 --2- 172.21.15.7:0/460933487 >> v2:172.21.15.193:3300/0 conn(0x7fbab40c32d0 0x7fbab40c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab9bb3640 1 -- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbaa40052b0 msgr2=0x7fbaa400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab9bb3640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbaa40052b0 0x7fbaa400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab9bb3640 1 -- 172.21.15.7:0/460933487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40047d0 msgr2=0x7fbaa4004c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab9bb3640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40047d0 0x7fbaa4004c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbab9bb3640 1 -- 172.21.15.7:0/460933487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa4003530 con 0x7fbaa40039d0 2023-11-28T18:42:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.433+0000 7fbaba3b4640 1 -- 172.21.15.7:0/460933487 reap_dead start 2023-11-28T18:42:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbab9bb3640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaa40039d0 0x7fbaa4003dd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800a780 tx=0x7fbaa801d210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e3b8c2fb60e220 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa80ad630 con 0x7fbaa40039d0 2023-11-28T18:42:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa80a8070 con 0x7fbaa40039d0 2023-11-28T18:42:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbab8bb1640 1 -- 172.21.15.7:0/460933487 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa80ada90 con 0x7fbaa40039d0 2023-11-28T18:42:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbabbe17640 1 -- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaa40039d0 msgr2=0x7fbaa4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbabbe17640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaa40039d0 0x7fbaa4003dd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800a780 tx=0x7fbaa801d210 comp rx=0 tx=0).stop 2023-11-28T18:42:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbabbe17640 1 -- 172.21.15.7:0/460933487 shutdown_connections 2023-11-28T18:42:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbabbe17640 1 --2- 172.21.15.7:0/460933487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaa40039d0 0x7fbaa4003dd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.437+0000 7fbabbe17640 1 -- 172.21.15.7:0/460933487 >> 172.21.15.7:0/460933487 conn(0x7fbab40662e0 msgr2=0x7fbab40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.441+0000 7fbabbe17640 1 -- 172.21.15.7:0/460933487 shutdown_connections 2023-11-28T18:42:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.441+0000 7fbabbe17640 1 -- 172.21.15.7:0/460933487 wait complete. 2023-11-28T18:42:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.441+0000 7fbabbe17640 1 Processor -- start 2023-11-28T18:42:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.441+0000 7fbabbe17640 1 -- start start 2023-11-28T18:42:01.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.445+0000 7fbabbe17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40039d0 0x7fbab41f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbabbe17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbab4150950 0x7fbab4071250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbabbe17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbab4148560 0x7fbab4144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbabbe17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40c7bd0 con 0x7fbaa40039d0 2023-11-28T18:42:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbabbe17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4070e30 con 0x7fbab4148560 2023-11-28T18:42:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbabbe17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4072ba0 con 0x7fbab4150950 2023-11-28T18:42:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbab4148560 0x7fbab4144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbab4148560 0x7fbab4144ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:38080/0 (socket says 172.21.15.7:38080) 2023-11-28T18:42:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 -- 172.21.15.7:0/3052611999 learned_addr learned my addr 172.21.15.7:0/3052611999 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbab9bb3640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40039d0 0x7fbab41f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 -- 172.21.15.7:0/3052611999 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbab4150950 msgr2=0x7fbab4071250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T18:42:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbab4150950 0x7fbab4071250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 -- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40039d0 msgr2=0x7fbab41f3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40039d0 0x7fbab41f3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 -- 172.21.15.7:0/3052611999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab4159fa0 con 0x7fbab4148560 2023-11-28T18:42:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbab9bb3640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40039d0 0x7fbab41f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:42:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbaba3b4640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbab4148560 0x7fbab4144ac0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbab40cb670 tx=0x7fbaac00e8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbab27fc640 1 -- 172.21.15.7:0/3052611999 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0135f0 con 0x7fbab4148560 2023-11-28T18:42:01.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbab27fc640 1 -- 172.21.15.7:0/3052611999 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac013770 con 0x7fbab4148560 2023-11-28T18:42:01.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbab27fc640 1 -- 172.21.15.7:0/3052611999 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac013b20 con 0x7fbab4148560 2023-11-28T18:42:01.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab41f3e60 con 0x7fbab4148560 2023-11-28T18:42:01.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.449+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab4072340 con 0x7fbab4148560 2023-11-28T18:42:01.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.453+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab4070f70 con 0x7fbab4148560 2023-11-28T18:42:01.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.453+0000 7fbab27fc640 1 -- 172.21.15.7:0/3052611999 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbaac013c80 con 0x7fbab4148560 2023-11-28T18:42:01.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.453+0000 7fbab27fc640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba980668d0 0x7fba98068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:01.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.453+0000 7fbab27fc640 1 -- 172.21.15.7:0/3052611999 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fbaac083ce0 con 0x7fbab4148560 2023-11-28T18:42:01.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.457+0000 7fbab9bb3640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba980668d0 0x7fba98068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:01.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.457+0000 7fbab9bb3640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba980668d0 0x7fba98068d90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40057b0 tx=0x7fbaa8007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:01.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.461+0000 7fbab27fc640 1 -- 172.21.15.7:0/3052611999 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbab4070f70 con 0x7fbab4148560 2023-11-28T18:42:01.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.861+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbab41f3e60 con 0x7fbab4148560 2023-11-28T18:42:01.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.865+0000 7fbab27fc640 1 -- 172.21.15.7:0/3052611999 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6568 (secure 0 0 0) 0x7fbab41f3e60 con 0x7fbab4148560 2023-11-28T18:42:01.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:01.869 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:41:38.479534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:01.869 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 83 2023-11-28T18:42:01.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba980668d0 msgr2=0x7fba98068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:01.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba980668d0 0x7fba98068d90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40057b0 tx=0x7fbaa8007040 comp rx=0 tx=0).stop 2023-11-28T18:42:01.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbab4148560 msgr2=0x7fbab4144ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:01.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbab4148560 0x7fbab4144ac0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbab40cb670 tx=0x7fbaac00e8a0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 shutdown_connections 2023-11-28T18:42:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fba980668d0 0x7fba98068d90 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbab4148560 0x7fbab4144ac0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbab4150950 0x7fbab4071250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 --2- 172.21.15.7:0/3052611999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbaa40039d0 0x7fbab41f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 >> 172.21.15.7:0/3052611999 conn(0x7fbab40662e0 msgr2=0x7fbab406d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 shutdown_connections 2023-11-28T18:42:01.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:01.869+0000 7fbabbe17640 1 -- 172.21.15.7:0/3052611999 wait complete. 2023-11-28T18:42:01.902 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:42:03.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.161+0000 7f010fe9c640 1 Processor -- start 2023-11-28T18:42:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.161+0000 7f010fe9c640 1 -- start start 2023-11-28T18:42:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010fe9c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f01080c8520 0x7f01080c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010fe9c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010815a920 con 0x7f01081540d0 2023-11-28T18:42:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010fe9c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010815e330 con 0x7f0108159de0 2023-11-28T18:42:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010fe9c640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010815ab70 con 0x7f01080c8520 2023-11-28T18:42:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010d437640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f01080c8520 0x7f01080c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010d437640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f01080c8520 0x7f01080c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:55350/0 (socket says 172.21.15.7:55350) 2023-11-28T18:42:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010d437640 1 -- 172.21.15.7:0/1171663207 learned_addr learned my addr 172.21.15.7:0/1171663207 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010d437640 1 -- 172.21.15.7:0/1171663207 >> v1:172.21.15.193:6789/0 conn(0x7f0108159de0 legacy=0x7f010815a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010d437640 1 -- 172.21.15.7:0/1171663207 >> v1:172.21.15.7:6789/0 conn(0x7f01081540d0 legacy=0x7f01080c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.165+0000 7f010d437640 1 -- 172.21.15.7:0/1171663207 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010815d570 con 0x7f01080c8520 2023-11-28T18:42:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.173+0000 7f010d437640 1 --2- 172.21.15.7:0/1171663207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01080c8520 0x7f01080c8990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0100009e50 tx=0x7f010001f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7ad3c939e16bcea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.173+0000 7f010cc36640 1 -- 172.21.15.7:0/1171663207 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01000af400 con 0x7f01080c8520 2023-11-28T18:42:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.173+0000 7f010cc36640 1 -- 172.21.15.7:0/1171663207 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01000ab050 con 0x7f01080c8520 2023-11-28T18:42:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.173+0000 7f010cc36640 1 -- 172.21.15.7:0/1171663207 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01000af780 con 0x7f01080c8520 2023-11-28T18:42:04.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.173+0000 7f010fe9c640 1 -- 172.21.15.7:0/1171663207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01080c8520 msgr2=0x7f01080c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.173+0000 7f010fe9c640 1 --2- 172.21.15.7:0/1171663207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01080c8520 0x7f01080c8990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0100009e50 tx=0x7f010001f1f0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.177+0000 7f010fe9c640 1 -- 172.21.15.7:0/1171663207 shutdown_connections 2023-11-28T18:42:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.177+0000 7f010fe9c640 1 --2- 172.21.15.7:0/1171663207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01080c8520 0x7f01080c8990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.177+0000 7f010fe9c640 1 -- 172.21.15.7:0/1171663207 >> 172.21.15.7:0/1171663207 conn(0x7f01080662e0 msgr2=0x7f01080666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.177+0000 7f010fe9c640 1 -- 172.21.15.7:0/1171663207 shutdown_connections 2023-11-28T18:42:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.177+0000 7f010fe9c640 1 -- 172.21.15.7:0/1171663207 wait complete. 2023-11-28T18:42:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.177+0000 7f010fe9c640 1 Processor -- start 2023-11-28T18:42:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.177+0000 7f010fe9c640 1 -- start start 2023-11-28T18:42:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010fe9c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01080c8520 0x7f0108153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010fe9c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01081540d0 0x7f010815b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010fe9c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0108159de0 0x7f010814cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010fe9c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010815c800 con 0x7f0108159de0 2023-11-28T18:42:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010fe9c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0108070cb0 con 0x7f01080c8520 2023-11-28T18:42:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010fe9c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01080703a0 con 0x7f01081540d0 2023-11-28T18:42:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010d437640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01081540d0 0x7f010815b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010d437640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01081540d0 0x7f010815b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:54664/0 (socket says 172.21.15.7:54664) 2023-11-28T18:42:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010d437640 1 -- 172.21.15.7:0/1381261905 learned_addr learned my addr 172.21.15.7:0/1381261905 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.181+0000 7f010d437640 1 -- 172.21.15.7:0/1381261905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01080c8520 msgr2=0x7f0108153060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T18:42:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f010d437640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01080c8520 0x7f0108153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f010d437640 1 -- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0108159de0 msgr2=0x7f010814cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f010d437640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0108159de0 0x7f010814cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f010d437640 1 -- 172.21.15.7:0/1381261905 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100009b30 con 0x7f01081540d0 2023-11-28T18:42:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f010d437640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01081540d0 0x7f010815b0b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f01000b2040 tx=0x7f010001bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f00fe7fc640 1 -- 172.21.15.7:0/1381261905 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01000c0070 con 0x7f01081540d0 2023-11-28T18:42:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f00fe7fc640 1 -- 172.21.15.7:0/1381261905 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100009510 con 0x7f01081540d0 2023-11-28T18:42:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f010fe9c640 1 -- 172.21.15.7:0/1381261905 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0108153930 con 0x7f01081540d0 2023-11-28T18:42:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f00fe7fc640 1 -- 172.21.15.7:0/1381261905 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01000afea0 con 0x7f01081540d0 2023-11-28T18:42:04.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.185+0000 7f010fe9c640 1 -- 172.21.15.7:0/1381261905 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010815cce0 con 0x7f01081540d0 2023-11-28T18:42:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.189+0000 7f00f3fff640 1 -- 172.21.15.7:0/1381261905 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e8005260 con 0x7f01081540d0 2023-11-28T18:42:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.189+0000 7f00fe7fc640 1 -- 172.21.15.7:0/1381261905 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f01000b6040 con 0x7f01081540d0 2023-11-28T18:42:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.189+0000 7f00fe7fc640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f00ec066900 0x7f00ec068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.189+0000 7f00fe7fc640 1 -- 172.21.15.7:0/1381261905 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f01001223f0 con 0x7f01081540d0 2023-11-28T18:42:04.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.193+0000 7f010dc38640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f00ec066900 0x7f00ec068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:04.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.193+0000 7f010dc38640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f00ec066900 0x7f00ec068dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f00f8000f70 tx=0x7f00f80079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.193+0000 7f00fe7fc640 1 -- 172.21.15.7:0/1381261905 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01000ef350 con 0x7f01081540d0 2023-11-28T18:42:04.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.561+0000 7f00f3fff640 1 -- 172.21.15.7:0/1381261905 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00e8005510 con 0x7f01081540d0 2023-11-28T18:42:04.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.561+0000 7f00fe7fc640 1 -- 172.21.15.7:0/1381261905 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f01000f2970 con 0x7f01081540d0 2023-11-28T18:42:04.562 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:04.565 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:41:38.479534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:04.565 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 83 2023-11-28T18:42:04.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 -- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f00ec066900 msgr2=0x7f00ec068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:04.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f00ec066900 0x7f00ec068dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f00f8000f70 tx=0x7f00f80079c0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 -- 172.21.15.7:0/1381261905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01081540d0 msgr2=0x7f010815b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:04.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01081540d0 0x7f010815b0b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f01000b2040 tx=0x7f010001bf90 comp rx=0 tx=0).stop 2023-11-28T18:42:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 -- 172.21.15.7:0/1381261905 shutdown_connections 2023-11-28T18:42:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f00ec066900 0x7f00ec068dc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0108159de0 0x7f010814cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01081540d0 0x7f010815b0b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 --2- 172.21.15.7:0/1381261905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01080c8520 0x7f0108153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 -- 172.21.15.7:0/1381261905 >> 172.21.15.7:0/1381261905 conn(0x7f01080662e0 msgr2=0x7f01080666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:04.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 -- 172.21.15.7:0/1381261905 shutdown_connections 2023-11-28T18:42:04.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:04.565+0000 7f00f3fff640 1 -- 172.21.15.7:0/1381261905 wait complete. 2023-11-28T18:42:04.600 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:42:06.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.837+0000 7f07bd962640 1 Processor -- start 2023-11-28T18:42:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.837+0000 7f07bd962640 1 -- start start 2023-11-28T18:42:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd962640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07b8159ce0 0x7f07b80c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd962640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b80c2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:06.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd962640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b80c6f30 con 0x7f07b814fe40 2023-11-28T18:42:06.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd962640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b8155e60 con 0x7f07b8159ce0 2023-11-28T18:42:06.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd962640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b8156140 con 0x7f07b80c7ec0 2023-11-28T18:42:06.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bc960640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f07b814fe40 0x7f07b80c6780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51178/0 (socket says 172.21.15.7:51178) 2023-11-28T18:42:06.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07affff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07b8159ce0 0x7f07b80c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:06.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07affff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07b8159ce0 0x7f07b80c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44038/0 (socket says 172.21.15.7:44038) 2023-11-28T18:42:06.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07affff640 1 -- 172.21.15.7:0/36903070 learned_addr learned my addr 172.21.15.7:0/36903070 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:06.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd161640 1 --2- 172.21.15.7:0/36903070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b80c2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:06.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd161640 1 -- 172.21.15.7:0/36903070 >> v2:172.21.15.7:3300/0 conn(0x7f07b8159ce0 msgr2=0x7f07b80c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:06.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd161640 1 --2- 172.21.15.7:0/36903070 >> v2:172.21.15.7:3300/0 conn(0x7f07b8159ce0 0x7f07b80c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:06.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.841+0000 7f07bd161640 1 -- 172.21.15.7:0/36903070 >> v1:172.21.15.7:6789/0 conn(0x7f07b814fe40 legacy=0x7f07b80c6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:06.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.845+0000 7f07bd161640 1 -- 172.21.15.7:0/36903070 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b8156420 con 0x7f07b80c7ec0 2023-11-28T18:42:06.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.845+0000 7f07affff640 1 --2- 172.21.15.7:0/36903070 >> v2:172.21.15.7:3300/0 conn(0x7f07b8159ce0 0x7f07b80c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:42:06.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.849+0000 7f07bd161640 1 --2- 172.21.15.7:0/36903070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b80c2bd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f07b000f3a0 tx=0x7f07b00234f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e28ffe3a2cbae54c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:06.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.849+0000 7f07af7fe640 1 -- 172.21.15.7:0/36903070 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b000d2d0 con 0x7f07b80c7ec0 2023-11-28T18:42:06.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.849+0000 7f07af7fe640 1 -- 172.21.15.7:0/36903070 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b00af050 con 0x7f07b80c7ec0 2023-11-28T18:42:06.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.849+0000 7f07af7fe640 1 -- 172.21.15.7:0/36903070 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b000b740 con 0x7f07b80c7ec0 2023-11-28T18:42:06.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.853+0000 7f07bd962640 1 -- 172.21.15.7:0/36903070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 msgr2=0x7f07b80c2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:06.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.853+0000 7f07bd962640 1 --2- 172.21.15.7:0/36903070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b80c2bd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f07b000f3a0 tx=0x7f07b00234f0 comp rx=0 tx=0).stop 2023-11-28T18:42:06.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.857+0000 7f07bd962640 1 -- 172.21.15.7:0/36903070 shutdown_connections 2023-11-28T18:42:06.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.857+0000 7f07bd962640 1 --2- 172.21.15.7:0/36903070 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b80c2bd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:06.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.857+0000 7f07bd962640 1 --2- 172.21.15.7:0/36903070 >> v2:172.21.15.7:3300/0 conn(0x7f07b8159ce0 0x7f07b80c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:06.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.857+0000 7f07bd962640 1 -- 172.21.15.7:0/36903070 >> 172.21.15.7:0/36903070 conn(0x7f07b806c370 msgr2=0x7f07b80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:06.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.857+0000 7f07bd962640 1 -- 172.21.15.7:0/36903070 shutdown_connections 2023-11-28T18:42:06.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.857+0000 7f07bd962640 1 -- 172.21.15.7:0/36903070 wait complete. 2023-11-28T18:42:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.861+0000 7f07bd962640 1 Processor -- start 2023-11-28T18:42:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.869+0000 7f07bd962640 1 -- start start 2023-11-28T18:42:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bd962640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b814c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bd962640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07b814fe40 0x7f07b81578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bd962640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07b8159ce0 0x7f07b81453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bd962640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b8070690 con 0x7f07b814fe40 2023-11-28T18:42:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bd962640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b8156ea0 con 0x7f07b8159ce0 2023-11-28T18:42:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bd962640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b80ca650 con 0x7f07b80c7ec0 2023-11-28T18:42:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bc960640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b814c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bc960640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b814c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:54676/0 (socket says 172.21.15.7:54676) 2023-11-28T18:42:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bc960640 1 -- 172.21.15.7:0/1258694332 learned_addr learned my addr 172.21.15.7:0/1258694332 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bc960640 1 -- 172.21.15.7:0/1258694332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07b8159ce0 msgr2=0x7f07b81453f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T18:42:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bc960640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07b8159ce0 0x7f07b81453f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bc960640 1 -- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07b814fe40 msgr2=0x7f07b81578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:06.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bc960640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07b814fe40 0x7f07b81578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:06.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.873+0000 7f07bc960640 1 -- 172.21.15.7:0/1258694332 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a800a570 con 0x7f07b80c7ec0 2023-11-28T18:42:06.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.877+0000 7f07bc960640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b814c4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f07b400b460 tx=0x7f07b400b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:06.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.877+0000 7f07ad7fa640 1 -- 172.21.15.7:0/1258694332 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4011830 con 0x7f07b80c7ec0 2023-11-28T18:42:06.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.877+0000 7f07ad7fa640 1 -- 172.21.15.7:0/1258694332 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b40119b0 con 0x7f07b80c7ec0 2023-11-28T18:42:06.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.877+0000 7f07ad7fa640 1 -- 172.21.15.7:0/1258694332 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4011d60 con 0x7f07b80c7ec0 2023-11-28T18:42:06.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.877+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b000f050 con 0x7f07b80c7ec0 2023-11-28T18:42:06.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.877+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b81f46f0 con 0x7f07b80c7ec0 2023-11-28T18:42:06.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.877+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b81f4880 con 0x7f07b80c7ec0 2023-11-28T18:42:06.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.881+0000 7f07ad7fa640 1 -- 172.21.15.7:0/1258694332 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f07b401a400 con 0x7f07b80c7ec0 2023-11-28T18:42:06.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.881+0000 7f07ad7fa640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f07a4066920 0x7f07a4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:06.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.881+0000 7f07ad7fa640 1 -- 172.21.15.7:0/1258694332 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f07b4084790 con 0x7f07b80c7ec0 2023-11-28T18:42:06.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.881+0000 7f07affff640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f07a4066920 0x7f07a4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:06.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.881+0000 7f07affff640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f07a4066920 0x7f07a4068de0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f07b8146240 tx=0x7f07a800a250 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:06.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:06.885+0000 7f07ad7fa640 1 -- 172.21.15.7:0/1258694332 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f07b40517c0 con 0x7f07b80c7ec0 2023-11-28T18:42:07.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.297+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07b804e890 con 0x7f07b80c7ec0 2023-11-28T18:42:07.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.297+0000 7f07ad7fa640 1 -- 172.21.15.7:0/1258694332 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f07b4015090 con 0x7f07b80c7ec0 2023-11-28T18:42:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:07.301 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:41:38.479534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5359,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5359":{"gid":5359,"name":"d","rank":1,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.193:6837/1742899523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1742899523},{"type":"v1","addr":"172.21.15.193:6837","nonce":1742899523}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:07.302 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 83 2023-11-28T18:42:07.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f07a4066920 msgr2=0x7f07a4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:07.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f07a4066920 0x7f07a4068de0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f07b8146240 tx=0x7f07a800a250 comp rx=0 tx=0).stop 2023-11-28T18:42:07.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 msgr2=0x7f07b814c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:07.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b814c4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f07b400b460 tx=0x7f07b400b920 comp rx=0 tx=0).stop 2023-11-28T18:42:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 shutdown_connections 2023-11-28T18:42:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f07a4066920 0x7f07a4068de0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07b8159ce0 0x7f07b81453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07b814fe40 0x7f07b81578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 --2- 172.21.15.7:0/1258694332 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07b80c7ec0 0x7f07b814c4d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.301+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 >> 172.21.15.7:0/1258694332 conn(0x7f07b806c370 msgr2=0x7f07b80c6760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:07.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.309+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 shutdown_connections 2023-11-28T18:42:07.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:07.309+0000 7f07bd962640 1 -- 172.21.15.7:0/1258694332 wait complete. 2023-11-28T18:42:07.345 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-11-28T18:42:08.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:42:08.637 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:42:09.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:09.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.589+0000 7fd8fbfff640 1 Processor -- start 2023-11-28T18:42:09.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.589+0000 7fd8fbfff640 1 -- start start 2023-11-28T18:42:09.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fbfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc151f80 0x7fd8fc15b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:09.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd8fc0c5060 0x7fd8fc15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:09.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fa7fc640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc151f80 0x7fd8fc15b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:09.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fa7fc640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc151f80 0x7fd8fc15b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:54684/0 (socket says 172.21.15.7:54684) 2023-11-28T18:42:09.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fa7fc640 1 -- 172.21.15.7:0/1071669960 learned_addr learned my addr 172.21.15.7:0/1071669960 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:09.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fbfff640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0c6c60 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fbfff640 1 -- 172.21.15.7:0/1071669960 --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8fc070aa0 con 0x7fd8fc0c5060 2023-11-28T18:42:09.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fbfff640 1 -- 172.21.15.7:0/1071669960 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0c7550 con 0x7fd8fc151f80 2023-11-28T18:42:09.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fa7fc640 1 -- 172.21.15.7:0/1071669960 >> v2:172.21.15.7:3300/0 conn(0x7fd8fc0c5060 msgr2=0x7fd8fc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:09.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fa7fc640 1 --2- 172.21.15.7:0/1071669960 >> v2:172.21.15.7:3300/0 conn(0x7fd8fc0c5060 0x7fd8fc15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fa7fc640 1 -- 172.21.15.7:0/1071669960 >> v1:172.21.15.193:6789/0 conn(0x7fd8fc0c4c80 legacy=0x7fd8fc151870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T18:42:09.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.593+0000 7fd8fa7fc640 1 -- 172.21.15.7:0/1071669960 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc06ff20 con 0x7fd8fc151f80 2023-11-28T18:42:09.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.597+0000 7fd8fa7fc640 1 --2- 172.21.15.7:0/1071669960 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc151f80 0x7fd8fc15b400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009870 tx=0x7fd8ec01c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc9f9af5f2721fe7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:09.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.597+0000 7fd8f9ffb640 1 -- 172.21.15.7:0/1071669960 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0ac630 con 0x7fd8fc151f80 2023-11-28T18:42:09.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.597+0000 7fd8f9ffb640 1 -- 172.21.15.7:0/1071669960 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0ac790 con 0x7fd8fc151f80 2023-11-28T18:42:09.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.597+0000 7fd8f9ffb640 1 -- 172.21.15.7:0/1071669960 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0acb20 con 0x7fd8fc151f80 2023-11-28T18:42:09.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.601+0000 7fd8fbfff640 1 -- 172.21.15.7:0/1071669960 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc151f80 msgr2=0x7fd8fc15b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:09.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.601+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/1071669960 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc151f80 0x7fd8fc15b400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009870 tx=0x7fd8ec01c500 comp rx=0 tx=0).stop 2023-11-28T18:42:09.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.601+0000 7fd8fbfff640 1 -- 172.21.15.7:0/1071669960 shutdown_connections 2023-11-28T18:42:09.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.601+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/1071669960 >> v2:172.21.15.7:3300/0 conn(0x7fd8fc0c5060 0x7fd8fc15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.601+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/1071669960 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc151f80 0x7fd8fc15b400 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.601+0000 7fd8fbfff640 1 -- 172.21.15.7:0/1071669960 >> 172.21.15.7:0/1071669960 conn(0x7fd8fc0664b0 msgr2=0x7fd8fc0668c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:09.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.601+0000 7fd8fbfff640 1 -- 172.21.15.7:0/1071669960 shutdown_connections 2023-11-28T18:42:09.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.601+0000 7fd8fbfff640 1 -- 172.21.15.7:0/1071669960 wait complete. 2023-11-28T18:42:09.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fbfff640 1 Processor -- start 2023-11-28T18:42:09.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fbfff640 1 -- start start 2023-11-28T18:42:09.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fbfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc0c4c80 0x7fd8fc158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:09.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8fc0c5060 0x7fd8fc14a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:09.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8fc151f80 0x7fd8fc145cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:09.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fbfff640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc1f9da0 con 0x7fd8fc151f80 2023-11-28T18:42:09.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fbfff640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc159130 con 0x7fd8fc0c5060 2023-11-28T18:42:09.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fbfff640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc200360 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8faffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc0c4c80 0x7fd8fc158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:09.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8faffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc0c4c80 0x7fd8fc158bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:54690/0 (socket says 172.21.15.7:54690) 2023-11-28T18:42:09.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8faffd640 1 -- 172.21.15.7:0/2364121431 learned_addr learned my addr 172.21.15.7:0/2364121431 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:09.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fa7fc640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8fc0c5060 0x7fd8fc14a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:09.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8fb7fe640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8fc151f80 0x7fd8fc145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:09.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8faffd640 1 -- 172.21.15.7:0/2364121431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8fc0c5060 msgr2=0x7fd8fc14a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:09.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8faffd640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8fc0c5060 0x7fd8fc14a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8faffd640 1 -- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8fc151f80 msgr2=0x7fd8fc145cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:09.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8faffd640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8fc151f80 0x7fd8fc145cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.605+0000 7fd8faffd640 1 -- 172.21.15.7:0/2364121431 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0094d0 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.609+0000 7fd8faffd640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc0c4c80 0x7fd8fc158bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000b4a0 tx=0x7fd8f000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:09.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.609+0000 7fd8e37fe640 1 -- 172.21.15.7:0/2364121431 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0010610 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.609+0000 7fd8e37fe640 1 -- 172.21.15.7:0/2364121431 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0010790 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.609+0000 7fd8e37fe640 1 -- 172.21.15.7:0/2364121431 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0010b40 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.609+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8fc2005d0 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.609+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc2032b0 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.613+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc202e30 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.613+0000 7fd8e37fe640 1 -- 172.21.15.7:0/2364121431 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd8f0010ca0 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.617+0000 7fd8e37fe640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8e80668d0 0x7fd8e8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:09.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.617+0000 7fd8e37fe640 1 -- 172.21.15.7:0/2364121431 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fd8f00844a0 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.617+0000 7fd8fa7fc640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8e80668d0 0x7fd8e8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:09.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.617+0000 7fd8fa7fc640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8e80668d0 0x7fd8e8068d90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0ae000 tx=0x7fd8ec0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:09.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.621+0000 7fd8e37fe640 1 -- 172.21.15.7:0/2364121431 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd8fc202e30 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8fc202e30 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8e37fe640 1 -- 172.21.15.7:0/2364121431 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fd8fc202e30 con 0x7fd8fc0c4c80 2023-11-28T18:42:09.974 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:09.975 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:07.410757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:09.975 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 84 2023-11-28T18:42:09.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8e80668d0 msgr2=0x7fd8e8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:09.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8e80668d0 0x7fd8e8068d90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0ae000 tx=0x7fd8ec0a7000 comp rx=0 tx=0).stop 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc0c4c80 msgr2=0x7fd8fc158bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc0c4c80 0x7fd8fc158bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000b4a0 tx=0x7fd8f000b960 comp rx=0 tx=0).stop 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 shutdown_connections 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd8e80668d0 0x7fd8e8068d90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8fc151f80 0x7fd8fc145cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8fc0c5060 0x7fd8fc14a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 --2- 172.21.15.7:0/2364121431 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8fc0c4c80 0x7fd8fc158bf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 >> 172.21.15.7:0/2364121431 conn(0x7fd8fc0664b0 msgr2=0x7fd8fc160cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.973+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 shutdown_connections 2023-11-28T18:42:09.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:09.977+0000 7fd8fbfff640 1 -- 172.21.15.7:0/2364121431 wait complete. 2023-11-28T18:42:09.998 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-11-28T18:42:09.998 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:42:09.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:10.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbbd69a640 1 Processor -- start 2023-11-28T18:42:10.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbbd69a640 1 -- start start 2023-11-28T18:42:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbbd69a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbbd69a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbb806e560 0x7ffbb8157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbbd69a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb8071e60 con 0x7ffbb80701f0 2023-11-28T18:42:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbbd69a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb8070930 con 0x7ffbb806e560 2023-11-28T18:42:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbbd69a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb8070660 con 0x7ffbb8151cd0 2023-11-28T18:42:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbaffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbb806e560 0x7ffbb8157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbaffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbb806e560 0x7ffbb8157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44082/0 (socket says 172.21.15.7:44082) 2023-11-28T18:42:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbaffff640 1 -- 172.21.15.7:0/1911603747 learned_addr learned my addr 172.21.15.7:0/1911603747 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbb6ffd640 1 --2- 172.21.15.7:0/1911603747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbaffff640 1 -- 172.21.15.7:0/1911603747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 msgr2=0x7ffbb815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbaffff640 1 --2- 172.21.15.7:0/1911603747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbaffff640 1 -- 172.21.15.7:0/1911603747 >> v1:172.21.15.193:6789/0 conn(0x7ffbb80701f0 legacy=0x7ffbb80c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.165+0000 7ffbaffff640 1 -- 172.21.15.7:0/1911603747 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb8070bf0 con 0x7ffbb806e560 2023-11-28T18:42:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbaffff640 1 --2- 172.21.15.7:0/1911603747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb8157b90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009870 tx=0x7ffba001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98570ed295f9cc4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbb67fc640 1 -- 172.21.15.7:0/1911603747 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba00ac600 con 0x7ffbb806e560 2023-11-28T18:42:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbb67fc640 1 -- 172.21.15.7:0/1911603747 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba00a8050 con 0x7ffbb806e560 2023-11-28T18:42:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbb67fc640 1 -- 172.21.15.7:0/1911603747 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba00ac980 con 0x7ffbb806e560 2023-11-28T18:42:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbbd69a640 1 -- 172.21.15.7:0/1911603747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 msgr2=0x7ffbb8157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/1911603747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb8157b90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009870 tx=0x7ffba001c4b0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbbd69a640 1 -- 172.21.15.7:0/1911603747 shutdown_connections 2023-11-28T18:42:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/1911603747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb8157b90 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/1911603747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbbd69a640 1 -- 172.21.15.7:0/1911603747 >> 172.21.15.7:0/1911603747 conn(0x7ffbb80662e0 msgr2=0x7ffbb80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbbd69a640 1 -- 172.21.15.7:0/1911603747 shutdown_connections 2023-11-28T18:42:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.169+0000 7ffbbd69a640 1 -- 172.21.15.7:0/1911603747 wait complete. 2023-11-28T18:42:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 Processor -- start 2023-11-28T18:42:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 -- start start 2023-11-28T18:42:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb80ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbb80701f0 0x7ffbb81f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb81f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb81f6cb0 con 0x7ffbb806e560 2023-11-28T18:42:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb81f70d0 con 0x7ffbb80701f0 2023-11-28T18:42:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb8155a00 con 0x7ffbb8151cd0 2023-11-28T18:42:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb80ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb80ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44092/0 (socket says 172.21.15.7:44092) 2023-11-28T18:42:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 -- 172.21.15.7:0/3923641499 learned_addr learned my addr 172.21.15.7:0/3923641499 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbaffff640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbb80701f0 0x7ffbb81f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb77fe640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb81f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 -- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 msgr2=0x7ffbb81f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb81f3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 -- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbb80701f0 msgr2=0x7ffbb81f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbb80701f0 0x7ffbb81f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 -- 172.21.15.7:0/3923641499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba00094d0 con 0x7ffbb806e560 2023-11-28T18:42:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb77fe640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb81f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:42:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbb6ffd640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb80ca560 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ffba800e8a0 tx=0x7ffba800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbaf7fe640 1 -- 172.21.15.7:0/3923641499 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8009510 con 0x7ffbb806e560 2023-11-28T18:42:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbaf7fe640 1 -- 172.21.15.7:0/3923641499 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba800c940 con 0x7ffbb806e560 2023-11-28T18:42:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbaf7fe640 1 -- 172.21.15.7:0/3923641499 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba800ccb0 con 0x7ffbb806e560 2023-11-28T18:42:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb8157070 con 0x7ffbb806e560 2023-11-28T18:42:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb8155f10 con 0x7ffbb806e560 2023-11-28T18:42:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.173+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb8151820 con 0x7ffbb806e560 2023-11-28T18:42:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.177+0000 7ffbaf7fe640 1 -- 172.21.15.7:0/3923641499 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ffba8013470 con 0x7ffbb806e560 2023-11-28T18:42:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.177+0000 7ffbaf7fe640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb98066900 0x7ffb98068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.177+0000 7ffbaf7fe640 1 -- 172.21.15.7:0/3923641499 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7ffba8084820 con 0x7ffbb806e560 2023-11-28T18:42:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.177+0000 7ffbaffff640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb98066900 0x7ffb98068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.177+0000 7ffbaffff640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb98066900 0x7ffb98068dc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ffba00ae000 tx=0x7ffba00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:10.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.181+0000 7ffbaf7fe640 1 -- 172.21.15.7:0/3923641499 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffba8051730 con 0x7ffbb806e560 2023-11-28T18:42:10.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.369+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbb81f6f20 con 0x7ffbb806e560 2023-11-28T18:42:10.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbaf7fe640 1 -- 172.21.15.7:0/3923641499 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5769 (secure 0 0 0) 0x7ffba8054d50 con 0x7ffbb806e560 2023-11-28T18:42:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:10.373 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:07.410757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 84 2023-11-28T18:42:10.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb98066900 msgr2=0x7ffb98068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb98066900 0x7ffb98068dc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ffba00ae000 tx=0x7ffba00a7000 comp rx=0 tx=0).stop 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 msgr2=0x7ffbb80ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb80ca560 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ffba800e8a0 tx=0x7ffba800eca0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 shutdown_connections 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ffb98066900 0x7ffb98068dc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ffbb8151cd0 0x7ffbb81f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbb80701f0 0x7ffbb81f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 --2- 172.21.15.7:0/3923641499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbb806e560 0x7ffbb80ca560 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 >> 172.21.15.7:0/3923641499 conn(0x7ffbb80662e0 msgr2=0x7ffbb814b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 shutdown_connections 2023-11-28T18:42:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:10.373+0000 7ffbbd69a640 1 -- 172.21.15.7:0/3923641499 wait complete. 2023-11-28T18:42:10.395 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 84, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5450": { "addr": "172.21.15.193:6835/2303207996", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 2303207996, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 2303207996, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5450, "incarnation": 72, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 28 }, "gid_5528": { "addr": "172.21.15.7:6835/695953635", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 695953635, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 695953635, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5528, "incarnation": 84, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:42:07.410757+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5528, "mds_2": 5450, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:42:12.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:12.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e6fd0640 1 Processor -- start 2023-11-28T18:42:12.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e6fd0640 1 -- start start 2023-11-28T18:42:12.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e6fd0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e00c7c30 0x7f12e00c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:12.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e6fd0640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f12e0150f00 0x7f12e00c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:12.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e6fd0640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e00c5d10 con 0x7f12e00c8010 2023-11-28T18:42:12.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e6fd0640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e0070670 con 0x7f12e0150f00 2023-11-28T18:42:12.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e6fd0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e01569b0 con 0x7f12e00c7c30 2023-11-28T18:42:12.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e5fce640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e00c7c30 0x7f12e00c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:12.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e5fce640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e00c7c30 0x7f12e00c3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:54716/0 (socket says 172.21.15.7:54716) 2023-11-28T18:42:12.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e5fce640 1 -- 172.21.15.7:0/3788797740 learned_addr learned my addr 172.21.15.7:0/3788797740 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:12.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e67cf640 1 --2- 172.21.15.7:0/3788797740 >> v2:172.21.15.193:3300/0 conn(0x7f12e0150f00 0x7f12e00c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:12.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e5fce640 1 -- 172.21.15.7:0/3788797740 >> v2:172.21.15.193:3300/0 conn(0x7f12e0150f00 msgr2=0x7f12e00c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:12.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e5fce640 1 --2- 172.21.15.7:0/3788797740 >> v2:172.21.15.193:3300/0 conn(0x7f12e0150f00 0x7f12e00c61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e5fce640 1 -- 172.21.15.7:0/3788797740 >> v1:172.21.15.193:6789/0 conn(0x7f12e00c8010 legacy=0x7f12e01507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:12.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.553+0000 7f12e5fce640 1 -- 172.21.15.7:0/3788797740 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e00c4190 con 0x7f12e00c7c30 2023-11-28T18:42:12.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e5fce640 1 --2- 172.21.15.7:0/3788797740 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e00c7c30 0x7f12e00c3c50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009870 tx=0x7f12dc01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2176cadb762559ed server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:12.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e4fcc640 1 -- 172.21.15.7:0/3788797740 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0a8a30 con 0x7f12e00c7c30 2023-11-28T18:42:12.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e4fcc640 1 -- 172.21.15.7:0/3788797740 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc0a8b90 con 0x7f12e00c7c30 2023-11-28T18:42:12.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e4fcc640 1 -- 172.21.15.7:0/3788797740 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0b0440 con 0x7f12e00c7c30 2023-11-28T18:42:12.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e6fd0640 1 -- 172.21.15.7:0/3788797740 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e00c7c30 msgr2=0x7f12e00c3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:12.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/3788797740 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e00c7c30 0x7f12e00c3c50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009870 tx=0x7f12dc01ed30 comp rx=0 tx=0).stop 2023-11-28T18:42:12.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e6fd0640 1 -- 172.21.15.7:0/3788797740 shutdown_connections 2023-11-28T18:42:12.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/3788797740 >> v2:172.21.15.193:3300/0 conn(0x7f12e0150f00 0x7f12e00c61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/3788797740 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e00c7c30 0x7f12e00c3c50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e6fd0640 1 -- 172.21.15.7:0/3788797740 >> 172.21.15.7:0/3788797740 conn(0x7f12e006c370 msgr2=0x7f12e00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:12.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e6fd0640 1 -- 172.21.15.7:0/3788797740 shutdown_connections 2023-11-28T18:42:12.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.557+0000 7f12e6fd0640 1 -- 172.21.15.7:0/3788797740 wait complete. 2023-11-28T18:42:12.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 Processor -- start 2023-11-28T18:42:12.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 -- start start 2023-11-28T18:42:12.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12e00c7c30 0x7f12e013b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:12.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12e00c8010 0x7f12e0144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:12.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e0150f00 0x7f12e013aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:12.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0158ea0 con 0x7f12e00c8010 2023-11-28T18:42:12.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e01599a0 con 0x7f12e00c7c30 2023-11-28T18:42:12.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0159d30 con 0x7f12e0150f00 2023-11-28T18:42:12.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12e00c8010 0x7f12e0144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:12.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12e00c8010 0x7f12e0144140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44106/0 (socket says 172.21.15.7:44106) 2023-11-28T18:42:12.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 -- 172.21.15.7:0/90392843 learned_addr learned my addr 172.21.15.7:0/90392843 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:12.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e5fce640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12e00c7c30 0x7f12e013b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:12.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e67cf640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e0150f00 0x7f12e013aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:12.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 -- 172.21.15.7:0/90392843 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e0150f00 msgr2=0x7f12e013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:12.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e0150f00 0x7f12e013aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 -- 172.21.15.7:0/90392843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12e00c7c30 msgr2=0x7f12e013b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:12.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12e00c7c30 0x7f12e013b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 -- 172.21.15.7:0/90392843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc0094d0 con 0x7f12e00c8010 2023-11-28T18:42:12.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e57cd640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12e00c8010 0x7f12e0144140 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f12d000c430 tx=0x7f12d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:12.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12d67fc640 1 -- 172.21.15.7:0/90392843 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d00052d0 con 0x7f12e00c8010 2023-11-28T18:42:12.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12d67fc640 1 -- 172.21.15.7:0/90392843 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d0005430 con 0x7f12e00c8010 2023-11-28T18:42:12.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12d67fc640 1 -- 172.21.15.7:0/90392843 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d0007260 con 0x7f12e00c8010 2023-11-28T18:42:12.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e0142ec0 con 0x7f12e00c8010 2023-11-28T18:42:12.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.561+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e01f0fe0 con 0x7f12e00c8010 2023-11-28T18:42:12.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.565+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e004e890 con 0x7f12e00c8010 2023-11-28T18:42:12.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.565+0000 7f12d67fc640 1 -- 172.21.15.7:0/90392843 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f12d00073c0 con 0x7f12e00c8010 2023-11-28T18:42:12.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.565+0000 7f12d67fc640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12c40668d0 0x7f12c4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:12.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.565+0000 7f12d67fc640 1 -- 172.21.15.7:0/90392843 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f12d0083c50 con 0x7f12e00c8010 2023-11-28T18:42:12.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.565+0000 7f12e5fce640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12c40668d0 0x7f12c4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:12.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.565+0000 7f12e5fce640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12c40668d0 0x7f12c4068d90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0b2000 tx=0x7f12dc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:12.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.569+0000 7f12d67fc640 1 -- 172.21.15.7:0/90392843 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f12d0050a40 con 0x7f12e00c8010 2023-11-28T18:42:12.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e013fc90 con 0x7f12e00c8010 2023-11-28T18:42:12.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12d67fc640 1 -- 172.21.15.7:0/90392843 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f12d0054060 con 0x7f12e00c8010 2023-11-28T18:42:12.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:12.771 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:07.410757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:12.772 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 84 2023-11-28T18:42:12.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12c40668d0 msgr2=0x7f12c4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:12.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12c40668d0 0x7f12c4068d90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0b2000 tx=0x7f12dc0aa070 comp rx=0 tx=0).stop 2023-11-28T18:42:12.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12e00c8010 msgr2=0x7f12e0144140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:12.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12e00c8010 0x7f12e0144140 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f12d000c430 tx=0x7f12d000c8f0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 shutdown_connections 2023-11-28T18:42:12.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f12c40668d0 0x7f12c4068d90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12e0150f00 0x7f12e013aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12e00c8010 0x7f12e0144140 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 --2- 172.21.15.7:0/90392843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12e00c7c30 0x7f12e013b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:12.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 >> 172.21.15.7:0/90392843 conn(0x7f12e006c370 msgr2=0x7f12e01545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:12.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 shutdown_connections 2023-11-28T18:42:12.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:12.769+0000 7f12e6fd0640 1 -- 172.21.15.7:0/90392843 wait complete. 2023-11-28T18:42:12.793 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:42:14.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:14.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.945+0000 7ff3c8bb8640 1 Processor -- start 2023-11-28T18:42:14.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c8bb8640 1 -- start start 2023-11-28T18:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c8bb8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3c4150b30 0x7ff3c40c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c8bb8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c40c7980 0x7ff3c40c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c8bb8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c40c6140 con 0x7ff3c41575c0 2023-11-28T18:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c8bb8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c40c72b0 con 0x7ff3c4150b30 2023-11-28T18:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c8bb8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4070c00 con 0x7ff3c40c7980 2023-11-28T18:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3c4150b30 0x7ff3c40c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3bbfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff3c41575c0 0x7ff3c40c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40422/0 (socket says 172.21.15.7:40422) 2023-11-28T18:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3bbfff640 1 -- 172.21.15.7:0/1412543578 learned_addr learned my addr 172.21.15.7:0/1412543578 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c259c640 1 -- 172.21.15.7:0/1412543578 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c40c7980 msgr2=0x7ff3c40c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c2d9d640 1 --2- 172.21.15.7:0/1412543578 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c40c7980 0x7ff3c40c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c259c640 1 --2- 172.21.15.7:0/1412543578 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c40c7980 0x7ff3c40c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c259c640 1 -- 172.21.15.7:0/1412543578 >> v1:172.21.15.7:6789/0 conn(0x7ff3c41575c0 legacy=0x7ff3c40c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c259c640 1 -- 172.21.15.7:0/1412543578 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c40c8350 con 0x7ff3c4150b30 2023-11-28T18:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c2d9d640 1 --2- 172.21.15.7:0/1412543578 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c40c7980 0x7ff3c40c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.949+0000 7ff3c1d9b640 1 -- 172.21.15.7:0/1412543578 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969274608 0 0) 0x7ff3c40c6140 con 0x7ff3c41575c0 2023-11-28T18:42:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c259c640 1 --2- 172.21.15.7:0/1412543578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c4150b30 0x7ff3c40c24f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40097c0 tx=0x7ff3b401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65096b1506be08b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c1d9b640 1 -- 172.21.15.7:0/1412543578 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b40a8970 con 0x7ff3c4150b30 2023-11-28T18:42:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c1d9b640 1 -- 172.21.15.7:0/1412543578 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b40a8ad0 con 0x7ff3c4150b30 2023-11-28T18:42:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c1d9b640 1 -- 172.21.15.7:0/1412543578 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b40a8e60 con 0x7ff3c4150b30 2023-11-28T18:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/1412543578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c4150b30 msgr2=0x7ff3c40c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/1412543578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c4150b30 0x7ff3c40c24f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40097c0 tx=0x7ff3b401ec70 comp rx=0 tx=0).stop 2023-11-28T18:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/1412543578 shutdown_connections 2023-11-28T18:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/1412543578 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c40c7980 0x7ff3c40c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/1412543578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c4150b30 0x7ff3c40c24f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/1412543578 >> 172.21.15.7:0/1412543578 conn(0x7ff3c40662e0 msgr2=0x7ff3c40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/1412543578 shutdown_connections 2023-11-28T18:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/1412543578 wait complete. 2023-11-28T18:42:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 Processor -- start 2023-11-28T18:42:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.953+0000 7ff3c8bb8640 1 -- start start 2023-11-28T18:42:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3c40c7980 0x7ff3c4156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c4150b30 0x7ff3c41549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c41575c0 0x7ff3c4154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c41f0760 con 0x7ff3c41575c0 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c41ef8f0 con 0x7ff3c40c7980 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4154540 con 0x7ff3c4150b30 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c41575c0 0x7ff3c4154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c41575c0 0x7ff3c4154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47336/0 (socket says 172.21.15.7:47336) 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 -- 172.21.15.7:0/496024392 learned_addr learned my addr 172.21.15.7:0/496024392 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3bbfff640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c4150b30 0x7ff3c41549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c259c640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3c40c7980 0x7ff3c4156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 -- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c4150b30 msgr2=0x7ff3c41549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c4150b30 0x7ff3c41549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 -- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3c40c7980 msgr2=0x7ff3c4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3c40c7980 0x7ff3c4156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 -- 172.21.15.7:0/496024392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4009420 con 0x7ff3c41575c0 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3bbfff640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c4150b30 0x7ff3c41549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c259c640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3c40c7980 0x7ff3c4156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c2d9d640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c41575c0 0x7ff3c4154f30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00cf20 tx=0x7ff3bc00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3baffd640 1 -- 172.21.15.7:0/496024392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc00d840 con 0x7ff3c41575c0 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3baffd640 1 -- 172.21.15.7:0/496024392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc00d9a0 con 0x7ff3c41575c0 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3baffd640 1 -- 172.21.15.7:0/496024392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc00dd50 con 0x7ff3c41575c0 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c41efae0 con 0x7ff3c41575c0 2023-11-28T18:42:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c41f00a0 con 0x7ff3c41575c0 2023-11-28T18:42:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.957+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c4157260 con 0x7ff3c41575c0 2023-11-28T18:42:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.961+0000 7ff3baffd640 1 -- 172.21.15.7:0/496024392 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff3bc01e3b0 con 0x7ff3c41575c0 2023-11-28T18:42:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.961+0000 7ff3baffd640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3a4066920 0x7ff3a4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.961+0000 7ff3baffd640 1 -- 172.21.15.7:0/496024392 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7ff3bc085320 con 0x7ff3c41575c0 2023-11-28T18:42:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.961+0000 7ff3c259c640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3a4066920 0x7ff3a4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.961+0000 7ff3c259c640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3a4066920 0x7ff3a4068de0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40b2000 tx=0x7ff3b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:14.965+0000 7ff3baffd640 1 -- 172.21.15.7:0/496024392 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff3bc052170 con 0x7ff3c41575c0 2023-11-28T18:42:15.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3c404e890 con 0x7ff3c41575c0 2023-11-28T18:42:15.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3baffd640 1 -- 172.21.15.7:0/496024392 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5771 (secure 0 0 0) 0x7ff3bc055790 con 0x7ff3c41575c0 2023-11-28T18:42:15.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:15.158 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:12.736244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:resolve","state_seq":24,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 85 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3a4066920 msgr2=0x7ff3a4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3a4066920 0x7ff3a4068de0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40b2000 tx=0x7ff3b4008040 comp rx=0 tx=0).stop 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c41575c0 msgr2=0x7ff3c4154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c41575c0 0x7ff3c4154f30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00cf20 tx=0x7ff3bc00a7d0 comp rx=0 tx=0).stop 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 shutdown_connections 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff3a4066920 0x7ff3a4068de0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3c41575c0 0x7ff3c4154f30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff3c4150b30 0x7ff3c41549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 --2- 172.21.15.7:0/496024392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3c40c7980 0x7ff3c4156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 >> 172.21.15.7:0/496024392 conn(0x7ff3c40662e0 msgr2=0x7ff3c40666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 shutdown_connections 2023-11-28T18:42:15.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:15.157+0000 7ff3c8bb8640 1 -- 172.21.15.7:0/496024392 wait complete. 2023-11-28T18:42:15.179 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:42:17.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:17.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.333+0000 7f3b71980640 1 Processor -- start 2023-11-28T18:42:17.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.333+0000 7f3b71980640 1 -- start start 2023-11-28T18:42:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b71980640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3b6c154000 0x7f3b6c0c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b71980640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c0c3350 0x7f3b6c159e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b71980640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c15e0f0 con 0x7f3b6c0c6ba0 2023-11-28T18:42:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b71980640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c15d520 con 0x7f3b6c154000 2023-11-28T18:42:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b71980640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c0c7c70 con 0x7f3b6c0c3350 2023-11-28T18:42:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b63fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3b6c0c6ba0 0x7f3b6c0c2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40428/0 (socket says 172.21.15.7:40428) 2023-11-28T18:42:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b63fff640 1 -- 172.21.15.7:0/3828199375 learned_addr learned my addr 172.21.15.7:0/3828199375 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:42:17.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6affd640 1 --2- 172.21.15.7:0/3828199375 >> v2:172.21.15.193:3300/0 conn(0x7f3b6c154000 0x7f3b6c0c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6b7fe640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c0c3350 0x7f3b6c159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765550196 0 0) 0x7f3b6c15e0f0 con 0x7f3b6c0c6ba0 2023-11-28T18:42:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b58003530 con 0x7f3b6c0c6ba0 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3829982689 0 0) 0x7f3b58003530 con 0x7f3b6c0c6ba0 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c0c3350 msgr2=0x7f3b6c159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c0c3350 0x7f3b6c159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 >> v2:172.21.15.193:3300/0 conn(0x7f3b6c154000 msgr2=0x7f3b6c0c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 --2- 172.21.15.7:0/3828199375 >> v2:172.21.15.193:3300/0 conn(0x7f3b6c154000 0x7f3b6c0c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c15d7c0 con 0x7f3b6c0c6ba0 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f3b54001680 con 0x7f3b6c0c6ba0 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6affd640 1 --2- 172.21.15.7:0/3828199375 >> v2:172.21.15.193:3300/0 conn(0x7f3b6c154000 0x7f3b6c0c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 >> v1:172.21.15.7:6789/0 conn(0x7f3b6c0c6ba0 legacy=0x7f3b6c0c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c0c3350 0x7f3b58003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b58004d10 0x7f3b58005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c15d520 con 0x7f3b58003f00 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c0c7c70 con 0x7f3b58004d10 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5800b9c0 con 0x7f3b6c0c3350 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b63fff640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6affd640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c0c3350 0x7f3b58003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6b7fe640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b58004d10 0x7f3b58005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b63fff640 1 -- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c0c3350 msgr2=0x7f3b58003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b63fff640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c0c3350 0x7f3b58003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b63fff640 1 -- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b58004d10 msgr2=0x7f3b58005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b63fff640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b58004d10 0x7f3b58005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b63fff640 1 -- 172.21.15.7:0/3828199375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b58003530 con 0x7f3b58003f00 2023-11-28T18:42:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.337+0000 7f3b6b7fe640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b58004d10 0x7f3b58005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:42:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b63fff640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b580047d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400c210 tx=0x7f3b5401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a9060c44f4145ac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b540a5960 con 0x7f3b58003f00 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b540a5ac0 con 0x7f3b58003f00 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b6a7fc640 1 -- 172.21.15.7:0/3828199375 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b540058e0 con 0x7f3b58003f00 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b71980640 1 -- 172.21.15.7:0/3828199375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 msgr2=0x7f3b580047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b71980640 1 --2- 172.21.15.7:0/3828199375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b580047d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400c210 tx=0x7f3b5401cc60 comp rx=0 tx=0).stop 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b6b7fe640 1 -- 172.21.15.7:0/3828199375 reap_dead start 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b71980640 1 -- 172.21.15.7:0/3828199375 shutdown_connections 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b71980640 1 -- 172.21.15.7:0/3828199375 >> 172.21.15.7:0/3828199375 conn(0x7f3b6c0662e0 msgr2=0x7f3b6c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b71980640 1 -- 172.21.15.7:0/3828199375 shutdown_connections 2023-11-28T18:42:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.341+0000 7f3b71980640 1 -- 172.21.15.7:0/3828199375 wait complete. 2023-11-28T18:42:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 Processor -- start 2023-11-28T18:42:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 -- start start 2023-11-28T18:42:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b6c1f0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b6c073150 0x7f3b6c0ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c1505c0 0x7f3b6c14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c14fe10 con 0x7f3b58003f00 2023-11-28T18:42:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c14f560 con 0x7f3b6c073150 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c14f840 con 0x7f3b6c1505c0 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b6c1f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b6c1f0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47362/0 (socket says 172.21.15.7:47362) 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 -- 172.21.15.7:0/3950553864 learned_addr learned my addr 172.21.15.7:0/3950553864 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6b7fe640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c1505c0 0x7f3b6c14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b63fff640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b6c073150 0x7f3b6c0ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 -- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c1505c0 msgr2=0x7f3b6c14e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c1505c0 0x7f3b6c14e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 -- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b6c073150 msgr2=0x7f3b6c0ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b6c073150 0x7f3b6c0ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 -- 172.21.15.7:0/3950553864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c0c5680 con 0x7f3b58003f00 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6b7fe640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c1505c0 0x7f3b6c14e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b63fff640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b6c073150 0x7f3b6c0ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b6affd640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b6c1f0000 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c011cd0 tx=0x7f3b5c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b68ff9640 1 -- 172.21.15.7:0/3950553864 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c011e80 con 0x7f3b58003f00 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b68ff9640 1 -- 172.21.15.7:0/3950553864 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c004020 con 0x7f3b58003f00 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b68ff9640 1 -- 172.21.15.7:0/3950553864 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0043b0 con 0x7f3b58003f00 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c1500f0 con 0x7f3b58003f00 2023-11-28T18:42:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.345+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c14eb60 con 0x7f3b58003f00 2023-11-28T18:42:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.349+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c04e890 con 0x7f3b58003f00 2023-11-28T18:42:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.349+0000 7f3b68ff9640 1 -- 172.21.15.7:0/3950553864 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3b5c003620 con 0x7f3b58003f00 2023-11-28T18:42:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.349+0000 7f3b68ff9640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.349+0000 7f3b68ff9640 1 -- 172.21.15.7:0/3950553864 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f3b5c0850f0 con 0x7f3b58003f00 2023-11-28T18:42:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.349+0000 7f3b63fff640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:17.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.349+0000 7f3b63fff640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3b540aa000 tx=0x7f3b54001a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.353+0000 7f3b68ff9640 1 -- 172.21.15.7:0/3950553864 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3b5c051fc0 con 0x7f3b58003f00 2023-11-28T18:42:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b6c14fa00 con 0x7f3b58003f00 2023-11-28T18:42:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b68ff9640 1 -- 172.21.15.7:0/3950553864 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f3b5c022090 con 0x7f3b58003f00 2023-11-28T18:42:17.547 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:17.547 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:12.736244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:resolve","state_seq":24,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 85 2023-11-28T18:42:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 msgr2=0x7f3b4c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3b540aa000 tx=0x7f3b54001a10 comp rx=0 tx=0).stop 2023-11-28T18:42:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 msgr2=0x7f3b6c1f0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b6c1f0000 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c011cd0 tx=0x7f3b5c00f610 comp rx=0 tx=0).stop 2023-11-28T18:42:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 shutdown_connections 2023-11-28T18:42:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3b4c066920 0x7f3b4c068de0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3b6c1505c0 0x7f3b6c14e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b6c073150 0x7f3b6c0ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 --2- 172.21.15.7:0/3950553864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b58003f00 0x7f3b6c1f0000 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 >> 172.21.15.7:0/3950553864 conn(0x7f3b6c0662e0 msgr2=0x7f3b6c152d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 shutdown_connections 2023-11-28T18:42:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:17.545+0000 7f3b71980640 1 -- 172.21.15.7:0/3950553864 wait complete. 2023-11-28T18:42:17.572 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:42:19.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.721+0000 7fae2d632640 1 Processor -- start 2023-11-28T18:42:19.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.721+0000 7fae2d632640 1 -- start start 2023-11-28T18:42:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae2d632640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fae28159e90 0x7fae2815a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae2d632640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2815e0f0 con 0x7fae280c2c40 2023-11-28T18:42:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae2d632640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2815d520 con 0x7fae28154000 2023-11-28T18:42:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae2d632640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae280c7c70 con 0x7fae28159e90 2023-11-28T18:42:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae1ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fae280c2c40 0x7fae280c3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40440/0 (socket says 172.21.15.7:40440) 2023-11-28T18:42:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae1ffff640 1 -- 172.21.15.7:0/3244597867 learned_addr learned my addr 172.21.15.7:0/3244597867 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:42:19.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae277fe640 1 --2- 172.21.15.7:0/3244597867 >> v2:172.21.15.193:3300/0 conn(0x7fae28159e90 0x7fae2815a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:19.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae267fc640 1 -- 172.21.15.7:0/3244597867 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346479514 0 0) 0x7fae2815e0f0 con 0x7fae280c2c40 2023-11-28T18:42:19.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae267fc640 1 -- 172.21.15.7:0/3244597867 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae14003530 con 0x7fae280c2c40 2023-11-28T18:42:19.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae277fe640 1 -- 172.21.15.7:0/3244597867 >> v1:172.21.15.193:6789/0 conn(0x7fae28154000 legacy=0x7fae280c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:19.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae267fc640 1 -- 172.21.15.7:0/3244597867 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3615429673 0 0) 0x7fae14003530 con 0x7fae280c2c40 2023-11-28T18:42:19.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae277fe640 1 -- 172.21.15.7:0/3244597867 >> v1:172.21.15.7:6789/0 conn(0x7fae280c2c40 legacy=0x7fae280c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:19.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.725+0000 7fae277fe640 1 -- 172.21.15.7:0/3244597867 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2815d7c0 con 0x7fae28159e90 2023-11-28T18:42:19.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae277fe640 1 --2- 172.21.15.7:0/3244597867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 0x7fae2815a2c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fae20009d90 tx=0x7fae2001f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c822a7cc15e542bf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:19.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae267fc640 1 -- 172.21.15.7:0/3244597867 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae10001680 con 0x7fae28159e90 2023-11-28T18:42:19.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae267fc640 1 -- 172.21.15.7:0/3244597867 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae200a8bf0 con 0x7fae28159e90 2023-11-28T18:42:19.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae267fc640 1 -- 172.21.15.7:0/3244597867 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae200b0440 con 0x7fae28159e90 2023-11-28T18:42:19.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 -- 172.21.15.7:0/3244597867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 msgr2=0x7fae2815a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:19.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 --2- 172.21.15.7:0/3244597867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 0x7fae2815a2c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fae20009d90 tx=0x7fae2001f060 comp rx=0 tx=0).stop 2023-11-28T18:42:19.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 -- 172.21.15.7:0/3244597867 shutdown_connections 2023-11-28T18:42:19.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 --2- 172.21.15.7:0/3244597867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 0x7fae2815a2c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:19.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 -- 172.21.15.7:0/3244597867 >> 172.21.15.7:0/3244597867 conn(0x7fae280662e0 msgr2=0x7fae280666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:19.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 -- 172.21.15.7:0/3244597867 shutdown_connections 2023-11-28T18:42:19.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 -- 172.21.15.7:0/3244597867 wait complete. 2023-11-28T18:42:19.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 Processor -- start 2023-11-28T18:42:19.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.729+0000 7fae2d632640 1 -- start start 2023-11-28T18:42:19.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae280c2c40 0x7fae281f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:19.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae28154000 0x7fae2813fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:19.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 0x7fae28145380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:19.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae281527e0 con 0x7fae280c2c40 2023-11-28T18:42:19.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae280cad50 con 0x7fae28159e90 2023-11-28T18:42:19.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae281f3ad0 con 0x7fae28154000 2023-11-28T18:42:19.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae280c2c40 0x7fae281f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae280c2c40 0x7fae281f3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47368/0 (socket says 172.21.15.7:47368) 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 -- 172.21.15.7:0/759721175 learned_addr learned my addr 172.21.15.7:0/759721175 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae1ffff640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae28154000 0x7fae2813fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae277fe640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 0x7fae28145380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 -- 172.21.15.7:0/759721175 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae28154000 msgr2=0x7fae2813fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae28154000 0x7fae2813fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 -- 172.21.15.7:0/759721175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 msgr2=0x7fae28145380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 0x7fae28145380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 -- 172.21.15.7:0/759721175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae20009a70 con 0x7fae280c2c40 2023-11-28T18:42:19.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae26ffd640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae280c2c40 0x7fae281f3590 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fae1800c500 tx=0x7fae1800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae1f7fe640 1 -- 172.21.15.7:0/759721175 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18005220 con 0x7fae280c2c40 2023-11-28T18:42:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae1f7fe640 1 -- 172.21.15.7:0/759721175 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae18005380 con 0x7fae280c2c40 2023-11-28T18:42:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae1f7fe640 1 -- 172.21.15.7:0/759721175 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18007270 con 0x7fae280c2c40 2023-11-28T18:42:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae280caf60 con 0x7fae280c2c40 2023-11-28T18:42:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae281f2d90 con 0x7fae280c2c40 2023-11-28T18:42:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae281f2bd0 con 0x7fae280c2c40 2023-11-28T18:42:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.733+0000 7fae1f7fe640 1 -- 172.21.15.7:0/759721175 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fae180073d0 con 0x7fae280c2c40 2023-11-28T18:42:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.737+0000 7fae1f7fe640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae080668d0 0x7fae08068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.737+0000 7fae1f7fe640 1 -- 172.21.15.7:0/759721175 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fae18083dd0 con 0x7fae280c2c40 2023-11-28T18:42:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.737+0000 7fae1ffff640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae080668d0 0x7fae08068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.737+0000 7fae1ffff640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae080668d0 0x7fae08068d90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fae2814dca0 tx=0x7fae10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:19.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.741+0000 7fae1f7fe640 1 -- 172.21.15.7:0/759721175 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fae18050bb0 con 0x7fae280c2c40 2023-11-28T18:42:19.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.933+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae2804e890 con 0x7fae280c2c40 2023-11-28T18:42:19.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.933+0000 7fae1f7fe640 1 -- 172.21.15.7:0/759721175 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fae180541d0 con 0x7fae280c2c40 2023-11-28T18:42:19.935 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:19.935 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:18.526006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:19.936 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 86 2023-11-28T18:42:19.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae080668d0 msgr2=0x7fae08068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:19.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae080668d0 0x7fae08068d90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fae2814dca0 tx=0x7fae10009040 comp rx=0 tx=0).stop 2023-11-28T18:42:19.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae280c2c40 msgr2=0x7fae281f3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:19.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae280c2c40 0x7fae281f3590 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fae1800c500 tx=0x7fae1800c9c0 comp rx=0 tx=0).stop 2023-11-28T18:42:19.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 shutdown_connections 2023-11-28T18:42:19.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fae080668d0 0x7fae08068d90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:19.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae28159e90 0x7fae28145380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:19.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fae28154000 0x7fae2813fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:19.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 --2- 172.21.15.7:0/759721175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae280c2c40 0x7fae281f3590 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:19.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 >> 172.21.15.7:0/759721175 conn(0x7fae280662e0 msgr2=0x7fae280c73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:19.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 shutdown_connections 2023-11-28T18:42:19.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:19.937+0000 7fae2d632640 1 -- 172.21.15.7:0/759721175 wait complete. 2023-11-28T18:42:19.957 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:42:21.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:22.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff760bb640 1 Processor -- start 2023-11-28T18:42:22.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff760bb640 1 -- start start 2023-11-28T18:42:22.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff760bb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff70153ee0 0x7eff700cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:22.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff760bb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7eff700ca330 0x7eff700ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff760bb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7015d240 con 0x7eff70159c90 2023-11-28T18:42:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff760bb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7015d520 con 0x7eff70153ee0 2023-11-28T18:42:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff760bb640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff700c4bc0 con 0x7eff700ca330 2023-11-28T18:42:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff750b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff70153ee0 0x7eff700cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff750b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff70153ee0 0x7eff700cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47376/0 (socket says 172.21.15.7:47376) 2023-11-28T18:42:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff750b9640 1 -- 172.21.15.7:0/3113356628 learned_addr learned my addr 172.21.15.7:0/3113356628 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff758ba640 1 --2- 172.21.15.7:0/3113356628 >> v2:172.21.15.193:3300/0 conn(0x7eff700ca330 0x7eff700ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff750b9640 1 -- 172.21.15.7:0/3113356628 >> v2:172.21.15.193:3300/0 conn(0x7eff700ca330 msgr2=0x7eff700ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff750b9640 1 --2- 172.21.15.7:0/3113356628 >> v2:172.21.15.193:3300/0 conn(0x7eff700ca330 0x7eff700ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff750b9640 1 -- 172.21.15.7:0/3113356628 >> v1:172.21.15.193:6789/0 conn(0x7eff70159c90 legacy=0x7eff7015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.113+0000 7eff750b9640 1 -- 172.21.15.7:0/3113356628 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7015a240 con 0x7eff70153ee0 2023-11-28T18:42:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff750b9640 1 --2- 172.21.15.7:0/3113356628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70153ee0 0x7eff700cb600 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7eff7006b4e0 tx=0x7eff6c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e221180a5044d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff67fff640 1 -- 172.21.15.7:0/3113356628 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0a89f0 con 0x7eff70153ee0 2023-11-28T18:42:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff67fff640 1 -- 172.21.15.7:0/3113356628 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c0a8b50 con 0x7eff70153ee0 2023-11-28T18:42:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff67fff640 1 -- 172.21.15.7:0/3113356628 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0b0440 con 0x7eff70153ee0 2023-11-28T18:42:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff760bb640 1 -- 172.21.15.7:0/3113356628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70153ee0 msgr2=0x7eff700cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff760bb640 1 --2- 172.21.15.7:0/3113356628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70153ee0 0x7eff700cb600 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7eff7006b4e0 tx=0x7eff6c01ecf0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff760bb640 1 -- 172.21.15.7:0/3113356628 shutdown_connections 2023-11-28T18:42:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff760bb640 1 --2- 172.21.15.7:0/3113356628 >> v2:172.21.15.193:3300/0 conn(0x7eff700ca330 0x7eff700ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff760bb640 1 --2- 172.21.15.7:0/3113356628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70153ee0 0x7eff700cb600 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff760bb640 1 -- 172.21.15.7:0/3113356628 >> 172.21.15.7:0/3113356628 conn(0x7eff70071e20 msgr2=0x7eff70072230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff760bb640 1 -- 172.21.15.7:0/3113356628 shutdown_connections 2023-11-28T18:42:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.117+0000 7eff760bb640 1 -- 172.21.15.7:0/3113356628 wait complete. 2023-11-28T18:42:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 Processor -- start 2023-11-28T18:42:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 -- start start 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff700ca330 0x7eff70148480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff70153ee0 0x7eff701e4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70159c90 0x7eff7015b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7015cf40 con 0x7eff70159c90 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff700b55d0 con 0x7eff70153ee0 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7015cc60 con 0x7eff700ca330 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70159c90 0x7eff7015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70159c90 0x7eff7015b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47378/0 (socket says 172.21.15.7:47378) 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 -- 172.21.15.7:0/3594677986 learned_addr learned my addr 172.21.15.7:0/3594677986 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff750b9640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff700ca330 0x7eff70148480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff748b8640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff70153ee0 0x7eff701e4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 -- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff700ca330 msgr2=0x7eff70148480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff700ca330 0x7eff70148480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 -- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff70153ee0 msgr2=0x7eff701e4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff70153ee0 0x7eff701e4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 -- 172.21.15.7:0/3594677986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c0094d0 con 0x7eff70159c90 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff748b8640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff70153ee0 0x7eff701e4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff758ba640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70159c90 0x7eff7015b620 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7eff6800efc0 tx=0x7eff6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff65ffb640 1 -- 172.21.15.7:0/3594677986 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68009ba0 con 0x7eff70159c90 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff65ffb640 1 -- 172.21.15.7:0/3594677986 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6800f040 con 0x7eff70159c90 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff65ffb640 1 -- 172.21.15.7:0/3594677986 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff680146c0 con 0x7eff70159c90 2023-11-28T18:42:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff701479a0 con 0x7eff70159c90 2023-11-28T18:42:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff70147f60 con 0x7eff70159c90 2023-11-28T18:42:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.121+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff70148180 con 0x7eff70159c90 2023-11-28T18:42:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.125+0000 7eff65ffb640 1 -- 172.21.15.7:0/3594677986 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7eff68014820 con 0x7eff70159c90 2023-11-28T18:42:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.125+0000 7eff65ffb640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff540668d0 0x7eff54068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.125+0000 7eff65ffb640 1 -- 172.21.15.7:0/3594677986 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7eff680853a0 con 0x7eff70159c90 2023-11-28T18:42:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.125+0000 7eff750b9640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff540668d0 0x7eff54068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.125+0000 7eff750b9640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff540668d0 0x7eff54068d90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0b2000 tx=0x7eff6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:22.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.129+0000 7eff65ffb640 1 -- 172.21.15.7:0/3594677986 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7eff68052170 con 0x7eff70159c90 2023-11-28T18:42:22.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff7004e890 con 0x7eff70159c90 2023-11-28T18:42:22.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff65ffb640 1 -- 172.21.15.7:0/3594677986 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5773 (secure 0 0 0) 0x7eff68055790 con 0x7eff70159c90 2023-11-28T18:42:22.318 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:22.318 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:18.526006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:22.319 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 86 2023-11-28T18:42:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff540668d0 msgr2=0x7eff54068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff540668d0 0x7eff54068d90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0b2000 tx=0x7eff6c008040 comp rx=0 tx=0).stop 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70159c90 msgr2=0x7eff7015b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70159c90 0x7eff7015b620 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7eff6800efc0 tx=0x7eff6800c490 comp rx=0 tx=0).stop 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 shutdown_connections 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7eff540668d0 0x7eff54068d90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff70159c90 0x7eff7015b620 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff70153ee0 0x7eff701e4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 --2- 172.21.15.7:0/3594677986 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7eff700ca330 0x7eff70148480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 >> 172.21.15.7:0/3594677986 conn(0x7eff70071e20 msgr2=0x7eff70143b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 shutdown_connections 2023-11-28T18:42:22.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:22.317+0000 7eff760bb640 1 -- 172.21.15.7:0/3594677986 wait complete. 2023-11-28T18:42:22.341 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 86, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5450": { "addr": "172.21.15.193:6835/2303207996", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 2303207996, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 2303207996, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5450, "incarnation": 72, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 28 }, "gid_5528": { "addr": "172.21.15.7:6835/695953635", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 695953635, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 695953635, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5528, "incarnation": 84, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 26 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:42:18.526006+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5528, "mds_2": 5450, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:42:24.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.485+0000 7fc6b57d8640 1 Processor -- start 2023-11-28T18:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.485+0000 7fc6b57d8640 1 -- start start 2023-11-28T18:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.485+0000 7fc6b57d8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b0072950 0x7fc6b0074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.485+0000 7fc6b57d8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b00c8490 con 0x7fc6b006c920 2023-11-28T18:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.485+0000 7fc6b57d8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b00c81f0 con 0x7fc6b0073070 2023-11-28T18:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.485+0000 7fc6b57d8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b015ab00 con 0x7fc6b0072950 2023-11-28T18:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.485+0000 7fc6af7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc6b006c920 0x7fc6b006cda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58342/0 (socket says 172.21.15.7:58342) 2023-11-28T18:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.485+0000 7fc6af7fe640 1 -- 172.21.15.7:0/1449051546 learned_addr learned my addr 172.21.15.7:0/1449051546 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:42:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6aeffd640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b0072950 0x7fc6b0074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868537142 0 0) 0x7fc6b00c8490 con 0x7fc6b006c920 2023-11-28T18:42:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c003530 con 0x7fc6b006c920 2023-11-28T18:42:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252544957 0 0) 0x7fc6b00c81f0 con 0x7fc6b0073070 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b00c8490 con 0x7fc6b0073070 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814919859 0 0) 0x7fc69c003530 con 0x7fc6b006c920 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b0072950 msgr2=0x7fc6b0074030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b0072950 0x7fc6b0074030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 >> v1:172.21.15.193:6789/0 conn(0x7fc6b0073070 legacy=0x7fc6b00c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b015ade0 con 0x7fc6b006c920 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fc6a8001a00 con 0x7fc6b006c920 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 >> v1:172.21.15.7:6789/0 conn(0x7fc6b006c920 legacy=0x7fc6b006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 0x7fc69c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c0047d0 0x7fc69c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b0072950 0x7fc69c005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b015ab00 con 0x7fc69c0047d0 2023-11-28T18:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c00bb50 con 0x7fc69c003b40 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c00bd10 con 0x7fc6b0072950 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6a7fff640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c0047d0 0x7fc69c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6aeffd640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 0x7fc69c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6af7fe640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b0072950 0x7fc69c005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6aeffd640 1 -- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b0072950 msgr2=0x7fc69c005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6aeffd640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b0072950 0x7fc69c005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6aeffd640 1 -- 172.21.15.7:0/1449051546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c0047d0 msgr2=0x7fc69c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6aeffd640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c0047d0 0x7fc69c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6aeffd640 1 -- 172.21.15.7:0/1449051546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c003530 con 0x7fc69c003b40 2023-11-28T18:42:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.489+0000 7fc6a7fff640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c0047d0 0x7fc69c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:42:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6aeffd640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 0x7fc69c003f40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009ae0 tx=0x7fc6a001ca50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4c75f3c0760a617 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00a9420 con 0x7fc69c003b40 2023-11-28T18:42:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a00a9580 con 0x7fc69c003b40 2023-11-28T18:42:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6ae7fc640 1 -- 172.21.15.7:0/1449051546 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00a9930 con 0x7fc69c003b40 2023-11-28T18:42:24.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 msgr2=0x7fc69c003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:24.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6b57d8640 1 --2- 172.21.15.7:0/1449051546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 0x7fc69c003f40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009ae0 tx=0x7fc6a001ca50 comp rx=0 tx=0).stop 2023-11-28T18:42:24.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6af7fe640 1 -- 172.21.15.7:0/1449051546 reap_dead start 2023-11-28T18:42:24.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1449051546 shutdown_connections 2023-11-28T18:42:24.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1449051546 >> 172.21.15.7:0/1449051546 conn(0x7fc6b00cbf80 msgr2=0x7fc6b00cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:24.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1449051546 shutdown_connections 2023-11-28T18:42:24.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1449051546 wait complete. 2023-11-28T18:42:24.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6b57d8640 1 Processor -- start 2023-11-28T18:42:24.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.493+0000 7fc6b57d8640 1 -- start start 2023-11-28T18:42:24.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 0x7fc6b014feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:24.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b00c3380 0x7fc6b00c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:24.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b014a710 0x7fc6b014ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:24.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b00c5140 con 0x7fc6b00c3380 2023-11-28T18:42:24.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b0153c70 con 0x7fc69c003b40 2023-11-28T18:42:24.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b015df60 con 0x7fc6b014a710 2023-11-28T18:42:24.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b00c3380 0x7fc6b00c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:24.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b00c3380 0x7fc6b00c37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38182/0 (socket says 172.21.15.7:38182) 2023-11-28T18:42:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 -- 172.21.15.7:0/1677894196 learned_addr learned my addr 172.21.15.7:0/1677894196 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6aeffd640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 0x7fc6b014feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6af7fe640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b014a710 0x7fc6b014ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 -- 172.21.15.7:0/1677894196 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b014a710 msgr2=0x7fc6b014ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b014a710 0x7fc6b014ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 -- 172.21.15.7:0/1677894196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 msgr2=0x7fc6b014feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 0x7fc6b014feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 -- 172.21.15.7:0/1677894196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b015e240 con 0x7fc6b00c3380 2023-11-28T18:42:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a7fff640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b00c3380 0x7fc6b00c37c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc69800f880 tx=0x7fc69800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a77fe640 1 -- 172.21.15.7:0/1677894196 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6980052e0 con 0x7fc6b00c3380 2023-11-28T18:42:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a77fe640 1 -- 172.21.15.7:0/1677894196 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc698005440 con 0x7fc6b00c3380 2023-11-28T18:42:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b00c4840 con 0x7fc6b00c3380 2023-11-28T18:42:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6a77fe640 1 -- 172.21.15.7:0/1677894196 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69800d910 con 0x7fc6b00c3380 2023-11-28T18:42:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b0153ec0 con 0x7fc6b00c3380 2023-11-28T18:42:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.497+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b0154080 con 0x7fc6b00c3380 2023-11-28T18:42:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.501+0000 7fc6a77fe640 1 -- 172.21.15.7:0/1677894196 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc69800da70 con 0x7fc6b00c3380 2023-11-28T18:42:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.501+0000 7fc6a77fe640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6900668d0 0x7fc690068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.501+0000 7fc6a77fe640 1 -- 172.21.15.7:0/1677894196 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fc698084820 con 0x7fc6b00c3380 2023-11-28T18:42:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.501+0000 7fc6aeffd640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6900668d0 0x7fc690068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.501+0000 7fc6aeffd640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6900668d0 0x7fc690068d90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00ae000 tx=0x7fc6a00046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.505+0000 7fc6a77fe640 1 -- 172.21.15.7:0/1677894196 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc6980516a0 con 0x7fc6b00c3380 2023-11-28T18:42:24.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.693+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6b004e890 con 0x7fc6b00c3380 2023-11-28T18:42:24.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6a77fe640 1 -- 172.21.15.7:0/1677894196 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5770 (secure 0 0 0) 0x7fc698054cc0 con 0x7fc6b00c3380 2023-11-28T18:42:24.697 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:24.698 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:23.547525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:24.698 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 87 2023-11-28T18:42:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6900668d0 msgr2=0x7fc690068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6900668d0 0x7fc690068d90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00ae000 tx=0x7fc6a00046d0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b00c3380 msgr2=0x7fc6b00c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b00c3380 0x7fc6b00c37c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc69800f880 tx=0x7fc69800fd40 comp rx=0 tx=0).stop 2023-11-28T18:42:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 shutdown_connections 2023-11-28T18:42:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc6900668d0 0x7fc690068d90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc6b014a710 0x7fc6b014ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b00c3380 0x7fc6b00c37c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 --2- 172.21.15.7:0/1677894196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c003b40 0x7fc6b014feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 >> 172.21.15.7:0/1677894196 conn(0x7fc6b00cbf80 msgr2=0x7fc6b00c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 shutdown_connections 2023-11-28T18:42:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:24.697+0000 7fc6b57d8640 1 -- 172.21.15.7:0/1677894196 wait complete. 2023-11-28T18:42:24.719 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:42:26.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:26.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.873+0000 7f4e27646640 1 Processor -- start 2023-11-28T18:42:26.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.873+0000 7f4e27646640 1 -- start start 2023-11-28T18:42:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e27646640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e2006e560 0x7f4e2015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:26.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e27646640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 0x7f4e200701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e27646640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e20157fe0 con 0x7f4e20151cd0 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e27646640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e20070730 con 0x7f4e2006e560 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e27646640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e20071e60 con 0x7f4e20157b90 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e24be1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e2006e560 0x7f4e2015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e24be1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e2006e560 0x7f4e2015bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38188/0 (socket says 172.21.15.7:38188) 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e24be1640 1 -- 172.21.15.7:0/835926831 learned_addr learned my addr 172.21.15.7:0/835926831 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e25be3640 1 --2- 172.21.15.7:0/835926831 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 0x7f4e200701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e24be1640 1 -- 172.21.15.7:0/835926831 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 msgr2=0x7f4e200701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e24be1640 1 --2- 172.21.15.7:0/835926831 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 0x7f4e200701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:26.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e24be1640 1 -- 172.21.15.7:0/835926831 >> v1:172.21.15.193:6789/0 conn(0x7f4e20151cd0 legacy=0x7f4e200c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:26.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.877+0000 7f4e24be1640 1 -- 172.21.15.7:0/835926831 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e20070930 con 0x7f4e2006e560 2023-11-28T18:42:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e24be1640 1 --2- 172.21.15.7:0/835926831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e2015bd00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009870 tx=0x7f4e1001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2d02a04b5ebbc7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e17fff640 1 -- 172.21.15.7:0/835926831 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e100ac600 con 0x7f4e2006e560 2023-11-28T18:42:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e17fff640 1 -- 172.21.15.7:0/835926831 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e100a8050 con 0x7f4e2006e560 2023-11-28T18:42:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e17fff640 1 -- 172.21.15.7:0/835926831 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e100ac980 con 0x7f4e2006e560 2023-11-28T18:42:26.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 -- 172.21.15.7:0/835926831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 msgr2=0x7f4e2015bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:26.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 --2- 172.21.15.7:0/835926831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e2015bd00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009870 tx=0x7f4e1001c4b0 comp rx=0 tx=0).stop 2023-11-28T18:42:26.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 -- 172.21.15.7:0/835926831 shutdown_connections 2023-11-28T18:42:26.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 --2- 172.21.15.7:0/835926831 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 0x7f4e200701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:26.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 --2- 172.21.15.7:0/835926831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e2015bd00 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:26.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 -- 172.21.15.7:0/835926831 >> 172.21.15.7:0/835926831 conn(0x7f4e200662e0 msgr2=0x7f4e200666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:26.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 -- 172.21.15.7:0/835926831 shutdown_connections 2023-11-28T18:42:26.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 -- 172.21.15.7:0/835926831 wait complete. 2023-11-28T18:42:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 Processor -- start 2023-11-28T18:42:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 -- start start 2023-11-28T18:42:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e201446e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20151cd0 0x7f4e201439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.881+0000 7f4e27646640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 0x7f4e20158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e27646640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e201f6d30 con 0x7f4e2006e560 2023-11-28T18:42:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e27646640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e20145050 con 0x7f4e20151cd0 2023-11-28T18:42:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e27646640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e201f6790 con 0x7f4e20157b90 2023-11-28T18:42:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e201446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e201446e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38202/0 (socket says 172.21.15.7:38202) 2023-11-28T18:42:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 -- 172.21.15.7:0/386527387 learned_addr learned my addr 172.21.15.7:0/386527387 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e25be3640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 0x7f4e20158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:26.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e24be1640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20151cd0 0x7f4e201439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 -- 172.21.15.7:0/386527387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 msgr2=0x7f4e20158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 0x7f4e20158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 -- 172.21.15.7:0/386527387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20151cd0 msgr2=0x7f4e201439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20151cd0 0x7f4e201439a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 -- 172.21.15.7:0/386527387 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100094d0 con 0x7f4e2006e560 2023-11-28T18:42:26.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e253e2640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e201446e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00c500 tx=0x7f4e1c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:26.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e15ffb640 1 -- 172.21.15.7:0/386527387 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c005220 con 0x7f4e2006e560 2023-11-28T18:42:26.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e15ffb640 1 -- 172.21.15.7:0/386527387 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c005380 con 0x7f4e2006e560 2023-11-28T18:42:26.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e201f6a50 con 0x7f4e2006e560 2023-11-28T18:42:26.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e15ffb640 1 -- 172.21.15.7:0/386527387 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c007270 con 0x7f4e2006e560 2023-11-28T18:42:26.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2015aeb0 con 0x7f4e2006e560 2023-11-28T18:42:26.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2004e890 con 0x7f4e2006e560 2023-11-28T18:42:26.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.885+0000 7f4e15ffb640 1 -- 172.21.15.7:0/386527387 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4e1c0073d0 con 0x7f4e2006e560 2023-11-28T18:42:26.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.889+0000 7f4e15ffb640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4e04066900 0x7f4e04068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:26.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.889+0000 7f4e15ffb640 1 -- 172.21.15.7:0/386527387 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f4e1c083ff0 con 0x7f4e2006e560 2023-11-28T18:42:26.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.889+0000 7f4e24be1640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4e04066900 0x7f4e04068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:26.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.889+0000 7f4e24be1640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4e04066900 0x7f4e04068dc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4e100ae000 tx=0x7f4e100a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:26.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:26.893+0000 7f4e15ffb640 1 -- 172.21.15.7:0/386527387 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4e1c050e80 con 0x7f4e2006e560 2023-11-28T18:42:27.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.081+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e201555b0 con 0x7f4e2006e560 2023-11-28T18:42:27.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e15ffb640 1 -- 172.21.15.7:0/386527387 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5770 (secure 0 0 0) 0x7f4e1c0544a0 con 0x7f4e2006e560 2023-11-28T18:42:27.085 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:27.085 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:23.547525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:27.086 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 87 2023-11-28T18:42:27.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4e04066900 msgr2=0x7f4e04068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:27.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4e04066900 0x7f4e04068dc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4e100ae000 tx=0x7f4e100a7000 comp rx=0 tx=0).stop 2023-11-28T18:42:27.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 msgr2=0x7f4e201446e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:27.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e201446e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00c500 tx=0x7f4e1c00c9c0 comp rx=0 tx=0).stop 2023-11-28T18:42:27.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 shutdown_connections 2023-11-28T18:42:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4e04066900 0x7f4e04068dc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4e20157b90 0x7f4e20158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20151cd0 0x7f4e201439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 --2- 172.21.15.7:0/386527387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2006e560 0x7f4e201446e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 >> 172.21.15.7:0/386527387 conn(0x7f4e200662e0 msgr2=0x7f4e2014b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 shutdown_connections 2023-11-28T18:42:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:27.085+0000 7f4e27646640 1 -- 172.21.15.7:0/386527387 wait complete. 2023-11-28T18:42:27.107 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:42:29.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:29.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa261903640 1 Processor -- start 2023-11-28T18:42:29.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa261903640 1 -- start start 2023-11-28T18:42:29.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa261903640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c15ce10 0x7fa25c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:29.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa261903640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c15a3a0 con 0x7fa25c15ca30 2023-11-28T18:42:29.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa261903640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c15a620 con 0x7fa25c0cafe0 2023-11-28T18:42:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa261903640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c15a900 con 0x7fa25c15ce10 2023-11-28T18:42:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa260901640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa25c15ca30 0x7fa25c159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58356/0 (socket says 172.21.15.7:58356) 2023-11-28T18:42:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa260901640 1 -- 172.21.15.7:0/248190887 learned_addr learned my addr 172.21.15.7:0/248190887 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:42:29.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa25bfff640 1 --2- 172.21.15.7:0/248190887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c15ce10 0x7fa25c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:29.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa25b7fe640 1 -- 172.21.15.7:0/248190887 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216599757 0 0) 0x7fa25c15a3a0 con 0x7fa25c15ca30 2023-11-28T18:42:29.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa25b7fe640 1 -- 172.21.15.7:0/248190887 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa24c003530 con 0x7fa25c15ca30 2023-11-28T18:42:29.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa25bfff640 1 -- 172.21.15.7:0/248190887 >> v1:172.21.15.193:6789/0 conn(0x7fa25c0cafe0 legacy=0x7fa25c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:29.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa25bfff640 1 -- 172.21.15.7:0/248190887 >> v1:172.21.15.7:6789/0 conn(0x7fa25c15ca30 legacy=0x7fa25c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:29.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.265+0000 7fa25bfff640 1 -- 172.21.15.7:0/248190887 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c15abe0 con 0x7fa25c15ce10 2023-11-28T18:42:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa25bfff640 1 --2- 172.21.15.7:0/248190887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c15ce10 0x7fa25c0ca1d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa2480097f0 tx=0x7fa24801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb58980cfd88d88 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa25b7fe640 1 -- 172.21.15.7:0/248190887 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480a89b0 con 0x7fa25c15ce10 2023-11-28T18:42:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa25b7fe640 1 -- 172.21.15.7:0/248190887 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2480a8b10 con 0x7fa25c15ce10 2023-11-28T18:42:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa25b7fe640 1 -- 172.21.15.7:0/248190887 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480a8ea0 con 0x7fa25c15ce10 2023-11-28T18:42:29.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa261903640 1 -- 172.21.15.7:0/248190887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c15ce10 msgr2=0x7fa25c0ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:29.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa261903640 1 --2- 172.21.15.7:0/248190887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c15ce10 0x7fa25c0ca1d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa2480097f0 tx=0x7fa24801ecb0 comp rx=0 tx=0).stop 2023-11-28T18:42:29.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa261903640 1 -- 172.21.15.7:0/248190887 shutdown_connections 2023-11-28T18:42:29.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa261903640 1 --2- 172.21.15.7:0/248190887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c15ce10 0x7fa25c0ca1d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:29.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.269+0000 7fa261903640 1 -- 172.21.15.7:0/248190887 >> 172.21.15.7:0/248190887 conn(0x7fa25c06c060 msgr2=0x7fa25c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:29.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 -- 172.21.15.7:0/248190887 shutdown_connections 2023-11-28T18:42:29.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 -- 172.21.15.7:0/248190887 wait complete. 2023-11-28T18:42:29.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 Processor -- start 2023-11-28T18:42:29.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 -- start start 2023-11-28T18:42:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c0cafe0 0x7fa25c1f01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa25c15ca30 0x7fa25c150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa25c15ce10 0x7fa25c151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:29.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c14faf0 con 0x7fa25c15ca30 2023-11-28T18:42:29.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c151a20 con 0x7fa25c15ce10 2023-11-28T18:42:29.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa261903640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c157670 con 0x7fa25c0cafe0 2023-11-28T18:42:29.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.273+0000 7fa25bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa25c15ca30 0x7fa25c150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:29.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa25bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa25c15ca30 0x7fa25c150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38204/0 (socket says 172.21.15.7:38204) 2023-11-28T18:42:29.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa25bfff640 1 -- 172.21.15.7:0/675677592 learned_addr learned my addr 172.21.15.7:0/675677592 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa260901640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c0cafe0 0x7fa25c1f01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa261102640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa25c15ce10 0x7fa25c151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa25bfff640 1 -- 172.21.15.7:0/675677592 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c0cafe0 msgr2=0x7fa25c1f01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa25bfff640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c0cafe0 0x7fa25c1f01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa25bfff640 1 -- 172.21.15.7:0/675677592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa25c15ce10 msgr2=0x7fa25c151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa25bfff640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa25c15ce10 0x7fa25c151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa25bfff640 1 -- 172.21.15.7:0/675677592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248009420 con 0x7fa25c15ca30 2023-11-28T18:42:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa25bfff640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa25c15ca30 0x7fa25c150710 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa2480097f0 tx=0x7fa24801ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa2597fa640 1 -- 172.21.15.7:0/675677592 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b9630 con 0x7fa25c15ca30 2023-11-28T18:42:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa2597fa640 1 -- 172.21.15.7:0/675677592 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2480b97b0 con 0x7fa25c15ca30 2023-11-28T18:42:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa2597fa640 1 -- 172.21.15.7:0/675677592 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b9b60 con 0x7fa25c15ca30 2023-11-28T18:42:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa261903640 1 -- 172.21.15.7:0/675677592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c156b10 con 0x7fa25c15ca30 2023-11-28T18:42:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa261903640 1 -- 172.21.15.7:0/675677592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c1570b0 con 0x7fa25c15ca30 2023-11-28T18:42:29.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa246ffd640 1 -- 172.21.15.7:0/675677592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c1f06e0 con 0x7fa25c15ca30 2023-11-28T18:42:29.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.277+0000 7fa2597fa640 1 -- 172.21.15.7:0/675677592 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa2480b7040 con 0x7fa25c15ca30 2023-11-28T18:42:29.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.281+0000 7fa2597fa640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa240066900 0x7fa240068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:29.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.281+0000 7fa2597fa640 1 -- 172.21.15.7:0/675677592 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fa2481225f0 con 0x7fa25c15ca30 2023-11-28T18:42:29.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.281+0000 7fa260901640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa240066900 0x7fa240068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:29.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.281+0000 7fa260901640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa240066900 0x7fa240068dc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa2500011f0 tx=0x7fa250007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:29.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.285+0000 7fa2597fa640 1 -- 172.21.15.7:0/675677592 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa25c1f06e0 con 0x7fa25c15ca30 2023-11-28T18:42:29.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.517+0000 7fa246ffd640 1 -- 172.21.15.7:0/675677592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa25c1f06e0 con 0x7fa25c15ca30 2023-11-28T18:42:29.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa2597fa640 1 -- 172.21.15.7:0/675677592 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5770 (secure 0 0 0) 0x7fa25c1f06e0 con 0x7fa25c15ca30 2023-11-28T18:42:29.521 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:29.522 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:28.798985+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:29.522 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 88 2023-11-28T18:42:29.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 -- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa240066900 msgr2=0x7fa240068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:29.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa240066900 0x7fa240068dc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa2500011f0 tx=0x7fa250007d40 comp rx=0 tx=0).stop 2023-11-28T18:42:29.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 -- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa25c15ca30 msgr2=0x7fa25c150710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:29.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa25c15ca30 0x7fa25c150710 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa2480097f0 tx=0x7fa24801ba50 comp rx=0 tx=0).stop 2023-11-28T18:42:29.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 -- 172.21.15.7:0/675677592 shutdown_connections 2023-11-28T18:42:29.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa240066900 0x7fa240068dc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:29.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa25c15ce10 0x7fa25c151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:29.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa25c15ca30 0x7fa25c150710 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:29.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 --2- 172.21.15.7:0/675677592 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa25c0cafe0 0x7fa25c1f01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:29.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 -- 172.21.15.7:0/675677592 >> 172.21.15.7:0/675677592 conn(0x7fa25c06c060 msgr2=0x7fa25c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:29.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 -- 172.21.15.7:0/675677592 shutdown_connections 2023-11-28T18:42:29.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:29.521+0000 7fa246ffd640 1 -- 172.21.15.7:0/675677592 wait complete. 2023-11-28T18:42:29.547 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=5359 2023-11-28T18:42:29.547 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2023-11-28T18:42:33.548 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-11-28T18:42:33.548 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T18:42:33.548 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T18:42:33.552 INFO:tasks.ceph.mds.d:Started 2023-11-28T18:42:33.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:33.644 INFO:tasks.ceph.mds.d.smithi193.stdout:starting mds.d at 2023-11-28T18:42:33.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2beefaa640 1 Processor -- start 2023-11-28T18:42:33.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2beefaa640 1 -- start start 2023-11-28T18:42:33.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2beefaa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2be8164410 0x7f2be8166bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:33.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2beefaa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be80c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:33.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2beefaa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be80c75a0 con 0x7f2be816ab90 2023-11-28T18:42:33.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2beefaa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be816a140 con 0x7f2be8164410 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2beefaa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be816afe0 con 0x7f2be80c6660 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2becd46640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2be8164410 0x7f2be8166bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2be7fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2be816ab90 0x7f2be80c2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34950/0 (socket says 172.21.15.7:34950) 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2becd46640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2be8164410 0x7f2be8166bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41974/0 (socket says 172.21.15.7:41974) 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2be7fff640 1 -- 172.21.15.7:0/633094545 learned_addr learned my addr 172.21.15.7:0/633094545 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2bed547640 1 --2- 172.21.15.7:0/633094545 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be80c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.709+0000 7f2be77fe640 1 -- 172.21.15.7:0/633094545 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998995220 0 0) 0x7f2be80c75a0 con 0x7f2be816ab90 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.849+0000 7f2be77fe640 1 -- 172.21.15.7:0/633094545 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd8007940 con 0x7f2be816ab90 2023-11-28T18:42:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.849+0000 7f2be77fe640 1 -- 172.21.15.7:0/633094545 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 201539203 0 0) 0x7f2bd8007940 con 0x7f2be816ab90 2023-11-28T18:42:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.849+0000 7f2becd46640 1 -- 172.21.15.7:0/633094545 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 msgr2=0x7f2be80c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.849+0000 7f2becd46640 1 --2- 172.21.15.7:0/633094545 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be80c6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.849+0000 7f2becd46640 1 -- 172.21.15.7:0/633094545 >> v1:172.21.15.7:6789/0 conn(0x7f2be816ab90 legacy=0x7f2be80c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.849+0000 7f2becd46640 1 -- 172.21.15.7:0/633094545 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80c81d0 con 0x7f2be8164410 2023-11-28T18:42:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2becd46640 1 --2- 172.21.15.7:0/633094545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be8164410 0x7f2be8166bd0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc014fb0 tx=0x7f2bdc024d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bcc6bfc445ae1a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2be77fe640 1 -- 172.21.15.7:0/633094545 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4001680 con 0x7f2be8164410 2023-11-28T18:42:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2be77fe640 1 -- 172.21.15.7:0/633094545 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc0adb80 con 0x7f2be8164410 2023-11-28T18:42:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2be77fe640 1 -- 172.21.15.7:0/633094545 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc0b5440 con 0x7f2be8164410 2023-11-28T18:42:33.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2beefaa640 1 -- 172.21.15.7:0/633094545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be8164410 msgr2=0x7f2be8166bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:33.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2beefaa640 1 --2- 172.21.15.7:0/633094545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be8164410 0x7f2be8166bd0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc014fb0 tx=0x7f2bdc024d50 comp rx=0 tx=0).stop 2023-11-28T18:42:33.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2beefaa640 1 -- 172.21.15.7:0/633094545 shutdown_connections 2023-11-28T18:42:33.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2beefaa640 1 --2- 172.21.15.7:0/633094545 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be80c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:33.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2beefaa640 1 --2- 172.21.15.7:0/633094545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be8164410 0x7f2be8166bd0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:33.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2beefaa640 1 -- 172.21.15.7:0/633094545 >> 172.21.15.7:0/633094545 conn(0x7f2be80662e0 msgr2=0x7f2be80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:33.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2beefaa640 1 -- 172.21.15.7:0/633094545 shutdown_connections 2023-11-28T18:42:33.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.853+0000 7f2beefaa640 1 -- 172.21.15.7:0/633094545 wait complete. 2023-11-28T18:42:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2beefaa640 1 Processor -- start 2023-11-28T18:42:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2beefaa640 1 -- start start 2023-11-28T18:42:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2beefaa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be8164ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2beefaa640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2be8164410 0x7f2be815e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2beefaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be816ab90 0x7f2be81feef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2beefaa640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be815b7f0 con 0x7f2be816ab90 2023-11-28T18:42:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2beefaa640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be815ae90 con 0x7f2be8164410 2023-11-28T18:42:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2beefaa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be8165140 con 0x7f2be80c6660 2023-11-28T18:42:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be816ab90 0x7f2be81feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be816ab90 0x7f2be81feef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41982/0 (socket says 172.21.15.7:41982) 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 -- 172.21.15.7:0/2455502211 learned_addr learned my addr 172.21.15.7:0/2455502211 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2becd46640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be8164ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2be7fff640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2be8164410 0x7f2be815e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 -- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 msgr2=0x7f2be8164ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be8164ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 -- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2be8164410 msgr2=0x7f2be815e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2be8164410 0x7f2be815e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 -- 172.21.15.7:0/2455502211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc014c10 con 0x7f2be816ab90 2023-11-28T18:42:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2becd46640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be8164ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:42:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2bed547640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be816ab90 0x7f2be81feef0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2be000fcd0 tx=0x7f2be000d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.857+0000 7f2be57fa640 1 -- 172.21.15.7:0/2455502211 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0007850 con 0x7f2be816ab90 2023-11-28T18:42:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2be57fa640 1 -- 172.21.15.7:0/2455502211 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be00125f0 con 0x7f2be816ab90 2023-11-28T18:42:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2be57fa640 1 -- 172.21.15.7:0/2455502211 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be000dee0 con 0x7f2be816ab90 2023-11-28T18:42:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be815b130 con 0x7f2be816ab90 2023-11-28T18:42:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be8159020 con 0x7f2be816ab90 2023-11-28T18:42:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be815b930 con 0x7f2be816ab90 2023-11-28T18:42:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2be57fa640 1 -- 172.21.15.7:0/2455502211 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2be00129c0 con 0x7f2be816ab90 2023-11-28T18:42:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2be57fa640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2bcc0668d0 0x7f2bcc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2be57fa640 1 -- 172.21.15.7:0/2455502211 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f2be00850f0 con 0x7f2be816ab90 2023-11-28T18:42:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.861+0000 7f2becd46640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2bcc0668d0 0x7f2bcc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.865+0000 7f2becd46640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2bcc0668d0 0x7f2bcc068d90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0b7000 tx=0x7f2bdc0af070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:33.865+0000 7f2be57fa640 1 -- 172.21.15.7:0/2455502211 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2be815b930 con 0x7f2be816ab90 2023-11-28T18:42:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.061+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2be815b930 con 0x7f2be816ab90 2023-11-28T18:42:34.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.061+0000 7f2be57fa640 1 -- 172.21.15.7:0/2455502211 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5770 (secure 0 0 0) 0x7f2be815b930 con 0x7f2be816ab90 2023-11-28T18:42:34.064 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:34.064 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:28.798985+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 88 2023-11-28T18:42:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2bcc0668d0 msgr2=0x7f2bcc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2bcc0668d0 0x7f2bcc068d90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0b7000 tx=0x7f2bdc0af070 comp rx=0 tx=0).stop 2023-11-28T18:42:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be816ab90 msgr2=0x7f2be81feef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be816ab90 0x7f2be81feef0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2be000fcd0 tx=0x7f2be000d5c0 comp rx=0 tx=0).stop 2023-11-28T18:42:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 shutdown_connections 2023-11-28T18:42:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2bcc0668d0 0x7f2bcc068d90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be816ab90 0x7f2be81feef0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2be8164410 0x7f2be815e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 --2- 172.21.15.7:0/2455502211 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2be80c6660 0x7f2be8164ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 >> 172.21.15.7:0/2455502211 conn(0x7f2be80662e0 msgr2=0x7f2be80666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 shutdown_connections 2023-11-28T18:42:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:34.065+0000 7f2beefaa640 1 -- 172.21.15.7:0/2455502211 wait complete. 2023-11-28T18:42:34.085 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-11-28T18:42:36.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:42:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.333+0000 7f1c7a45a640 1 Processor -- start 2023-11-28T18:42:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.333+0000 7f1c7a45a640 1 -- start start 2023-11-28T18:42:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c7a45a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c7415bd00 0x7f1c74157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c7a45a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c74071e60 con 0x7f1c7406e560 2023-11-28T18:42:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c7a45a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c74070930 con 0x7f1c74151cd0 2023-11-28T18:42:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c7a45a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7415c150 con 0x7f1c7415bd00 2023-11-28T18:42:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c73fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f1c74151cd0 0x7f1c740c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:59806/0 (socket says 172.21.15.7:59806) 2023-11-28T18:42:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c73fff640 1 -- 172.21.15.7:0/2369115470 learned_addr learned my addr 172.21.15.7:0/2369115470 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:42:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c789f7640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c7415bd00 0x7f1c74157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379285693 0 0) 0x7f1c74070930 con 0x7f1c74151cd0 2023-11-28T18:42:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c5c003530 con 0x7f1c74151cd0 2023-11-28T18:42:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72239356 0 0) 0x7f1c74071e60 con 0x7f1c7406e560 2023-11-28T18:42:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c74070930 con 0x7f1c7406e560 2023-11-28T18:42:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584844117 0 0) 0x7f1c5c003530 con 0x7f1c74151cd0 2023-11-28T18:42:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c7415bd00 msgr2=0x7f1c74157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c7415bd00 0x7f1c74157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 >> v1:172.21.15.7:6789/0 conn(0x7f1c7406e560 legacy=0x7f1c74160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c74070bf0 con 0x7f1c74151cd0 2023-11-28T18:42:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f1c64001680 con 0x7f1c74151cd0 2023-11-28T18:42:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 >> v1:172.21.15.193:6789/0 conn(0x7f1c74151cd0 legacy=0x7f1c740c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c5c003b10 0x7f1c5c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5c0047d0 0x7f1c5c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c5c005190 0x7f1c5c00fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7415c150 con 0x7f1c5c0047d0 2023-11-28T18:42:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c010470 con 0x7f1c5c005190 2023-11-28T18:42:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.337+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c010630 con 0x7f1c5c003b10 2023-11-28T18:42:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.341+0000 7f1c73fff640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c5c003b10 0x7f1c5c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.341+0000 7f1c737fe640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5c0047d0 0x7f1c5c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:36.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.341+0000 7f1c789f7640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c5c005190 0x7f1c5c00fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:36.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.341+0000 7f1c737fe640 1 -- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c5c003b10 msgr2=0x7f1c5c003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:36.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.341+0000 7f1c737fe640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c5c003b10 0x7f1c5c003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.341+0000 7f1c737fe640 1 -- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c5c005190 msgr2=0x7f1c5c00fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:42:36.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.341+0000 7f1c737fe640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c5c005190 0x7f1c5c00fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.341+0000 7f1c737fe640 1 -- 172.21.15.7:0/2369115470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c003530 con 0x7f1c5c0047d0 2023-11-28T18:42:36.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.345+0000 7f1c737fe640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5c0047d0 0x7f1c5c004c50 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1c680018c0 tx=0x7f1c6801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39cbdfb208920fb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.345+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c680052d0 con 0x7f1c5c0047d0 2023-11-28T18:42:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.345+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c68005430 con 0x7f1c5c0047d0 2023-11-28T18:42:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.345+0000 7f1c72ffd640 1 -- 172.21.15.7:0/2369115470 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c680a8910 con 0x7f1c5c0047d0 2023-11-28T18:42:36.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.345+0000 7f1c7a45a640 1 -- 172.21.15.7:0/2369115470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5c0047d0 msgr2=0x7f1c5c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:36.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.345+0000 7f1c7a45a640 1 --2- 172.21.15.7:0/2369115470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5c0047d0 0x7f1c5c004c50 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1c680018c0 tx=0x7f1c6801ecd0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.349+0000 7f1c789f7640 1 -- 172.21.15.7:0/2369115470 reap_dead start 2023-11-28T18:42:36.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.349+0000 7f1c7a45a640 1 -- 172.21.15.7:0/2369115470 shutdown_connections 2023-11-28T18:42:36.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.349+0000 7f1c7a45a640 1 -- 172.21.15.7:0/2369115470 >> 172.21.15.7:0/2369115470 conn(0x7f1c740662e0 msgr2=0x7f1c740666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.353+0000 7f1c7a45a640 1 -- 172.21.15.7:0/2369115470 shutdown_connections 2023-11-28T18:42:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.353+0000 7f1c7a45a640 1 -- 172.21.15.7:0/2369115470 wait complete. 2023-11-28T18:42:36.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.357+0000 7f1c7a45a640 1 Processor -- start 2023-11-28T18:42:36.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.361+0000 7f1c7a45a640 1 -- start start 2023-11-28T18:42:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c7a45a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c74150b10 0x7f1c7415aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c7a45a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c740ca960 0x7f1c74155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c7a45a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c74150ef0 0x7f1c741f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c7a45a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741f4ba0 con 0x7f1c74150ef0 2023-11-28T18:42:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c7a45a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740cb070 con 0x7f1c740ca960 2023-11-28T18:42:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c7a45a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740cb350 con 0x7f1c74150b10 2023-11-28T18:42:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c73fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c74150b10 0x7f1c7415aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c73fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c74150b10 0x7f1c7415aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:56868/0 (socket says 172.21.15.7:56868) 2023-11-28T18:42:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c73fff640 1 -- 172.21.15.7:0/3206097830 learned_addr learned my addr 172.21.15.7:0/3206097830 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:42:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c789f7640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c74150ef0 0x7f1c741f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c737fe640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c740ca960 0x7f1c74155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c789f7640 1 -- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c74150b10 msgr2=0x7f1c7415aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c789f7640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c74150b10 0x7f1c7415aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c789f7640 1 -- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c740ca960 msgr2=0x7f1c74155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c789f7640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c740ca960 0x7f1c74155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c789f7640 1 -- 172.21.15.7:0/3206097830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c741f44e0 con 0x7f1c74150ef0 2023-11-28T18:42:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c73fff640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c74150b10 0x7f1c7415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:42:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c737fe640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c740ca960 0x7f1c74155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:42:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.365+0000 7f1c789f7640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c74150ef0 0x7f1c741f7740 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1c740c92e0 tx=0x7f1c6c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.369+0000 7f1c70ff9640 1 -- 172.21.15.7:0/3206097830 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c009e70 con 0x7f1c74150ef0 2023-11-28T18:42:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.369+0000 7f1c70ff9640 1 -- 172.21.15.7:0/3206097830 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c00f050 con 0x7f1c74150ef0 2023-11-28T18:42:36.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.369+0000 7f1c70ff9640 1 -- 172.21.15.7:0/3206097830 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c014670 con 0x7f1c74150ef0 2023-11-28T18:42:36.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.369+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c741f47c0 con 0x7f1c74150ef0 2023-11-28T18:42:36.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.369+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c741590d0 con 0x7f1c74150ef0 2023-11-28T18:42:36.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.369+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c741f4ce0 con 0x7f1c74150ef0 2023-11-28T18:42:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.373+0000 7f1c70ff9640 1 -- 172.21.15.7:0/3206097830 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1c6c0147d0 con 0x7f1c74150ef0 2023-11-28T18:42:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.373+0000 7f1c70ff9640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1c60066950 0x7f1c60068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:42:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.373+0000 7f1c70ff9640 1 -- 172.21.15.7:0/3206097830 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f1c6c084bb0 con 0x7f1c74150ef0 2023-11-28T18:42:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.373+0000 7f1c73fff640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1c60066950 0x7f1c60068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:42:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.373+0000 7f1c73fff640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1c60066950 0x7f1c60068e10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1c64001060 tx=0x7f1c64000fb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:42:36.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.385+0000 7f1c70ff9640 1 -- 172.21.15.7:0/3206097830 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1c741f4ce0 con 0x7f1c74150ef0 2023-11-28T18:42:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.753+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c741f4ce0 con 0x7f1c74150ef0 2023-11-28T18:42:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.757+0000 7f1c70ff9640 1 -- 172.21.15.7:0/3206097830 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f1c741f4ce0 con 0x7f1c74150ef0 2023-11-28T18:42:36.757 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:42:36.757 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:42:28.798985+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:42:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 89 2023-11-28T18:42:36.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.757+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1c60066950 msgr2=0x7f1c60068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:36.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.757+0000 7f1c7a45a640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1c60066950 0x7f1c60068e10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1c64001060 tx=0x7f1c64000fb0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.757+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c74150ef0 msgr2=0x7f1c741f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:42:36.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.757+0000 7f1c7a45a640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c74150ef0 0x7f1c741f7740 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1c740c92e0 tx=0x7f1c6c007700 comp rx=0 tx=0).stop 2023-11-28T18:42:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.761+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 shutdown_connections 2023-11-28T18:42:36.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.761+0000 7f1c7a45a640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1c60066950 0x7f1c60068e10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.761+0000 7f1c7a45a640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c74150ef0 0x7f1c741f7740 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.761+0000 7f1c7a45a640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c740ca960 0x7f1c74155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.761+0000 7f1c7a45a640 1 --2- 172.21.15.7:0/3206097830 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1c74150b10 0x7f1c7415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:42:36.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.761+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 >> 172.21.15.7:0/3206097830 conn(0x7f1c740662e0 msgr2=0x7f1c7414c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:42:36.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.761+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 shutdown_connections 2023-11-28T18:42:36.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:42:36.761+0000 7f1c7a45a640 1 -- 172.21.15.7:0/3206097830 wait complete. 2023-11-28T18:42:36.800 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-11-28T18:42:36.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-11-28T18:42:39.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:42:39.488 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:43:09.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:43:09.584 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:43:15.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:16.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.068+0000 7fea5e755640 1 Processor -- start 2023-11-28T18:43:16.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.068+0000 7fea5e755640 1 -- start start 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea5e755640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea58162200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea5e755640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea58162740 con 0x7fea58066e50 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea5e755640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5806fb70 con 0x7fea58158300 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea5e755640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5815ecd0 con 0x7fea581586e0 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea5ccf2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fea58066e50 0x7fea580662e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55892/0 (socket says 172.21.15.7:55892) 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea5ccf2640 1 -- 172.21.15.7:0/491176516 learned_addr learned my addr 172.21.15.7:0/491176516 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212129322 0 0) 0x7fea500014a0 con 0x7fea58066e50 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea40003530 con 0x7fea58066e50 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439096143 0 0) 0x7fea40003530 con 0x7fea58066e50 2023-11-28T18:43:16.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 msgr2=0x7fea58162200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea58162200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 >> v1:172.21.15.193:6789/0 conn(0x7fea58158300 legacy=0x7fea580c7b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5815f160 con 0x7fea58066e50 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fea50001b90 con 0x7fea58066e50 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 >> v1:172.21.15.7:6789/0 conn(0x7fea58066e50 legacy=0x7fea580662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea400040a0 0x7fea400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea40004d60 0x7fea400051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5815ecd0 con 0x7fea40004d60 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4001cc40 con 0x7fea400040a0 2023-11-28T18:43:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.072+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4001ce00 con 0x7fea581586e0 2023-11-28T18:43:16.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.076+0000 7fea57fff640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:16.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.076+0000 7fea577fe640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea400040a0 0x7fea400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:16.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.076+0000 7fea57fff640 1 -- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea400040a0 msgr2=0x7fea400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:16.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.076+0000 7fea57fff640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea400040a0 0x7fea400047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.076+0000 7fea57fff640 1 -- 172.21.15.7:0/491176516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea40004d60 msgr2=0x7fea400051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:16.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.076+0000 7fea57fff640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea40004d60 0x7fea400051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.076+0000 7fea57fff640 1 -- 172.21.15.7:0/491176516 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5815f160 con 0x7fea581586e0 2023-11-28T18:43:16.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.080+0000 7fea57fff640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea400039b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fea480097e0 tx=0x7fea4801c2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4badebd85ff74a4f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:16.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.080+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480ab600 con 0x7fea581586e0 2023-11-28T18:43:16.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.080+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea480a7050 con 0x7fea581586e0 2023-11-28T18:43:16.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.080+0000 7fea56ffd640 1 -- 172.21.15.7:0/491176516 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480ab9c0 con 0x7fea581586e0 2023-11-28T18:43:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.080+0000 7fea5e755640 1 -- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 msgr2=0x7fea400039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.080+0000 7fea5e755640 1 --2- 172.21.15.7:0/491176516 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea400039b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fea480097e0 tx=0x7fea4801c2e0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.080+0000 7fea5ccf2640 1 -- 172.21.15.7:0/491176516 reap_dead start 2023-11-28T18:43:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.084+0000 7fea5e755640 1 -- 172.21.15.7:0/491176516 shutdown_connections 2023-11-28T18:43:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.084+0000 7fea5e755640 1 -- 172.21.15.7:0/491176516 >> 172.21.15.7:0/491176516 conn(0x7fea580cc490 msgr2=0x7fea580cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.084+0000 7fea5e755640 1 -- 172.21.15.7:0/491176516 shutdown_connections 2023-11-28T18:43:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.084+0000 7fea5e755640 1 -- 172.21.15.7:0/491176516 wait complete. 2023-11-28T18:43:16.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.084+0000 7fea5e755640 1 Processor -- start 2023-11-28T18:43:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.084+0000 7fea5e755640 1 -- start start 2023-11-28T18:43:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5e755640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea581eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5e755640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea581eed30 0x7fea5813e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5e755640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea581448a0 0x7fea58144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5e755640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581efd10 con 0x7fea581448a0 2023-11-28T18:43:16.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5e755640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581ef470 con 0x7fea581eed30 2023-11-28T18:43:16.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5e755640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea581ef750 con 0x7fea581586e0 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea581448a0 0x7fea58144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea581448a0 0x7fea58144d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34524/0 (socket says 172.21.15.7:34524) 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 -- 172.21.15.7:0/103808038 learned_addr learned my addr 172.21.15.7:0/103808038 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 -- 172.21.15.7:0/103808038 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 msgr2=0x7fea581eb7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea581eb7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 -- 172.21.15.7:0/103808038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea581eed30 msgr2=0x7fea5813e100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea581eed30 0x7fea5813e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 -- 172.21.15.7:0/103808038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea581efa30 con 0x7fea581448a0 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea5ccf2640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea581448a0 0x7fea58144d20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fea50001600 tx=0x7fea5000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.088+0000 7fea54ff9640 1 -- 172.21.15.7:0/103808038 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5000a790 con 0x7fea581448a0 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.092+0000 7fea54ff9640 1 -- 172.21.15.7:0/103808038 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5000a8f0 con 0x7fea581448a0 2023-11-28T18:43:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.092+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea580c7b00 con 0x7fea581448a0 2023-11-28T18:43:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.092+0000 7fea54ff9640 1 -- 172.21.15.7:0/103808038 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5000ace0 con 0x7fea581448a0 2023-11-28T18:43:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.092+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5813ffa0 con 0x7fea581448a0 2023-11-28T18:43:16.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.092+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea580c9910 con 0x7fea581448a0 2023-11-28T18:43:16.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.092+0000 7fea54ff9640 1 -- 172.21.15.7:0/103808038 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fea50007070 con 0x7fea581448a0 2023-11-28T18:43:16.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.096+0000 7fea54ff9640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fea44066920 0x7fea44068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.096+0000 7fea54ff9640 1 -- 172.21.15.7:0/103808038 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fea50084490 con 0x7fea581448a0 2023-11-28T18:43:16.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.096+0000 7fea57fff640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fea44066920 0x7fea44068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:16.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.096+0000 7fea57fff640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fea44066920 0x7fea44068de0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fea480ad000 tx=0x7fea48004780 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:16.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.100+0000 7fea54ff9640 1 -- 172.21.15.7:0/103808038 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fea500512e0 con 0x7fea581448a0 2023-11-28T18:43:16.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.492+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea58162740 con 0x7fea581448a0 2023-11-28T18:43:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.492+0000 7fea54ff9640 1 -- 172.21.15.7:0/103808038 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6560 (secure 0 0 0) 0x7fea58162740 con 0x7fea581448a0 2023-11-28T18:43:16.497 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:16.497 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:14.513274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:16.497 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 91 2023-11-28T18:43:16.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fea44066920 msgr2=0x7fea44068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fea44066920 0x7fea44068de0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fea480ad000 tx=0x7fea48004780 comp rx=0 tx=0).stop 2023-11-28T18:43:16.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea581448a0 msgr2=0x7fea58144d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea581448a0 0x7fea58144d20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fea50001600 tx=0x7fea5000cf00 comp rx=0 tx=0).stop 2023-11-28T18:43:16.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 shutdown_connections 2023-11-28T18:43:16.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fea44066920 0x7fea44068de0 secure :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fea480ad000 tx=0x7fea48004780 comp rx=0 tx=0).stop 2023-11-28T18:43:16.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea581448a0 0x7fea58144d20 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea581eed30 0x7fea5813e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 --2- 172.21.15.7:0/103808038 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fea581586e0 0x7fea581eb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.496+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 >> 172.21.15.7:0/103808038 conn(0x7fea580cc490 msgr2=0x7fea58155750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:16.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.500+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 shutdown_connections 2023-11-28T18:43:16.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.500+0000 7fea5e755640 1 -- 172.21.15.7:0/103808038 wait complete. 2023-11-28T18:43:16.545 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=2) 2023-11-28T18:43:16.545 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-11-28T18:43:16.545 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:43:16.563 INFO:tasks.ceph.mds.f:Stopped 2023-11-28T18:43:16.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f4029c640 1 Processor -- start 2023-11-28T18:43:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f4029c640 1 -- start start 2023-11-28T18:43:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f4029c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1f38158450 0x7f1f380c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f4029c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380c32d0 con 0x7f1f38162210 2023-11-28T18:43:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f4029c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380c2390 con 0x7f1f3815e210 2023-11-28T18:43:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f4029c640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f380c2910 con 0x7f1f38158450 2023-11-28T18:43:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f3e839640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f1f3815e210 0x7f1f3815e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:58702/0 (socket says 172.21.15.7:58702) 2023-11-28T18:43:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f3e839640 1 -- 172.21.15.7:0/2659910327 learned_addr learned my addr 172.21.15.7:0/2659910327 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.828+0000 7f1f3e038640 1 --2- 172.21.15.7:0/2659910327 >> v2:172.21.15.193:3300/0 conn(0x7f1f38158450 0x7f1f380c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.832+0000 7f1f3d036640 1 -- 172.21.15.7:0/2659910327 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005213101 0 0) 0x7f1f380c2390 con 0x7f1f3815e210 2023-11-28T18:43:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.832+0000 7f1f3d036640 1 -- 172.21.15.7:0/2659910327 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f24003530 con 0x7f1f3815e210 2023-11-28T18:43:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.832+0000 7f1f3d036640 1 -- 172.21.15.7:0/2659910327 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125390026 0 0) 0x7f1f380c32d0 con 0x7f1f38162210 2023-11-28T18:43:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.832+0000 7f1f3e038640 1 -- 172.21.15.7:0/2659910327 >> v1:172.21.15.193:6789/0 conn(0x7f1f3815e210 legacy=0x7f1f3815e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.832+0000 7f1f3e038640 1 -- 172.21.15.7:0/2659910327 >> v1:172.21.15.7:6789/0 conn(0x7f1f38162210 legacy=0x7f1f380c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.832+0000 7f1f3e038640 1 -- 172.21.15.7:0/2659910327 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380c2630 con 0x7f1f38158450 2023-11-28T18:43:16.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.844+0000 7f1f3e038640 1 --2- 172.21.15.7:0/2659910327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f38158450 0x7f1f380c2c40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1f340097c0 tx=0x7f1f3401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d14486b48ed57a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:16.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.844+0000 7f1f3d036640 1 -- 172.21.15.7:0/2659910327 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f340ab650 con 0x7f1f38158450 2023-11-28T18:43:16.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.844+0000 7f1f3d036640 1 -- 172.21.15.7:0/2659910327 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f340ab7b0 con 0x7f1f38158450 2023-11-28T18:43:16.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.844+0000 7f1f3d036640 1 -- 172.21.15.7:0/2659910327 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f340abb40 con 0x7f1f38158450 2023-11-28T18:43:16.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.844+0000 7f1f4029c640 1 -- 172.21.15.7:0/2659910327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f38158450 msgr2=0x7f1f380c2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.848+0000 7f1f4029c640 1 --2- 172.21.15.7:0/2659910327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f38158450 0x7f1f380c2c40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1f340097c0 tx=0x7f1f3401c450 comp rx=0 tx=0).stop 2023-11-28T18:43:16.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.848+0000 7f1f4029c640 1 -- 172.21.15.7:0/2659910327 shutdown_connections 2023-11-28T18:43:16.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.848+0000 7f1f4029c640 1 --2- 172.21.15.7:0/2659910327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f38158450 0x7f1f380c2c40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.848+0000 7f1f4029c640 1 -- 172.21.15.7:0/2659910327 >> 172.21.15.7:0/2659910327 conn(0x7f1f380662e0 msgr2=0x7f1f380666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:16.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.848+0000 7f1f4029c640 1 -- 172.21.15.7:0/2659910327 shutdown_connections 2023-11-28T18:43:16.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.848+0000 7f1f4029c640 1 -- 172.21.15.7:0/2659910327 wait complete. 2023-11-28T18:43:16.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.852+0000 7f1f4029c640 1 Processor -- start 2023-11-28T18:43:16.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.852+0000 7f1f4029c640 1 -- start start 2023-11-28T18:43:16.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f4029c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3815e210 0x7f1f381694c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f4029c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f38162210 0x7f1f38203260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f4029c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f382037a0 0x7f1f380716e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f4029c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f38071250 con 0x7f1f3815e210 2023-11-28T18:43:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f4029c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f380cc600 con 0x7f1f382037a0 2023-11-28T18:43:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f4029c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f38203f10 con 0x7f1f38162210 2023-11-28T18:43:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f382037a0 0x7f1f380716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f382037a0 0x7f1f380716e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:37016/0 (socket says 172.21.15.7:37016) 2023-11-28T18:43:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 -- 172.21.15.7:0/2381772525 learned_addr learned my addr 172.21.15.7:0/2381772525 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 -- 172.21.15.7:0/2381772525 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f38162210 msgr2=0x7f1f38203260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f38162210 0x7f1f38203260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 -- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3815e210 msgr2=0x7f1f381694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3815e210 0x7f1f381694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 -- 172.21.15.7:0/2381772525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f34009420 con 0x7f1f382037a0 2023-11-28T18:43:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f3e839640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f382037a0 0x7f1f380716e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000cb40 tx=0x7f1f3000a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f2effd640 1 -- 172.21.15.7:0/2381772525 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3000aca0 con 0x7f1f382037a0 2023-11-28T18:43:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f2effd640 1 -- 172.21.15.7:0/2381772525 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3000ae00 con 0x7f1f382037a0 2023-11-28T18:43:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f2effd640 1 -- 172.21.15.7:0/2381772525 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f30012620 con 0x7f1f382037a0 2023-11-28T18:43:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.856+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f381589b0 con 0x7f1f382037a0 2023-11-28T18:43:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.860+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f38203c30 con 0x7f1f382037a0 2023-11-28T18:43:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.860+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3804e890 con 0x7f1f382037a0 2023-11-28T18:43:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.860+0000 7f1f2effd640 1 -- 172.21.15.7:0/2381772525 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1f30007070 con 0x7f1f382037a0 2023-11-28T18:43:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.864+0000 7f1f2effd640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f1c066920 0x7f1f1c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.864+0000 7f1f2effd640 1 -- 172.21.15.7:0/2381772525 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f1f300847d0 con 0x7f1f382037a0 2023-11-28T18:43:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.864+0000 7f1f3e038640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f1c066920 0x7f1f1c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.864+0000 7f1f3e038640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f1c066920 0x7f1f1c068de0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1f340af000 tx=0x7f1f34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:16.868+0000 7f1f2effd640 1 -- 172.21.15.7:0/2381772525 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1f30051730 con 0x7f1f382037a0 2023-11-28T18:43:17.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.276+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f380c7230 con 0x7f1f382037a0 2023-11-28T18:43:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.280+0000 7f1f2effd640 1 -- 172.21.15.7:0/2381772525 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6560 (secure 0 0 0) 0x7f1f30054d50 con 0x7f1f382037a0 2023-11-28T18:43:17.283 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:17.284 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:14.513274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:17.284 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 91 2023-11-28T18:43:17.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f1c066920 msgr2=0x7f1f1c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:17.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f1c066920 0x7f1f1c068de0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1f340af000 tx=0x7f1f34008040 comp rx=0 tx=0).stop 2023-11-28T18:43:17.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f382037a0 msgr2=0x7f1f380716e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:17.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f382037a0 0x7f1f380716e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000cb40 tx=0x7f1f3000a060 comp rx=0 tx=0).stop 2023-11-28T18:43:17.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 shutdown_connections 2023-11-28T18:43:17.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1f1c066920 0x7f1f1c068de0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:17.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f382037a0 0x7f1f380716e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:17.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1f38162210 0x7f1f38203260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:17.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 --2- 172.21.15.7:0/2381772525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3815e210 0x7f1f381694c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:17.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 >> 172.21.15.7:0/2381772525 conn(0x7f1f380662e0 msgr2=0x7f1f380c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:17.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 shutdown_connections 2023-11-28T18:43:17.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:17.284+0000 7f1f4029c640 1 -- 172.21.15.7:0/2381772525 wait complete. 2023-11-28T18:43:17.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:43:19.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:19.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f617cdf2640 1 Processor -- start 2023-11-28T18:43:19.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f617cdf2640 1 -- start start 2023-11-28T18:43:19.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f617cdf2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ca30 0x7f6178159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:19.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f617cdf2640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f617815ce10 0x7f61780ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:19.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f617cdf2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617815a620 con 0x7f61780cafe0 2023-11-28T18:43:19.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f617cdf2640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617815a900 con 0x7f617815ce10 2023-11-28T18:43:19.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f617cdf2640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617815abe0 con 0x7f617815ca30 2023-11-28T18:43:19.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f61777fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ca30 0x7f6178159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:19.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f6177fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f61780cafe0 0x7f61780cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:58714/0 (socket says 172.21.15.7:58714) 2023-11-28T18:43:19.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f6177fff640 1 -- 172.21.15.7:0/1796149307 learned_addr learned my addr 172.21.15.7:0/1796149307 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:19.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.676+0000 7f616effd640 1 --2- 172.21.15.7:0/1796149307 >> v2:172.21.15.193:3300/0 conn(0x7f617815ce10 0x7f61780ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:19.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.680+0000 7f6176ffd640 1 -- 172.21.15.7:0/1796149307 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165960198 0 0) 0x7f617815a620 con 0x7f61780cafe0 2023-11-28T18:43:19.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.680+0000 7f6176ffd640 1 -- 172.21.15.7:0/1796149307 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6164003530 con 0x7f61780cafe0 2023-11-28T18:43:19.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.680+0000 7f61777fe640 1 -- 172.21.15.7:0/1796149307 >> v2:172.21.15.193:3300/0 conn(0x7f617815ce10 msgr2=0x7f61780ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:19.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.680+0000 7f61777fe640 1 --2- 172.21.15.7:0/1796149307 >> v2:172.21.15.193:3300/0 conn(0x7f617815ce10 0x7f61780ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:19.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.680+0000 7f61777fe640 1 -- 172.21.15.7:0/1796149307 >> v1:172.21.15.193:6789/0 conn(0x7f61780cafe0 legacy=0x7f61780cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:19.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.680+0000 7f61777fe640 1 -- 172.21.15.7:0/1796149307 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61780cbb10 con 0x7f617815ca30 2023-11-28T18:43:19.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.680+0000 7f616effd640 1 --2- 172.21.15.7:0/1796149307 >> v2:172.21.15.193:3300/0 conn(0x7f617815ce10 0x7f61780ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:19.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f61777fe640 1 --2- 172.21.15.7:0/1796149307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ca30 0x7f6178159c90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6168009d90 tx=0x7f616801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e001526f6dadbc3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:19.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f6176ffd640 1 -- 172.21.15.7:0/1796149307 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61680af600 con 0x7f617815ca30 2023-11-28T18:43:19.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f6176ffd640 1 -- 172.21.15.7:0/1796149307 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61680ab050 con 0x7f617815ca30 2023-11-28T18:43:19.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f6176ffd640 1 -- 172.21.15.7:0/1796149307 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61680af980 con 0x7f617815ca30 2023-11-28T18:43:19.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f617cdf2640 1 -- 172.21.15.7:0/1796149307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ca30 msgr2=0x7f6178159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:19.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1796149307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ca30 0x7f6178159c90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6168009d90 tx=0x7f616801f130 comp rx=0 tx=0).stop 2023-11-28T18:43:19.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f617cdf2640 1 -- 172.21.15.7:0/1796149307 shutdown_connections 2023-11-28T18:43:19.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1796149307 >> v2:172.21.15.193:3300/0 conn(0x7f617815ce10 0x7f61780ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:19.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1796149307 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ca30 0x7f6178159c90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:19.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f617cdf2640 1 -- 172.21.15.7:0/1796149307 >> 172.21.15.7:0/1796149307 conn(0x7f617806c060 msgr2=0x7f61780662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:19.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f617cdf2640 1 -- 172.21.15.7:0/1796149307 shutdown_connections 2023-11-28T18:43:19.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.684+0000 7f617cdf2640 1 -- 172.21.15.7:0/1796149307 wait complete. 2023-11-28T18:43:19.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.688+0000 7f617cdf2640 1 Processor -- start 2023-11-28T18:43:19.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.688+0000 7f617cdf2640 1 -- start start 2023-11-28T18:43:19.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f617cdf2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61780cafe0 0x7f61781e2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f617cdf2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f617815ca30 0x7f61781569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f617cdf2640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ce10 0x7f61781513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f617cdf2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6178156530 con 0x7f61780cafe0 2023-11-28T18:43:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f617cdf2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61780c3550 con 0x7f617815ca30 2023-11-28T18:43:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f617cdf2640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6178156200 con 0x7f617815ce10 2023-11-28T18:43:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61780cafe0 0x7f61781e2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61780cafe0 0x7f61781e2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34528/0 (socket says 172.21.15.7:34528) 2023-11-28T18:43:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 -- 172.21.15.7:0/1152948312 learned_addr learned my addr 172.21.15.7:0/1152948312 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f6177fff640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ce10 0x7f61781513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 -- 172.21.15.7:0/1152948312 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ce10 msgr2=0x7f61781513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ce10 0x7f61781513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 -- 172.21.15.7:0/1152948312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f617815ca30 msgr2=0x7f61781569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f617815ca30 0x7f61781569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 -- 172.21.15.7:0/1152948312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6160009420 con 0x7f61780cafe0 2023-11-28T18:43:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f61777fe640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61780cafe0 0x7f61781e2550 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f616801d7e0 tx=0x7f616801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:19.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f6174ff9640 1 -- 172.21.15.7:0/1152948312 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61680afc30 con 0x7f61780cafe0 2023-11-28T18:43:19.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f6174ff9640 1 -- 172.21.15.7:0/1152948312 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61680afd90 con 0x7f61780cafe0 2023-11-28T18:43:19.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f6174ff9640 1 -- 172.21.15.7:0/1152948312 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61680ba640 con 0x7f61780cafe0 2023-11-28T18:43:19.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6168009a70 con 0x7f61780cafe0 2023-11-28T18:43:19.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.692+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61780c3270 con 0x7f61780cafe0 2023-11-28T18:43:19.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.696+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6178156060 con 0x7f61780cafe0 2023-11-28T18:43:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.696+0000 7f6174ff9640 1 -- 172.21.15.7:0/1152948312 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f61680ba7a0 con 0x7f61780cafe0 2023-11-28T18:43:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.700+0000 7f6174ff9640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f61580668d0 0x7f6158068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.700+0000 7f6174ff9640 1 -- 172.21.15.7:0/1152948312 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f6168123140 con 0x7f61780cafe0 2023-11-28T18:43:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.700+0000 7f616effd640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f61580668d0 0x7f6158068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:19.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.700+0000 7f616effd640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f61580668d0 0x7f6158068d90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6160006e90 tx=0x7f6160006de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:19.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:19.704+0000 7f6174ff9640 1 -- 172.21.15.7:0/1152948312 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6178156060 con 0x7f61780cafe0 2023-11-28T18:43:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.188+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6178156370 con 0x7f61780cafe0 2023-11-28T18:43:20.198 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:20.199 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:14.513274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 91 2023-11-28T18:43:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.192+0000 7f6174ff9640 1 -- 172.21.15.7:0/1152948312 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6560 (secure 0 0 0) 0x7f6178156370 con 0x7f61780cafe0 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f61580668d0 msgr2=0x7f6158068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f61580668d0 0x7f6158068d90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6160006e90 tx=0x7f6160006de0 comp rx=0 tx=0).stop 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61780cafe0 msgr2=0x7f61781e2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61780cafe0 0x7f61781e2550 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f616801d7e0 tx=0x7f616801bed0 comp rx=0 tx=0).stop 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 shutdown_connections 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f61580668d0 0x7f6158068d90 secure :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6160006e90 tx=0x7f6160006de0 comp rx=0 tx=0).stop 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f617815ce10 0x7f61781513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f617815ca30 0x7f61781569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 --2- 172.21.15.7:0/1152948312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61780cafe0 0x7f61781e2550 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.196+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 >> 172.21.15.7:0/1152948312 conn(0x7f617806c060 msgr2=0x7f61780cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:20.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.204+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 shutdown_connections 2023-11-28T18:43:20.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:20.204+0000 7f617cdf2640 1 -- 172.21.15.7:0/1152948312 wait complete. 2023-11-28T18:43:20.251 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:43:22.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:22.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.484+0000 7fb9e32c9640 1 Processor -- start 2023-11-28T18:43:22.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.484+0000 7fb9e32c9640 1 -- start start 2023-11-28T18:43:22.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.484+0000 7fb9e32c9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb9dc1542e0 0x7fb9dc15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:22.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.484+0000 7fb9e32c9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc0cbb70 0x7fb9dc0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:22.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.484+0000 7fb9e32c9640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fb9dc0cc810 0x7fb9dc072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:22.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.484+0000 7fb9e32c9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9dc071d50 con 0x7fb9dc1542e0 2023-11-28T18:43:22.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.484+0000 7fb9e32c9640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9dc072030 con 0x7fb9dc0cc810 2023-11-28T18:43:22.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.484+0000 7fb9e32c9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0ca920 con 0x7fb9dc0cbb70 2023-11-28T18:43:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e0864640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc0cbb70 0x7fb9dc0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e0864640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc0cbb70 0x7fb9dc0cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:55768/0 (socket says 172.21.15.7:55768) 2023-11-28T18:43:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e1866640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fb9dc0cc810 0x7fb9dc072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e0864640 1 -- 172.21.15.7:0/3763571576 learned_addr learned my addr 172.21.15.7:0/3763571576 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e1065640 1 --2- 172.21.15.7:0/3763571576 >> v2:172.21.15.7:3300/0 conn(0x7fb9dc1542e0 0x7fb9dc15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e1866640 1 -- 172.21.15.7:0/3763571576 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc0cbb70 msgr2=0x7fb9dc0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e1866640 1 --2- 172.21.15.7:0/3763571576 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc0cbb70 0x7fb9dc0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e1866640 1 -- 172.21.15.7:0/3763571576 >> v2:172.21.15.7:3300/0 conn(0x7fb9dc1542e0 msgr2=0x7fb9dc15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e1866640 1 --2- 172.21.15.7:0/3763571576 >> v2:172.21.15.7:3300/0 conn(0x7fb9dc1542e0 0x7fb9dc15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.488+0000 7fb9e1866640 1 -- 172.21.15.7:0/3763571576 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc15e550 con 0x7fb9dc0cc810 2023-11-28T18:43:22.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.492+0000 7fb9e1866640 1 --2- 172.21.15.7:0/3763571576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc072450 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc066fc0 tx=0x7fb9d401c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=358acfb4826a21e5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:22.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.492+0000 7fb9d3fff640 1 -- 172.21.15.7:0/3763571576 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d40a58a0 con 0x7fb9dc0cc810 2023-11-28T18:43:22.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.492+0000 7fb9d3fff640 1 -- 172.21.15.7:0/3763571576 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d40a5a00 con 0x7fb9dc0cc810 2023-11-28T18:43:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.492+0000 7fb9d3fff640 1 -- 172.21.15.7:0/3763571576 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d40a5d90 con 0x7fb9dc0cc810 2023-11-28T18:43:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.492+0000 7fb9e32c9640 1 -- 172.21.15.7:0/3763571576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 msgr2=0x7fb9dc072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.492+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/3763571576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc072450 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc066fc0 tx=0x7fb9d401c930 comp rx=0 tx=0).stop 2023-11-28T18:43:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.496+0000 7fb9e32c9640 1 -- 172.21.15.7:0/3763571576 shutdown_connections 2023-11-28T18:43:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.496+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/3763571576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc072450 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.496+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/3763571576 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc0cbb70 0x7fb9dc0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.496+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/3763571576 >> v2:172.21.15.7:3300/0 conn(0x7fb9dc1542e0 0x7fb9dc15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.496+0000 7fb9e32c9640 1 -- 172.21.15.7:0/3763571576 >> 172.21.15.7:0/3763571576 conn(0x7fb9dc0717e0 msgr2=0x7fb9dc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:22.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.496+0000 7fb9e32c9640 1 -- 172.21.15.7:0/3763571576 shutdown_connections 2023-11-28T18:43:22.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.496+0000 7fb9e32c9640 1 -- 172.21.15.7:0/3763571576 wait complete. 2023-11-28T18:43:22.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e32c9640 1 Processor -- start 2023-11-28T18:43:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e32c9640 1 -- start start 2023-11-28T18:43:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e32c9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc0cbb70 0x7fb9dc1445e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e32c9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e32c9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc1542e0 0x7fb9dc142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e32c9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc15d930 con 0x7fb9dc0cbb70 2023-11-28T18:43:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e32c9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc15e300 con 0x7fb9dc0cc810 2023-11-28T18:43:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e32c9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d0030 con 0x7fb9dc1542e0 2023-11-28T18:43:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e1065640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc0cbb70 0x7fb9dc1445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc142340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:37060/0 (socket says 172.21.15.7:37060) 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 -- 172.21.15.7:0/466895003 learned_addr learned my addr 172.21.15.7:0/466895003 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e1866640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc1542e0 0x7fb9dc142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 -- 172.21.15.7:0/466895003 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc1542e0 msgr2=0x7fb9dc142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc1542e0 0x7fb9dc142880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 -- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc0cbb70 msgr2=0x7fb9dc1445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc0cbb70 0x7fb9dc1445e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 -- 172.21.15.7:0/466895003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d4009a70 con 0x7fb9dc0cc810 2023-11-28T18:43:22.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.500+0000 7fb9e0864640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc142340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00e910 tx=0x7fb9cc00edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:22.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.504+0000 7fb9d1ffb640 1 -- 172.21.15.7:0/466895003 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc009640 con 0x7fb9dc0cc810 2023-11-28T18:43:22.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.504+0000 7fb9d1ffb640 1 -- 172.21.15.7:0/466895003 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9cc00ca00 con 0x7fb9dc0cc810 2023-11-28T18:43:22.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.504+0000 7fb9d1ffb640 1 -- 172.21.15.7:0/466895003 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc009640 con 0x7fb9dc0cc810 2023-11-28T18:43:22.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.504+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc0d05b0 con 0x7fb9dc0cc810 2023-11-28T18:43:22.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.504+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc15dcb0 con 0x7fb9dc0cc810 2023-11-28T18:43:22.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.504+0000 7fb9d1ffb640 1 -- 172.21.15.7:0/466895003 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb9cc00cb70 con 0x7fb9dc0cc810 2023-11-28T18:43:22.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.504+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc04e890 con 0x7fb9dc0cc810 2023-11-28T18:43:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.508+0000 7fb9d1ffb640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb9c0066990 0x7fb9c0068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.508+0000 7fb9d1ffb640 1 -- 172.21.15.7:0/466895003 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fb9cc084680 con 0x7fb9dc0cc810 2023-11-28T18:43:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.508+0000 7fb9e1065640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb9c0066990 0x7fb9c0068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.508+0000 7fb9e1065640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb9c0066990 0x7fb9c0068e50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800b910 tx=0x7fb9d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:22.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.512+0000 7fb9d1ffb640 1 -- 172.21.15.7:0/466895003 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb9cc051480 con 0x7fb9dc0cc810 2023-11-28T18:43:22.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.952+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9dc164770 con 0x7fb9dc0cc810 2023-11-28T18:43:22.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.952+0000 7fb9d1ffb640 1 -- 172.21.15.7:0/466895003 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6560 (secure 0 0 0) 0x7fb9cc054aa0 con 0x7fb9dc0cc810 2023-11-28T18:43:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:22.959 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:14.513274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:22.959 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 91 2023-11-28T18:43:22.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb9c0066990 msgr2=0x7fb9c0068e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:22.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb9c0066990 0x7fb9c0068e50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800b910 tx=0x7fb9d8008040 comp rx=0 tx=0).stop 2023-11-28T18:43:22.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 msgr2=0x7fb9dc142340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:22.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc142340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00e910 tx=0x7fb9cc00edd0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 shutdown_connections 2023-11-28T18:43:22.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb9c0066990 0x7fb9c0068e50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9dc1542e0 0x7fb9dc142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9dc0cc810 0x7fb9dc142340 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00e910 tx=0x7fb9cc00edd0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 --2- 172.21.15.7:0/466895003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc0cbb70 0x7fb9dc1445e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:22.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.956+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 >> 172.21.15.7:0/466895003 conn(0x7fb9dc0717e0 msgr2=0x7fb9dc152830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.960+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 shutdown_connections 2023-11-28T18:43:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:22.960+0000 7fb9e32c9640 1 -- 172.21.15.7:0/466895003 wait complete. 2023-11-28T18:43:23.010 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:43:25.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:25.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.264+0000 7f865dd1d640 1 Processor -- start 2023-11-28T18:43:25.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.264+0000 7f865dd1d640 1 -- start start 2023-11-28T18:43:25.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f865dd1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c24f0 0x7f86581575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:25.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f865dd1d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f86580c7980 0x7f86580c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:25.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f865dd1d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86580c6140 con 0x7f8658150b30 2023-11-28T18:43:25.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f865dd1d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86580c72b0 con 0x7f86580c7980 2023-11-28T18:43:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f865dd1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8658070c00 con 0x7f86580c24f0 2023-11-28T18:43:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c24f0 0x7f86581575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c24f0 0x7f86581575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:59940/0 (socket says 172.21.15.7:59940) 2023-11-28T18:43:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f8656ffd640 1 -- 172.21.15.7:0/4135462803 learned_addr learned my addr 172.21.15.7:0/4135462803 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f8657fff640 1 --2- 172.21.15.7:0/4135462803 >> v2:172.21.15.193:3300/0 conn(0x7f86580c7980 0x7f86580c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:25.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f8656ffd640 1 -- 172.21.15.7:0/4135462803 >> v2:172.21.15.193:3300/0 conn(0x7f86580c7980 msgr2=0x7f86580c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:25.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f8656ffd640 1 --2- 172.21.15.7:0/4135462803 >> v2:172.21.15.193:3300/0 conn(0x7f86580c7980 0x7f86580c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f8656ffd640 1 -- 172.21.15.7:0/4135462803 >> v1:172.21.15.193:6789/0 conn(0x7f8658150b30 legacy=0x7f86580c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:25.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.268+0000 7f8656ffd640 1 -- 172.21.15.7:0/4135462803 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8658157b00 con 0x7f86580c24f0 2023-11-28T18:43:25.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.272+0000 7f8656ffd640 1 --2- 172.21.15.7:0/4135462803 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c24f0 0x7f86581575c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f864c00b920 tx=0x7f864c01f180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bac9707f92ce9024 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:25.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.272+0000 7f86567fc640 1 -- 172.21.15.7:0/4135462803 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0af600 con 0x7f86580c24f0 2023-11-28T18:43:25.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.272+0000 7f86567fc640 1 -- 172.21.15.7:0/4135462803 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c0ab050 con 0x7f86580c24f0 2023-11-28T18:43:25.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.272+0000 7f86567fc640 1 -- 172.21.15.7:0/4135462803 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0af980 con 0x7f86580c24f0 2023-11-28T18:43:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 -- 172.21.15.7:0/4135462803 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c24f0 msgr2=0x7f86581575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 --2- 172.21.15.7:0/4135462803 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c24f0 0x7f86581575c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f864c00b920 tx=0x7f864c01f180 comp rx=0 tx=0).stop 2023-11-28T18:43:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 -- 172.21.15.7:0/4135462803 shutdown_connections 2023-11-28T18:43:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 --2- 172.21.15.7:0/4135462803 >> v2:172.21.15.193:3300/0 conn(0x7f86580c7980 0x7f86580c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 --2- 172.21.15.7:0/4135462803 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c24f0 0x7f86581575c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 -- 172.21.15.7:0/4135462803 >> 172.21.15.7:0/4135462803 conn(0x7f86580662e0 msgr2=0x7f86580666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 -- 172.21.15.7:0/4135462803 shutdown_connections 2023-11-28T18:43:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 -- 172.21.15.7:0/4135462803 wait complete. 2023-11-28T18:43:25.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.276+0000 7f865dd1d640 1 Processor -- start 2023-11-28T18:43:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f865dd1d640 1 -- start start 2023-11-28T18:43:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f865dd1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c7980 0x7f8658144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f865dd1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8658150b30 0x7f86581ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f865dd1d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8658146460 0x7f8658072a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:25.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8658150b30 0x7f86581ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:25.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8658150b30 0x7f86581ebc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60358/0 (socket says 172.21.15.7:60358) 2023-11-28T18:43:25.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8656ffd640 1 -- 172.21.15.7:0/3573346078 learned_addr learned my addr 172.21.15.7:0/3573346078 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658151360 con 0x7f8658150b30 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86581efa50 con 0x7f8658146460 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86581eb5a0 con 0x7f86580c7980 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8657fff640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8658146460 0x7f8658072a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8656ffd640 1 -- 172.21.15.7:0/3573346078 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c7980 msgr2=0x7f8658144ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8656ffd640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c7980 0x7f8658144ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8656ffd640 1 -- 172.21.15.7:0/3573346078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8658146460 msgr2=0x7f8658072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8656ffd640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8658146460 0x7f8658072a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.280+0000 7f8656ffd640 1 -- 172.21.15.7:0/3573346078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c00b580 con 0x7f8658150b30 2023-11-28T18:43:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.284+0000 7f8656ffd640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8658150b30 0x7f86581ebc90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f864c01cdb0 tx=0x7f864c01bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.284+0000 7f863ffff640 1 -- 172.21.15.7:0/3573346078 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0b9410 con 0x7f8658150b30 2023-11-28T18:43:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.284+0000 7f863ffff640 1 -- 172.21.15.7:0/3573346078 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c0b9590 con 0x7f8658150b30 2023-11-28T18:43:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.284+0000 7f863ffff640 1 -- 172.21.15.7:0/3573346078 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0b9940 con 0x7f8658150b30 2023-11-28T18:43:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.284+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8658147200 con 0x7f8658150b30 2023-11-28T18:43:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.284+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86581ef520 con 0x7f8658150b30 2023-11-28T18:43:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.284+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86581eb980 con 0x7f8658150b30 2023-11-28T18:43:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.288+0000 7f863ffff640 1 -- 172.21.15.7:0/3573346078 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f864c0b6070 con 0x7f8658150b30 2023-11-28T18:43:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.288+0000 7f863ffff640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8644066920 0x7f8644068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.288+0000 7f863ffff640 1 -- 172.21.15.7:0/3573346078 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f864c0f44b0 con 0x7f8658150b30 2023-11-28T18:43:25.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.288+0000 7f86577fe640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8644066920 0x7f8644068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:25.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.296+0000 7f86577fe640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8644066920 0x7f8644068de0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8648003c80 tx=0x7f86480079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:25.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.296+0000 7f863ffff640 1 -- 172.21.15.7:0/3573346078 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f864c0f00f0 con 0x7f8658150b30 2023-11-28T18:43:25.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.796+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f865804e890 con 0x7f8658150b30 2023-11-28T18:43:25.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.796+0000 7f863ffff640 1 -- 172.21.15.7:0/3573346078 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6560 (secure 0 0 0) 0x7f864c0b4020 con 0x7f8658150b30 2023-11-28T18:43:25.802 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:25.802 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:14.513274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:25.803 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 91 2023-11-28T18:43:25.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8644066920 msgr2=0x7f8644068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:25.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8644066920 0x7f8644068de0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8648003c80 tx=0x7f86480079c0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8658150b30 msgr2=0x7f86581ebc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:25.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8658150b30 0x7f86581ebc90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f864c01cdb0 tx=0x7f864c01bf20 comp rx=0 tx=0).stop 2023-11-28T18:43:25.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 shutdown_connections 2023-11-28T18:43:25.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8644066920 0x7f8644068de0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8658146460 0x7f8658072a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8658150b30 0x7f86581ebc90 secure :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f864c01cdb0 tx=0x7f864c01bf20 comp rx=0 tx=0).stop 2023-11-28T18:43:25.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 --2- 172.21.15.7:0/3573346078 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86580c7980 0x7f8658144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:25.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 >> 172.21.15.7:0/3573346078 conn(0x7f86580662e0 msgr2=0x7f865814f510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:25.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 shutdown_connections 2023-11-28T18:43:25.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:25.800+0000 7f865dd1d640 1 -- 172.21.15.7:0/3573346078 wait complete. 2023-11-28T18:43:25.854 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:43:27.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:28.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.096+0000 7f6d83c6b640 1 Processor -- start 2023-11-28T18:43:28.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.096+0000 7f6d83c6b640 1 -- start start 2023-11-28T18:43:28.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d83c6b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d7c1542e0 0x7f6d7c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:28.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d83c6b640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6d7c072450 0x7f6d7c072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d83c6b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7c0ca920 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d83c6b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d7c071d50 con 0x7f6d7c1542e0 2023-11-28T18:43:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d83c6b640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d7c072030 con 0x7f6d7c072450 2023-11-28T18:43:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d81206640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6d7c0cbb70 0x7f6d7c0cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45840/0 (socket says 172.21.15.7:45840) 2023-11-28T18:43:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d81206640 1 -- 172.21.15.7:0/3420333458 learned_addr learned my addr 172.21.15.7:0/3420333458 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d82208640 1 --2- 172.21.15.7:0/3420333458 >> v2:172.21.15.193:3300/0 conn(0x7f6d7c072450 0x7f6d7c072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d81a07640 1 --2- 172.21.15.7:0/3420333458 >> v2:172.21.15.7:3300/0 conn(0x7f6d7c1542e0 0x7f6d7c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d82208640 1 -- 172.21.15.7:0/3420333458 >> v2:172.21.15.7:3300/0 conn(0x7f6d7c1542e0 msgr2=0x7f6d7c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d82208640 1 --2- 172.21.15.7:0/3420333458 >> v2:172.21.15.7:3300/0 conn(0x7f6d7c1542e0 0x7f6d7c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d82208640 1 -- 172.21.15.7:0/3420333458 >> v1:172.21.15.7:6789/0 conn(0x7f6d7c0cbb70 legacy=0x7f6d7c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:28.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.100+0000 7f6d82208640 1 -- 172.21.15.7:0/3420333458 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c15e550 con 0x7f6d7c072450 2023-11-28T18:43:28.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.104+0000 7f6d82208640 1 --2- 172.21.15.7:0/3420333458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c072450 0x7f6d7c072870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c066f90 tx=0x7f6d7401c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8655767293254c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:28.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.104+0000 7f6d80a05640 1 -- 172.21.15.7:0/3420333458 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d740a58e0 con 0x7f6d7c072450 2023-11-28T18:43:28.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.104+0000 7f6d80a05640 1 -- 172.21.15.7:0/3420333458 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d740a5a40 con 0x7f6d7c072450 2023-11-28T18:43:28.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.104+0000 7f6d80a05640 1 -- 172.21.15.7:0/3420333458 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d740a5dd0 con 0x7f6d7c072450 2023-11-28T18:43:28.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.108+0000 7f6d83c6b640 1 -- 172.21.15.7:0/3420333458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c072450 msgr2=0x7f6d7c072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:28.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.108+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/3420333458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c072450 0x7f6d7c072870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c066f90 tx=0x7f6d7401c910 comp rx=0 tx=0).stop 2023-11-28T18:43:28.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.108+0000 7f6d83c6b640 1 -- 172.21.15.7:0/3420333458 shutdown_connections 2023-11-28T18:43:28.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.108+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/3420333458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c072450 0x7f6d7c072870 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.108+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/3420333458 >> v2:172.21.15.7:3300/0 conn(0x7f6d7c1542e0 0x7f6d7c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.108+0000 7f6d83c6b640 1 -- 172.21.15.7:0/3420333458 >> 172.21.15.7:0/3420333458 conn(0x7f6d7c0717e0 msgr2=0x7f6d7c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:28.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.108+0000 7f6d83c6b640 1 -- 172.21.15.7:0/3420333458 shutdown_connections 2023-11-28T18:43:28.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.108+0000 7f6d83c6b640 1 -- 172.21.15.7:0/3420333458 wait complete. 2023-11-28T18:43:28.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 Processor -- start 2023-11-28T18:43:28.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 -- start start 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6d7c072450 0x7f6d7c0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c0cbb70 0x7f6d7c157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c1542e0 0x7f6d7c0c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1f5fd0 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c15b230 con 0x7f6d7c1542e0 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c0c80b0 con 0x7f6d7c072450 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c0cbb70 0x7f6d7c157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c0cbb70 0x7f6d7c157170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60374/0 (socket says 172.21.15.7:60374) 2023-11-28T18:43:28.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 -- 172.21.15.7:0/2603743009 learned_addr learned my addr 172.21.15.7:0/2603743009 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:28.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81a07640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6d7c072450 0x7f6d7c0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:28.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d82208640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c1542e0 0x7f6d7c0c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:28.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 -- 172.21.15.7:0/2603743009 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6d7c072450 msgr2=0x7f6d7c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6d7c072450 0x7f6d7c0c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 -- 172.21.15.7:0/2603743009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c1542e0 msgr2=0x7f6d7c0c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c1542e0 0x7f6d7c0c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 -- 172.21.15.7:0/2603743009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d74009a70 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d81206640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c0cbb70 0x7f6d7c157170 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00c450 tx=0x7f6d6c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d727fc640 1 -- 172.21.15.7:0/2603743009 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c005cf0 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d727fc640 1 -- 172.21.15.7:0/2603743009 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c005e50 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d727fc640 1 -- 172.21.15.7:0/2603743009 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c007290 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.112+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7c1f6220 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.116+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7c15a750 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.116+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7c06c670 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.116+0000 7f6d727fc640 1 -- 172.21.15.7:0/2603743009 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6d6c0073f0 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.116+0000 7f6d727fc640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6d600668d0 0x7f6d60068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.116+0000 7f6d727fc640 1 -- 172.21.15.7:0/2603743009 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f6d6c083cf0 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.120+0000 7f6d81a07640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6d600668d0 0x7f6d60068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.120+0000 7f6d81a07640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6d600668d0 0x7f6d60068d90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009ac0 tx=0x7f6d78009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:28.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.128+0000 7f6d727fc640 1 -- 172.21.15.7:0/2603743009 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6d6c050b70 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.584+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d7c15a1f0 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.584+0000 7f6d727fc640 1 -- 172.21.15.7:0/2603743009 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6560 (secure 0 0 0) 0x7f6d6c017070 con 0x7f6d7c0cbb70 2023-11-28T18:43:28.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:28.589 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:14.513274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:28.589 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 91 2023-11-28T18:43:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6d600668d0 msgr2=0x7f6d60068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6d600668d0 0x7f6d60068d90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009ac0 tx=0x7f6d78009190 comp rx=0 tx=0).stop 2023-11-28T18:43:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c0cbb70 msgr2=0x7f6d7c157170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c0cbb70 0x7f6d7c157170 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00c450 tx=0x7f6d6c00c910 comp rx=0 tx=0).stop 2023-11-28T18:43:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 shutdown_connections 2023-11-28T18:43:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6d600668d0 0x7f6d60068d90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d7c1542e0 0x7f6d7c0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c0cbb70 0x7f6d7c157170 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 --2- 172.21.15.7:0/2603743009 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6d7c072450 0x7f6d7c0c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.588+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 >> 172.21.15.7:0/2603743009 conn(0x7f6d7c0717e0 msgr2=0x7f6d7c148150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:28.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.592+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 shutdown_connections 2023-11-28T18:43:28.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:28.596+0000 7f6d83c6b640 1 -- 172.21.15.7:0/2603743009 wait complete. 2023-11-28T18:43:28.650 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:43:30.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.884+0000 7f97969c4640 1 Processor -- start 2023-11-28T18:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.884+0000 7f97969c4640 1 -- start start 2023-11-28T18:43:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f97969c4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97901540e0 0x7f97900ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f97969c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979015e380 con 0x7f979015a8a0 2023-11-28T18:43:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f97969c4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979015d7e0 con 0x7f97900cf6c0 2023-11-28T18:43:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f97969c4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97900cab80 con 0x7f97901540e0 2023-11-28T18:43:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f97877fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f979015a8a0 0x7f979015dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45846/0 (socket says 172.21.15.7:45846) 2023-11-28T18:43:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f97877fe640 1 -- 172.21.15.7:0/1935164884 learned_addr learned my addr 172.21.15.7:0/1935164884 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f978ffff640 1 --2- 172.21.15.7:0/1935164884 >> v2:172.21.15.7:3300/0 conn(0x7f97901540e0 0x7f97900ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:30.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f978ffff640 1 -- 172.21.15.7:0/1935164884 >> v1:172.21.15.193:6789/0 conn(0x7f97900cf6c0 legacy=0x7f9790164760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:30.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f978ffff640 1 -- 172.21.15.7:0/1935164884 >> v1:172.21.15.7:6789/0 conn(0x7f979015a8a0 legacy=0x7f979015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:30.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.888+0000 7f978ffff640 1 -- 172.21.15.7:0/1935164884 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900cb850 con 0x7f97901540e0 2023-11-28T18:43:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f978ffff640 1 --2- 172.21.15.7:0/1935164884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 0x7f97900ca4d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9780009d90 tx=0x7f978001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bde194e1528958f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f978f7fe640 1 -- 172.21.15.7:0/1935164884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97800a5900 con 0x7f97901540e0 2023-11-28T18:43:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f978f7fe640 1 -- 172.21.15.7:0/1935164884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97800a5a60 con 0x7f97901540e0 2023-11-28T18:43:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f978f7fe640 1 -- 172.21.15.7:0/1935164884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97800a5df0 con 0x7f97901540e0 2023-11-28T18:43:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f97969c4640 1 -- 172.21.15.7:0/1935164884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 msgr2=0x7f97900ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f97969c4640 1 --2- 172.21.15.7:0/1935164884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 0x7f97900ca4d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9780009d90 tx=0x7f978001c950 comp rx=0 tx=0).stop 2023-11-28T18:43:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f97969c4640 1 -- 172.21.15.7:0/1935164884 shutdown_connections 2023-11-28T18:43:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f97969c4640 1 --2- 172.21.15.7:0/1935164884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 0x7f97900ca4d0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f97969c4640 1 -- 172.21.15.7:0/1935164884 >> 172.21.15.7:0/1935164884 conn(0x7f97900662e0 msgr2=0x7f97900666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f97969c4640 1 -- 172.21.15.7:0/1935164884 shutdown_connections 2023-11-28T18:43:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.892+0000 7f97969c4640 1 -- 172.21.15.7:0/1935164884 wait complete. 2023-11-28T18:43:30.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97969c4640 1 Processor -- start 2023-11-28T18:43:30.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97969c4640 1 -- start start 2023-11-28T18:43:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97969c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97900cf6c0 0x7f97900c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97969c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 0x7f97900c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97969c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f979015a8a0 0x7f9790155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97969c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9790156030 con 0x7f97901540e0 2023-11-28T18:43:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97969c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97900c9050 con 0x7f979015a8a0 2023-11-28T18:43:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97969c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97900c77b0 con 0x7f97900cf6c0 2023-11-28T18:43:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f979015a8a0 0x7f9790155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f978ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97900cf6c0 0x7f97900c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f979015a8a0 0x7f9790155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:40944/0 (socket says 172.21.15.7:40944) 2023-11-28T18:43:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 -- 172.21.15.7:0/304303093 learned_addr learned my addr 172.21.15.7:0/304303093 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f97877fe640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 0x7f97900c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f978ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97900cf6c0 0x7f97900c94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:59956/0 (socket says 172.21.15.7:59956) 2023-11-28T18:43:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 -- 172.21.15.7:0/304303093 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97900cf6c0 msgr2=0x7f97900c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97900cf6c0 0x7f97900c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 -- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 msgr2=0x7f97900c68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 0x7f97900c68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 -- 172.21.15.7:0/304303093 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9780009a70 con 0x7f979015a8a0 2023-11-28T18:43:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f9794f61640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f979015a8a0 0x7f9790155af0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f978800c460 tx=0x7f978800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.896+0000 7f978d7fa640 1 -- 172.21.15.7:0/304303093 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788005220 con 0x7f979015a8a0 2023-11-28T18:43:30.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.900+0000 7f978d7fa640 1 -- 172.21.15.7:0/304303093 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9788005380 con 0x7f979015a8a0 2023-11-28T18:43:30.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.900+0000 7f978d7fa640 1 -- 172.21.15.7:0/304303093 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788007270 con 0x7f979015a8a0 2023-11-28T18:43:30.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.900+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97900c7a70 con 0x7f979015a8a0 2023-11-28T18:43:30.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.900+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9790159bb0 con 0x7f979015a8a0 2023-11-28T18:43:30.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.900+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979004e890 con 0x7f979015a8a0 2023-11-28T18:43:30.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.900+0000 7f978d7fa640 1 -- 172.21.15.7:0/304303093 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f97880073d0 con 0x7f979015a8a0 2023-11-28T18:43:30.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.900+0000 7f978d7fa640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9770066920 0x7f9770068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:30.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.900+0000 7f978d7fa640 1 -- 172.21.15.7:0/304303093 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f9788083ef0 con 0x7f979015a8a0 2023-11-28T18:43:30.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.904+0000 7f978ffff640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9770066920 0x7f9770068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:30.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.904+0000 7f978ffff640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9770066920 0x7f9770068de0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f97800af000 tx=0x7f97800a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:30.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:30.904+0000 7f978d7fa640 1 -- 172.21.15.7:0/304303093 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9788050e80 con 0x7f979015a8a0 2023-11-28T18:43:31.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.196+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97900c9190 con 0x7f979015a8a0 2023-11-28T18:43:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.196+0000 7f978d7fa640 1 -- 172.21.15.7:0/304303093 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6556 (secure 0 0 0) 0x7f97880544a0 con 0x7f979015a8a0 2023-11-28T18:43:31.201 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:31.201 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:28.485218+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":5450,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.193:6835/2303207996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2303207996},{"type":"v1","addr":"172.21.15.193:6835","nonce":2303207996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:31.201 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 92 2023-11-28T18:43:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9770066920 msgr2=0x7f9770068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9770066920 0x7f9770068de0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f97800af000 tx=0x7f97800a7070 comp rx=0 tx=0).stop 2023-11-28T18:43:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f979015a8a0 msgr2=0x7f9790155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f979015a8a0 0x7f9790155af0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f978800c460 tx=0x7f978800c920 comp rx=0 tx=0).stop 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 shutdown_connections 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9770066920 0x7f9770068de0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f979015a8a0 0x7f9790155af0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97901540e0 0x7f97900c68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 --2- 172.21.15.7:0/304303093 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97900cf6c0 0x7f97900c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 >> 172.21.15.7:0/304303093 conn(0x7f97900662e0 msgr2=0x7f97900ce460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 shutdown_connections 2023-11-28T18:43:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:31.200+0000 7f97969c4640 1 -- 172.21.15.7:0/304303093 wait complete. 2023-11-28T18:43:31.243 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-11-28T18:43:33.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:33.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22aa32a640 1 Processor -- start 2023-11-28T18:43:33.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22aa32a640 1 -- start start 2023-11-28T18:43:33.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22aa32a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22a40cbb70 0x7f22a40cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22aa32a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a40cc810 0x7f22a4072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22aa32a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a40ca920 con 0x7f22a41542e0 2023-11-28T18:43:33.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22aa32a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a4071d50 con 0x7f22a40cbb70 2023-11-28T18:43:33.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22aa32a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4072030 con 0x7f22a40cc810 2023-11-28T18:43:33.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22a37fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22a40cbb70 0x7f22a40cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22a37fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22a40cbb70 0x7f22a40cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60402/0 (socket says 172.21.15.7:60402) 2023-11-28T18:43:33.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22a37fe640 1 -- 172.21.15.7:0/183286286 learned_addr learned my addr 172.21.15.7:0/183286286 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:33.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22a3fff640 1 --1- 172.21.15.7:0/183286286 >> v1:172.21.15.193:6789/0 conn(0x7f22a41542e0 0x7f22a415e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:37842/0 (socket says 172.21.15.7:37842) 2023-11-28T18:43:33.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.400+0000 7f22a88c7640 1 --2- 172.21.15.7:0/183286286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a40cc810 0x7f22a4072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a2ffd640 1 -- 172.21.15.7:0/183286286 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017099004 0 0) 0x7f22a40ca920 con 0x7f22a41542e0 2023-11-28T18:43:33.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a2ffd640 1 -- 172.21.15.7:0/183286286 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c003530 con 0x7f22a41542e0 2023-11-28T18:43:33.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a88c7640 1 -- 172.21.15.7:0/183286286 >> v2:172.21.15.7:3300/0 conn(0x7f22a40cbb70 msgr2=0x7f22a40cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:33.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a88c7640 1 --2- 172.21.15.7:0/183286286 >> v2:172.21.15.7:3300/0 conn(0x7f22a40cbb70 0x7f22a40cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a88c7640 1 -- 172.21.15.7:0/183286286 >> v1:172.21.15.193:6789/0 conn(0x7f22a41542e0 legacy=0x7f22a415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a88c7640 1 -- 172.21.15.7:0/183286286 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40c4cd0 con 0x7f22a40cc810 2023-11-28T18:43:33.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a37fe640 1 --2- 172.21.15.7:0/183286286 >> v2:172.21.15.7:3300/0 conn(0x7f22a40cbb70 0x7f22a40cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:33.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a88c7640 1 --2- 172.21.15.7:0/183286286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a40cc810 0x7f22a4072450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f229c009d90 tx=0x7f229c01c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b677b6cc0128036 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:33.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a2ffd640 1 -- 172.21.15.7:0/183286286 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c0a5920 con 0x7f22a40cc810 2023-11-28T18:43:33.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a2ffd640 1 -- 172.21.15.7:0/183286286 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229c0a5a80 con 0x7f22a40cc810 2023-11-28T18:43:33.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.404+0000 7f22a2ffd640 1 -- 172.21.15.7:0/183286286 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c0a5e10 con 0x7f22a40cc810 2023-11-28T18:43:33.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- 172.21.15.7:0/183286286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a40cc810 msgr2=0x7f22a4072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 --2- 172.21.15.7:0/183286286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a40cc810 0x7f22a4072450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f229c009d90 tx=0x7f229c01c950 comp rx=0 tx=0).stop 2023-11-28T18:43:33.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- 172.21.15.7:0/183286286 shutdown_connections 2023-11-28T18:43:33.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 --2- 172.21.15.7:0/183286286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a40cc810 0x7f22a4072450 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 --2- 172.21.15.7:0/183286286 >> v2:172.21.15.7:3300/0 conn(0x7f22a40cbb70 0x7f22a40cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- 172.21.15.7:0/183286286 >> 172.21.15.7:0/183286286 conn(0x7f22a40717e0 msgr2=0x7f22a40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- 172.21.15.7:0/183286286 shutdown_connections 2023-11-28T18:43:33.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- 172.21.15.7:0/183286286 wait complete. 2023-11-28T18:43:33.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 Processor -- start 2023-11-28T18:43:33.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- start start 2023-11-28T18:43:33.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a40cbb70 0x7f22a41f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22a40cc810 0x7f22a40c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a41542e0 0x7f22a415a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40c6ba0 con 0x7f22a40cc810 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40c7160 con 0x7f22a40cbb70 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22aa32a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a41f60f0 con 0x7f22a41542e0 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22a37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22a40cc810 0x7f22a40c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a40cbb70 0x7f22a41f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22a88c7640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a41542e0 0x7f22a415a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a40cbb70 0x7f22a41f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:40956/0 (socket says 172.21.15.7:40956) 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22a88c7640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a41542e0 0x7f22a415a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:59972/0 (socket says 172.21.15.7:59972) 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.408+0000 7f22a3fff640 1 -- 172.21.15.7:0/3073812931 learned_addr learned my addr 172.21.15.7:0/3073812931 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a37fe640 1 -- 172.21.15.7:0/3073812931 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a41542e0 msgr2=0x7f22a415a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a37fe640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a41542e0 0x7f22a415a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a37fe640 1 -- 172.21.15.7:0/3073812931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a40cbb70 msgr2=0x7f22a41f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a37fe640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a40cbb70 0x7f22a41f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a37fe640 1 -- 172.21.15.7:0/3073812931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22980094d0 con 0x7f22a40cc810 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a88c7640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a41542e0 0x7f22a415a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a3fff640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a40cbb70 0x7f22a41f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a37fe640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22a40cc810 0x7f22a40c75f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f229800e940 tx=0x7f229800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a0ff9640 1 -- 172.21.15.7:0/3073812931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229800ca00 con 0x7f22a40cc810 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a0ff9640 1 -- 172.21.15.7:0/3073812931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229800cb60 con 0x7f22a40cc810 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a0ff9640 1 -- 172.21.15.7:0/3073812931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298004030 con 0x7f22a40cc810 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229c009a70 con 0x7f22a40cc810 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a41f6690 con 0x7f22a40cc810 2023-11-28T18:43:33.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a406c670 con 0x7f22a40cc810 2023-11-28T18:43:33.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.412+0000 7f22a0ff9640 1 -- 172.21.15.7:0/3073812931 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2298004190 con 0x7f22a40cc810 2023-11-28T18:43:33.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.416+0000 7f22a0ff9640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2290066900 0x7f2290068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.416+0000 7f22a0ff9640 1 -- 172.21.15.7:0/3073812931 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f22980847e0 con 0x7f22a40cc810 2023-11-28T18:43:33.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.416+0000 7f22a3fff640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2290066900 0x7f2290068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.416+0000 7f22a3fff640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2290066900 0x7f2290068dc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f22940011f0 tx=0x7f2294007e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:33.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.420+0000 7f22a0ff9640 1 -- 172.21.15.7:0/3073812931 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2298051690 con 0x7f22a40cc810 2023-11-28T18:43:33.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.620+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22a40c72f0 con 0x7f22a40cc810 2023-11-28T18:43:33.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.620+0000 7f22a0ff9640 1 -- 172.21.15.7:0/3073812931 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5762 (secure 0 0 0) 0x7f2298054cb0 con 0x7f22a40cc810 2023-11-28T18:43:33.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:33.624 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:32.460764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:33.624 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 93 2023-11-28T18:43:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2290066900 msgr2=0x7f2290068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2290066900 0x7f2290068dc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f22940011f0 tx=0x7f2294007e00 comp rx=0 tx=0).stop 2023-11-28T18:43:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22a40cc810 msgr2=0x7f22a40c75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22a40cc810 0x7f22a40c75f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f229800e940 tx=0x7f229800ee00 comp rx=0 tx=0).stop 2023-11-28T18:43:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 shutdown_connections 2023-11-28T18:43:33.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2290066900 0x7f2290068dc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22a41542e0 0x7f22a415a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22a40cc810 0x7f22a40c75f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 --2- 172.21.15.7:0/3073812931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a40cbb70 0x7f22a41f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 >> 172.21.15.7:0/3073812931 conn(0x7f22a40717e0 msgr2=0x7f22a4151b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:33.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 shutdown_connections 2023-11-28T18:43:33.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.624+0000 7f22aa32a640 1 -- 172.21.15.7:0/3073812931 wait complete. 2023-11-28T18:43:33.646 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-11-28T18:43:33.647 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:43:33.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.788+0000 7f0161772640 1 Processor -- start 2023-11-28T18:43:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.788+0000 7f0161772640 1 -- start start 2023-11-28T18:43:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f0161772640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c15d350 0x7f015c06a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f0161772640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f015c06ab30 0x7f015c0c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f0161772640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c0c8750 con 0x7f015c1540a0 2023-11-28T18:43:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f0161772640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c0c8a30 con 0x7f015c06ab30 2023-11-28T18:43:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f0161772640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0731c0 con 0x7f015c15d350 2023-11-28T18:43:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f015c06ab30 0x7f015c0c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f015c06ab30 0x7f015c0c8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58604/0 (socket says 172.21.15.7:58604) 2023-11-28T18:43:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015b7fe640 1 -- 172.21.15.7:0/2019835105 learned_addr learned my addr 172.21.15.7:0/2019835105 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f0153fff640 1 --2- 172.21.15.7:0/2019835105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c15d350 0x7f015c06a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015b7fe640 1 -- 172.21.15.7:0/2019835105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c15d350 msgr2=0x7f015c06a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015b7fe640 1 --2- 172.21.15.7:0/2019835105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c15d350 0x7f015c06a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015b7fe640 1 -- 172.21.15.7:0/2019835105 >> v1:172.21.15.193:6789/0 conn(0x7f015c1540a0 legacy=0x7f015c072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015b7fe640 1 -- 172.21.15.7:0/2019835105 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c15d840 con 0x7f015c06ab30 2023-11-28T18:43:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015b7fe640 1 --2- 172.21.15.7:0/2019835105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c06ab30 0x7f015c0c8ec0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0154009e80 tx=0x7f015401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d292ed9e0e26ec3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015a7fc640 1 -- 172.21.15.7:0/2019835105 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540a56c0 con 0x7f015c06ab30 2023-11-28T18:43:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015a7fc640 1 -- 172.21.15.7:0/2019835105 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01540a5820 con 0x7f015c06ab30 2023-11-28T18:43:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.792+0000 7f015a7fc640 1 -- 172.21.15.7:0/2019835105 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540a5bb0 con 0x7f015c06ab30 2023-11-28T18:43:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- 172.21.15.7:0/2019835105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c06ab30 msgr2=0x7f015c0c8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 --2- 172.21.15.7:0/2019835105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c06ab30 0x7f015c0c8ec0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0154009e80 tx=0x7f015401c9f0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- 172.21.15.7:0/2019835105 shutdown_connections 2023-11-28T18:43:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 --2- 172.21.15.7:0/2019835105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c06ab30 0x7f015c0c8ec0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 --2- 172.21.15.7:0/2019835105 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c15d350 0x7f015c06a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- 172.21.15.7:0/2019835105 >> 172.21.15.7:0/2019835105 conn(0x7f015c073cd0 msgr2=0x7f015c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- 172.21.15.7:0/2019835105 shutdown_connections 2023-11-28T18:43:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- 172.21.15.7:0/2019835105 wait complete. 2023-11-28T18:43:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 Processor -- start 2023-11-28T18:43:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- start start 2023-11-28T18:43:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c06ab30 0x7f015c154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015c1540a0 0x7f015c151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c15d350 0x7f015c152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c14e910 con 0x7f015c15d350 2023-11-28T18:43:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c14f7b0 con 0x7f015c1540a0 2023-11-28T18:43:33.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0161772640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1519b0 con 0x7f015c06ab30 2023-11-28T18:43:33.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c15d350 0x7f015c152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c15d350 0x7f015c152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58612/0 (socket says 172.21.15.7:58612) 2023-11-28T18:43:33.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015b7fe640 1 -- 172.21.15.7:0/3781142828 learned_addr learned my addr 172.21.15.7:0/3781142828 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:33.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015affd640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c06ab30 0x7f015c154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f0153fff640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015c1540a0 0x7f015c151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015b7fe640 1 -- 172.21.15.7:0/3781142828 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c06ab30 msgr2=0x7f015c154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015b7fe640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c06ab30 0x7f015c154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015b7fe640 1 -- 172.21.15.7:0/3781142828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015c1540a0 msgr2=0x7f015c151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:33.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015b7fe640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015c1540a0 0x7f015c151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:33.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.796+0000 7f015b7fe640 1 -- 172.21.15.7:0/3781142828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154009b30 con 0x7f015c15d350 2023-11-28T18:43:33.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.800+0000 7f015b7fe640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c15d350 0x7f015c152380 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f015401a620 tx=0x7f0154019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:33.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.800+0000 7f01537fe640 1 -- 172.21.15.7:0/3781142828 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540b8670 con 0x7f015c15d350 2023-11-28T18:43:33.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.800+0000 7f01537fe640 1 -- 172.21.15.7:0/3781142828 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01540b87d0 con 0x7f015c15d350 2023-11-28T18:43:33.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.800+0000 7f01537fe640 1 -- 172.21.15.7:0/3781142828 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540b8b60 con 0x7f015c15d350 2023-11-28T18:43:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.800+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c14e560 con 0x7f015c15d350 2023-11-28T18:43:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.800+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c14dd60 con 0x7f015c15d350 2023-11-28T18:43:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.800+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c151b40 con 0x7f015c15d350 2023-11-28T18:43:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.800+0000 7f01537fe640 1 -- 172.21.15.7:0/3781142828 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f01540b8cc0 con 0x7f015c15d350 2023-11-28T18:43:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.804+0000 7f01537fe640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:33.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.804+0000 7f01537fe640 1 -- 172.21.15.7:0/3781142828 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f0154120070 con 0x7f015c15d350 2023-11-28T18:43:33.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.804+0000 7f015affd640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:33.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.804+0000 7f015affd640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f014c007fd0 tx=0x7f014c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:33.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.804+0000 7f01537fe640 1 -- 172.21.15.7:0/3781142828 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01540ed0c0 con 0x7f015c15d350 2023-11-28T18:43:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.996+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f015c04e890 con 0x7f015c15d350 2023-11-28T18:43:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:33.996+0000 7f01537fe640 1 -- 172.21.15.7:0/3781142828 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5762 (secure 0 0 0) 0x7f01540bc090 con 0x7f015c15d350 2023-11-28T18:43:34.000 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:34.000 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:32.460764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 93 2023-11-28T18:43:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 msgr2=0x7f013c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f014c007fd0 tx=0x7f014c009040 comp rx=0 tx=0).stop 2023-11-28T18:43:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c15d350 msgr2=0x7f015c152380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c15d350 0x7f015c152380 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f015401a620 tx=0x7f0154019790 comp rx=0 tx=0).stop 2023-11-28T18:43:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 shutdown_connections 2023-11-28T18:43:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015c15d350 0x7f015c152380 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015c1540a0 0x7f015c151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 --2- 172.21.15.7:0/3781142828 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f015c06ab30 0x7f015c154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 >> 172.21.15.7:0/3781142828 conn(0x7f015c073cd0 msgr2=0x7f015c0c9970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 shutdown_connections 2023-11-28T18:43:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:34.000+0000 7f0161772640 1 -- 172.21.15.7:0/3781142828 wait complete. 2023-11-28T18:43:34.023 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5528": { "addr": "172.21.15.7:6835/695953635", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 695953635, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 695953635, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5528, "incarnation": 84, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 30 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:43:32.460764+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5528, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:43:36.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:36.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905b438640 1 Processor -- start 2023-11-28T18:43:36.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905b438640 1 -- start start 2023-11-28T18:43:36.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905b438640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ca30 0x7f9054159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:36.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905b438640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f905415ce10 0x7f90540ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:36.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905b438640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905415a620 con 0x7f90540cafe0 2023-11-28T18:43:36.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905b438640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905415a900 con 0x7f905415ce10 2023-11-28T18:43:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905b438640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905415abe0 con 0x7f905415ca30 2023-11-28T18:43:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f9059c35640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f905415ce10 0x7f90540ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f9059c35640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f905415ce10 0x7f90540ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58614/0 (socket says 172.21.15.7:58614) 2023-11-28T18:43:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f9059c35640 1 -- 172.21.15.7:0/3244014727 learned_addr learned my addr 172.21.15.7:0/3244014727 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905ac37640 1 --1- 172.21.15.7:0/3244014727 >> v1:172.21.15.7:6789/0 conn(0x7f90540cafe0 0x7f90540cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41376/0 (socket says 172.21.15.7:41376) 2023-11-28T18:43:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f905a436640 1 --2- 172.21.15.7:0/3244014727 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ca30 0x7f9054159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f9059c35640 1 -- 172.21.15.7:0/3244014727 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ca30 msgr2=0x7f9054159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:36.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f9059c35640 1 --2- 172.21.15.7:0/3244014727 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ca30 0x7f9054159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f9059c35640 1 -- 172.21.15.7:0/3244014727 >> v1:172.21.15.7:6789/0 conn(0x7f90540cafe0 legacy=0x7f90540cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:36.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.188+0000 7f9059c35640 1 -- 172.21.15.7:0/3244014727 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90540cbb10 con 0x7f905415ce10 2023-11-28T18:43:36.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f9059c35640 1 --2- 172.21.15.7:0/3244014727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ce10 0x7f90540ca1d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9044009870 tx=0x7f904401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26347623e3d82ae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:36.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f9059434640 1 -- 172.21.15.7:0/3244014727 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440a8a30 con 0x7f905415ce10 2023-11-28T18:43:36.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f9059434640 1 -- 172.21.15.7:0/3244014727 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90440a8b90 con 0x7f905415ce10 2023-11-28T18:43:36.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f9059434640 1 -- 172.21.15.7:0/3244014727 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440b0440 con 0x7f905415ce10 2023-11-28T18:43:36.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f905b438640 1 -- 172.21.15.7:0/3244014727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ce10 msgr2=0x7f90540ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:36.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f905b438640 1 --2- 172.21.15.7:0/3244014727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ce10 0x7f90540ca1d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9044009870 tx=0x7f904401ed30 comp rx=0 tx=0).stop 2023-11-28T18:43:36.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f905b438640 1 -- 172.21.15.7:0/3244014727 shutdown_connections 2023-11-28T18:43:36.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f905b438640 1 --2- 172.21.15.7:0/3244014727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ce10 0x7f90540ca1d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f905b438640 1 --2- 172.21.15.7:0/3244014727 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ca30 0x7f9054159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f905b438640 1 -- 172.21.15.7:0/3244014727 >> 172.21.15.7:0/3244014727 conn(0x7f905406c060 msgr2=0x7f90540662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:36.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f905b438640 1 -- 172.21.15.7:0/3244014727 shutdown_connections 2023-11-28T18:43:36.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.192+0000 7f905b438640 1 -- 172.21.15.7:0/3244014727 wait complete. 2023-11-28T18:43:36.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905b438640 1 Processor -- start 2023-11-28T18:43:36.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905b438640 1 -- start start 2023-11-28T18:43:36.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905b438640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90540cafe0 0x7f9054154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:36.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905b438640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ca30 0x7f90541553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:36.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905b438640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ce10 0x7f905414e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:36.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905b438640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90541e48e0 con 0x7f905415ca30 2023-11-28T18:43:36.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905b438640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90540c87a0 con 0x7f90540cafe0 2023-11-28T18:43:36.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905b438640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90541e4ad0 con 0x7f905415ce10 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ca30 0x7f90541553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ca30 0x7f90541553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58616/0 (socket says 172.21.15.7:58616) 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 -- 172.21.15.7:0/1002092803 learned_addr learned my addr 172.21.15.7:0/1002092803 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905a436640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90540cafe0 0x7f9054154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f905ac37640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ce10 0x7f905414e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 -- 172.21.15.7:0/1002092803 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ce10 msgr2=0x7f905414e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ce10 0x7f905414e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 -- 172.21.15.7:0/1002092803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90540cafe0 msgr2=0x7f9054154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90540cafe0 0x7f9054154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 -- 172.21.15.7:0/1002092803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440094d0 con 0x7f905415ca30 2023-11-28T18:43:36.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.196+0000 7f9059c35640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ca30 0x7f90541553e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f90440b3040 tx=0x7f904401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f904affd640 1 -- 172.21.15.7:0/1002092803 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440b0dd0 con 0x7f905415ca30 2023-11-28T18:43:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f904affd640 1 -- 172.21.15.7:0/1002092803 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90440bb460 con 0x7f905415ca30 2023-11-28T18:43:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90541e4db0 con 0x7f905415ca30 2023-11-28T18:43:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f904affd640 1 -- 172.21.15.7:0/1002092803 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440bb810 con 0x7f905415ca30 2023-11-28T18:43:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90541e5370 con 0x7f905415ca30 2023-11-28T18:43:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905404e890 con 0x7f905415ca30 2023-11-28T18:43:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f904affd640 1 -- 172.21.15.7:0/1002092803 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f90440b83f0 con 0x7f905415ca30 2023-11-28T18:43:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f904affd640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9038066900 0x7f9038068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f904affd640 1 -- 172.21.15.7:0/1002092803 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f9044121fa0 con 0x7f905415ca30 2023-11-28T18:43:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f905a436640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9038066900 0x7f9038068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.200+0000 7f905a436640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9038066900 0x7f9038068dc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9050009b60 tx=0x7f9050009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.208+0000 7f904affd640 1 -- 172.21.15.7:0/1002092803 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f90440eeef0 con 0x7f905415ca30 2023-11-28T18:43:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90540cad10 con 0x7f905415ca30 2023-11-28T18:43:36.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f904affd640 1 -- 172.21.15.7:0/1002092803 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5762 (secure 0 0 0) 0x7f90440c0090 con 0x7f905415ca30 2023-11-28T18:43:36.414 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:36.415 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:32.460764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 93 2023-11-28T18:43:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9038066900 msgr2=0x7f9038068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9038066900 0x7f9038068dc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9050009b60 tx=0x7f9050009190 comp rx=0 tx=0).stop 2023-11-28T18:43:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ca30 msgr2=0x7f90541553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ca30 0x7f90541553e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f90440b3040 tx=0x7f904401bad0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 shutdown_connections 2023-11-28T18:43:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9038066900 0x7f9038068dc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f905415ce10 0x7f905414e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905415ca30 0x7f90541553e0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 --2- 172.21.15.7:0/1002092803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90540cafe0 0x7f9054154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.412+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 >> 172.21.15.7:0/1002092803 conn(0x7f905406c060 msgr2=0x7f905415b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.416+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 shutdown_connections 2023-11-28T18:43:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:36.416+0000 7f905b438640 1 -- 172.21.15.7:0/1002092803 wait complete. 2023-11-28T18:43:36.438 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:43:38.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:38.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.592+0000 7fabfd572640 1 Processor -- start 2023-11-28T18:43:38.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.592+0000 7fabfd572640 1 -- start start 2023-11-28T18:43:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabfd572640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fabf8072450 0x7fabf8072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabfd572640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80ca920 con 0x7fabf81542e0 2023-11-28T18:43:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabfd572640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80c4cd0 con 0x7fabf80cbb70 2023-11-28T18:43:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabfd572640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf8071d50 con 0x7fabf8072450 2023-11-28T18:43:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf6ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fabf81542e0 0x7fabf815e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41386/0 (socket says 172.21.15.7:41386) 2023-11-28T18:43:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf6ffd640 1 -- 172.21.15.7:0/503392900 learned_addr learned my addr 172.21.15.7:0/503392900 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf77fe640 1 --2- 172.21.15.7:0/503392900 >> v2:172.21.15.193:3300/0 conn(0x7fabf8072450 0x7fabf8072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:38.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf67fc640 1 -- 172.21.15.7:0/503392900 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615075474 0 0) 0x7fabf80ca920 con 0x7fabf81542e0 2023-11-28T18:43:38.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf67fc640 1 -- 172.21.15.7:0/503392900 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe4003530 con 0x7fabf81542e0 2023-11-28T18:43:38.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf77fe640 1 -- 172.21.15.7:0/503392900 >> v1:172.21.15.193:6789/0 conn(0x7fabf80cbb70 legacy=0x7fabf80cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:38.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf67fc640 1 -- 172.21.15.7:0/503392900 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707578941 0 0) 0x7fabf80c4cd0 con 0x7fabf80cbb70 2023-11-28T18:43:38.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf77fe640 1 -- 172.21.15.7:0/503392900 >> v1:172.21.15.7:6789/0 conn(0x7fabf81542e0 legacy=0x7fabf815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:38.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.596+0000 7fabf77fe640 1 -- 172.21.15.7:0/503392900 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80cab70 con 0x7fabf8072450 2023-11-28T18:43:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabf77fe640 1 --2- 172.21.15.7:0/503392900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf8072450 0x7fabf8072870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009d90 tx=0x7fabf001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b825cfc19440dc83 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabf67fc640 1 -- 172.21.15.7:0/503392900 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf00a5920 con 0x7fabf8072450 2023-11-28T18:43:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabf67fc640 1 -- 172.21.15.7:0/503392900 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf00a5a80 con 0x7fabf8072450 2023-11-28T18:43:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabf67fc640 1 -- 172.21.15.7:0/503392900 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf00a5e10 con 0x7fabf8072450 2023-11-28T18:43:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 -- 172.21.15.7:0/503392900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf8072450 msgr2=0x7fabf8072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 --2- 172.21.15.7:0/503392900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf8072450 0x7fabf8072870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009d90 tx=0x7fabf001c950 comp rx=0 tx=0).stop 2023-11-28T18:43:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 -- 172.21.15.7:0/503392900 shutdown_connections 2023-11-28T18:43:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 --2- 172.21.15.7:0/503392900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf8072450 0x7fabf8072870 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 -- 172.21.15.7:0/503392900 >> 172.21.15.7:0/503392900 conn(0x7fabf80717e0 msgr2=0x7fabf80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 -- 172.21.15.7:0/503392900 shutdown_connections 2023-11-28T18:43:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 -- 172.21.15.7:0/503392900 wait complete. 2023-11-28T18:43:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 Processor -- start 2023-11-28T18:43:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 -- start start 2023-11-28T18:43:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf8072450 0x7fabf8149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.600+0000 7fabfd572640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabf80cbb70 0x7fabf814ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabfd572640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf81542e0 0x7fabf814b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabfd572640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8072030 con 0x7fabf8072450 2023-11-28T18:43:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabfd572640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf814c6f0 con 0x7fabf81542e0 2023-11-28T18:43:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabfd572640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf81f2d70 con 0x7fabf80cbb70 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf8072450 0x7fabf8149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf8072450 0x7fabf8149200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58620/0 (socket says 172.21.15.7:58620) 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 -- 172.21.15.7:0/179387037 learned_addr learned my addr 172.21.15.7:0/179387037 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabeffff640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabf80cbb70 0x7fabf814ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf77fe640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf81542e0 0x7fabf814b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 -- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabf80cbb70 msgr2=0x7fabf814ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabf80cbb70 0x7fabf814ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 -- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf81542e0 msgr2=0x7fabf814b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf81542e0 0x7fabf814b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 -- 172.21.15.7:0/179387037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf0009a70 con 0x7fabf8072450 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf77fe640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf81542e0 0x7fabf814b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:43:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabf6ffd640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf8072450 0x7fabf8149200 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fabe800c3c0 tx=0x7fabe800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:38.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabef7fe640 1 -- 172.21.15.7:0/179387037 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe8005cf0 con 0x7fabf8072450 2023-11-28T18:43:38.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabef7fe640 1 -- 172.21.15.7:0/179387037 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe8005e50 con 0x7fabf8072450 2023-11-28T18:43:38.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabef7fe640 1 -- 172.21.15.7:0/179387037 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe80072f0 con 0x7fabf8072450 2023-11-28T18:43:38.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf81f3050 con 0x7fabf8072450 2023-11-28T18:43:38.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf81f3610 con 0x7fabf8072450 2023-11-28T18:43:38.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf806c670 con 0x7fabf8072450 2023-11-28T18:43:38.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.604+0000 7fabef7fe640 1 -- 172.21.15.7:0/179387037 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fabe8007450 con 0x7fabf8072450 2023-11-28T18:43:38.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.608+0000 7fabef7fe640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabd80668d0 0x7fabd8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:38.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.608+0000 7fabef7fe640 1 -- 172.21.15.7:0/179387037 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fabe8083eb0 con 0x7fabf8072450 2023-11-28T18:43:38.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.608+0000 7fabeffff640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabd80668d0 0x7fabd8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:38.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.608+0000 7fabeffff640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabd80668d0 0x7fabd8068d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fabf815ac20 tx=0x7fabe0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:38.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.612+0000 7fabef7fe640 1 -- 172.21.15.7:0/179387037 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fabe8050c40 con 0x7fabf8072450 2023-11-28T18:43:38.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.800+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabf804e890 con 0x7fabf8072450 2023-11-28T18:43:38.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.800+0000 7fabef7fe640 1 -- 172.21.15.7:0/179387037 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5764 (secure 0 0 0) 0x7fabe8017030 con 0x7fabf8072450 2023-11-28T18:43:38.804 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:38.804 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:37.502146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:resolve","state_seq":17,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:38.804 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 94 2023-11-28T18:43:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabd80668d0 msgr2=0x7fabd8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabd80668d0 0x7fabd8068d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fabf815ac20 tx=0x7fabe0009040 comp rx=0 tx=0).stop 2023-11-28T18:43:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf8072450 msgr2=0x7fabf8149200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf8072450 0x7fabf8149200 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fabe800c3c0 tx=0x7fabe800c880 comp rx=0 tx=0).stop 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 shutdown_connections 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fabd80668d0 0x7fabd8068d90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabf81542e0 0x7fabf814b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fabf80cbb70 0x7fabf814ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 --2- 172.21.15.7:0/179387037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf8072450 0x7fabf8149200 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 >> 172.21.15.7:0/179387037 conn(0x7fabf80717e0 msgr2=0x7fabf80ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 shutdown_connections 2023-11-28T18:43:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:38.804+0000 7fabfd572640 1 -- 172.21.15.7:0/179387037 wait complete. 2023-11-28T18:43:38.826 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:43:40.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:43:40.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:43:40.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:40.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.984+0000 7f4792b80640 1 Processor -- start 2023-11-28T18:43:40.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.984+0000 7f4792b80640 1 -- start start 2023-11-28T18:43:40.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.984+0000 7f4792b80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c15a5b0 0x7f478c0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:40.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.984+0000 7f4792b80640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c0c8430 con 0x7f478c154040 2023-11-28T18:43:40.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.984+0000 7f4792b80640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c06ae50 con 0x7f478c06a8a0 2023-11-28T18:43:40.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.984+0000 7f4792b80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c06b110 con 0x7f478c15a5b0 2023-11-28T18:43:40.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.984+0000 7f479091c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f478c154040 0x7f478c159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41390/0 (socket says 172.21.15.7:41390) 2023-11-28T18:43:40.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.984+0000 7f479091c640 1 -- 172.21.15.7:0/2153361999 learned_addr learned my addr 172.21.15.7:0/2153361999 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:40.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478bfff640 1 --2- 172.21.15.7:0/2153361999 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c15a5b0 0x7f478c0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:40.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478b7fe640 1 -- 172.21.15.7:0/2153361999 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906773074 0 0) 0x7f478c0c8430 con 0x7f478c154040 2023-11-28T18:43:40.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478b7fe640 1 -- 172.21.15.7:0/2153361999 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f477c003530 con 0x7f478c154040 2023-11-28T18:43:40.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478b7fe640 1 -- 172.21.15.7:0/2153361999 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618568977 0 0) 0x7f478c06ae50 con 0x7f478c06a8a0 2023-11-28T18:43:40.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478bfff640 1 -- 172.21.15.7:0/2153361999 >> v1:172.21.15.193:6789/0 conn(0x7f478c06a8a0 legacy=0x7f478c0c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:40.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478bfff640 1 -- 172.21.15.7:0/2153361999 >> v1:172.21.15.7:6789/0 conn(0x7f478c154040 legacy=0x7f478c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:40.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478bfff640 1 -- 172.21.15.7:0/2153361999 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c0c7820 con 0x7f478c15a5b0 2023-11-28T18:43:40.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478bfff640 1 --2- 172.21.15.7:0/2153361999 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c15a5b0 0x7f478c0c89b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f47780097f0 tx=0x7f477801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53ae1bcc488f0b0c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:40.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.988+0000 7f478b7fe640 1 -- 172.21.15.7:0/2153361999 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47780a89b0 con 0x7f478c15a5b0 2023-11-28T18:43:40.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f478b7fe640 1 -- 172.21.15.7:0/2153361999 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47780a8b10 con 0x7f478c15a5b0 2023-11-28T18:43:40.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f478b7fe640 1 -- 172.21.15.7:0/2153361999 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47780a8ea0 con 0x7f478c15a5b0 2023-11-28T18:43:40.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 -- 172.21.15.7:0/2153361999 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c15a5b0 msgr2=0x7f478c0c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:40.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 --2- 172.21.15.7:0/2153361999 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c15a5b0 0x7f478c0c89b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f47780097f0 tx=0x7f477801ecb0 comp rx=0 tx=0).stop 2023-11-28T18:43:40.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 -- 172.21.15.7:0/2153361999 shutdown_connections 2023-11-28T18:43:40.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 --2- 172.21.15.7:0/2153361999 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c15a5b0 0x7f478c0c89b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:40.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 -- 172.21.15.7:0/2153361999 >> 172.21.15.7:0/2153361999 conn(0x7f478c06d4d0 msgr2=0x7f478c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:40.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 -- 172.21.15.7:0/2153361999 shutdown_connections 2023-11-28T18:43:40.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 -- 172.21.15.7:0/2153361999 wait complete. 2023-11-28T18:43:40.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 Processor -- start 2023-11-28T18:43:40.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.992+0000 7f4792b80640 1 -- start start 2023-11-28T18:43:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c06a8a0 0x7f478c0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f478c154040 0x7f478c1ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f478c15a5b0 0x7f478c1efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c1f0370 con 0x7f478c15a5b0 2023-11-28T18:43:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c1547d0 con 0x7f478c154040 2023-11-28T18:43:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c15dad0 con 0x7f478c06a8a0 2023-11-28T18:43:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479111d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f478c15a5b0 0x7f478c1efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479091c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c06a8a0 0x7f478c0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f478bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f478c154040 0x7f478c1ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479091c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c06a8a0 0x7f478c0cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:48374/0 (socket says 172.21.15.7:48374) 2023-11-28T18:43:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479091c640 1 -- 172.21.15.7:0/59923233 learned_addr learned my addr 172.21.15.7:0/59923233 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479111d640 1 -- 172.21.15.7:0/59923233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c06a8a0 msgr2=0x7f478c0cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479111d640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c06a8a0 0x7f478c0cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479111d640 1 -- 172.21.15.7:0/59923233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f478c154040 msgr2=0x7f478c1ef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:41.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479111d640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f478c154040 0x7f478c1ef8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:41.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479111d640 1 -- 172.21.15.7:0/59923233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4778009420 con 0x7f478c15a5b0 2023-11-28T18:43:41.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f478bfff640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f478c154040 0x7f478c1ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:41.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479091c640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c06a8a0 0x7f478c0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:41.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f479111d640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f478c15a5b0 0x7f478c1efe30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f478400c700 tx=0x7f478400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f47897fa640 1 -- 172.21.15.7:0/59923233 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4784003db0 con 0x7f478c15a5b0 2023-11-28T18:43:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f47897fa640 1 -- 172.21.15.7:0/59923233 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478400ce10 con 0x7f478c15a5b0 2023-11-28T18:43:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c15dd70 con 0x7f478c15a5b0 2023-11-28T18:43:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f47897fa640 1 -- 172.21.15.7:0/59923233 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47840052b0 con 0x7f478c15a5b0 2023-11-28T18:43:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c15e030 con 0x7f478c15a5b0 2023-11-28T18:43:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c04e890 con 0x7f478c15a5b0 2023-11-28T18:43:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:40.996+0000 7f47897fa640 1 -- 172.21.15.7:0/59923233 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4784005410 con 0x7f478c15a5b0 2023-11-28T18:43:41.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.000+0000 7f47897fa640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47700668d0 0x7f4770068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.000+0000 7f47897fa640 1 -- 172.21.15.7:0/59923233 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f4784084060 con 0x7f478c15a5b0 2023-11-28T18:43:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.000+0000 7f479091c640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47700668d0 0x7f4770068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.000+0000 7f479091c640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47700668d0 0x7f4770068d90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4780007fd0 tx=0x7f4780009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:41.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.004+0000 7f47897fa640 1 -- 172.21.15.7:0/59923233 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4784050e90 con 0x7f478c15a5b0 2023-11-28T18:43:41.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f478c158bc0 con 0x7f478c15a5b0 2023-11-28T18:43:41.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f47897fa640 1 -- 172.21.15.7:0/59923233 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5764 (secure 0 0 0) 0x7f4784017070 con 0x7f478c15a5b0 2023-11-28T18:43:41.234 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:41.234 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:37.502146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:resolve","state_seq":17,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 94 2023-11-28T18:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47700668d0 msgr2=0x7f4770068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47700668d0 0x7f4770068d90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4780007fd0 tx=0x7f4780009040 comp rx=0 tx=0).stop 2023-11-28T18:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f478c15a5b0 msgr2=0x7f478c1efe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f478c15a5b0 0x7f478c1efe30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f478400c700 tx=0x7f478400cbc0 comp rx=0 tx=0).stop 2023-11-28T18:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 shutdown_connections 2023-11-28T18:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f47700668d0 0x7f4770068d90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f478c15a5b0 0x7f478c1efe30 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f478c154040 0x7f478c1ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 --2- 172.21.15.7:0/59923233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f478c06a8a0 0x7f478c0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:41.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 >> 172.21.15.7:0/59923233 conn(0x7f478c06d4d0 msgr2=0x7f478c157f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:41.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 shutdown_connections 2023-11-28T18:43:41.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:41.232+0000 7f4792b80640 1 -- 172.21.15.7:0/59923233 wait complete. 2023-11-28T18:43:41.260 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:43:43.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:43.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0eaea640 1 Processor -- start 2023-11-28T18:43:43.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0eaea640 1 -- start start 2023-11-28T18:43:43.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0eaea640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4c081542e0 0x7f4c0815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:43.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0eaea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c080cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:43.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0eaea640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c080ca920 con 0x7f4c080cc810 2023-11-28T18:43:43.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0eaea640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c08071d50 con 0x7f4c081542e0 2023-11-28T18:43:43.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0eaea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c08072030 con 0x7f4c080cbb70 2023-11-28T18:43:43.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0c886640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4c081542e0 0x7f4c0815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:43.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c07fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c080cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:43.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0d087640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4c080cc810 0x7f4c08072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41402/0 (socket says 172.21.15.7:41402) 2023-11-28T18:43:43.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0d087640 1 -- 172.21.15.7:0/3828202255 learned_addr learned my addr 172.21.15.7:0/3828202255 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:43.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0c886640 1 -- 172.21.15.7:0/3828202255 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 msgr2=0x7f4c080cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:43.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0c886640 1 --2- 172.21.15.7:0/3828202255 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c080cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0c886640 1 -- 172.21.15.7:0/3828202255 >> v1:172.21.15.7:6789/0 conn(0x7f4c080cc810 legacy=0x7f4c08072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:43.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c0c886640 1 -- 172.21.15.7:0/3828202255 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0815e550 con 0x7f4c081542e0 2023-11-28T18:43:43.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c077fe640 1 -- 172.21.15.7:0/3828202255 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802215352 0 0) 0x7f4c080ca920 con 0x7f4c080cc810 2023-11-28T18:43:43.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.420+0000 7f4c07fff640 1 --2- 172.21.15.7:0/3828202255 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c080cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:43.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c0c886640 1 --2- 172.21.15.7:0/3828202255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c081542e0 0x7f4c0815e010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0097c0 tx=0x7f4bfc01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eee296b300a0ae5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:43.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c077fe640 1 -- 172.21.15.7:0/3828202255 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0ad640 con 0x7f4c081542e0 2023-11-28T18:43:43.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c077fe640 1 -- 172.21.15.7:0/3828202255 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc0ad7a0 con 0x7f4c081542e0 2023-11-28T18:43:43.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c077fe640 1 -- 172.21.15.7:0/3828202255 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0adb30 con 0x7f4c081542e0 2023-11-28T18:43:43.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c0eaea640 1 -- 172.21.15.7:0/3828202255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c081542e0 msgr2=0x7f4c0815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:43.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/3828202255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c081542e0 0x7f4c0815e010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0097c0 tx=0x7f4bfc01c480 comp rx=0 tx=0).stop 2023-11-28T18:43:43.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c0eaea640 1 -- 172.21.15.7:0/3828202255 shutdown_connections 2023-11-28T18:43:43.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/3828202255 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c080cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/3828202255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c081542e0 0x7f4c0815e010 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.424+0000 7f4c0eaea640 1 -- 172.21.15.7:0/3828202255 >> 172.21.15.7:0/3828202255 conn(0x7f4c080717e0 msgr2=0x7f4c080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:43.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 -- 172.21.15.7:0/3828202255 shutdown_connections 2023-11-28T18:43:43.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 -- 172.21.15.7:0/3828202255 wait complete. 2023-11-28T18:43:43.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 Processor -- start 2023-11-28T18:43:43.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 -- start start 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c081f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c080cc810 0x7f4c081f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c081542e0 0x7f4c08145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c08158100 con 0x7f4c081542e0 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c081593a0 con 0x7f4c080cc810 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0eaea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c080c8330 con 0x7f4c080cbb70 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c07fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c080cc810 0x7f4c081f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0d087640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c081542e0 0x7f4c08145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:43.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0c886640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c081f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:43.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c07fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c080cc810 0x7f4c081f2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:56828/0 (socket says 172.21.15.7:56828) 2023-11-28T18:43:43.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0c886640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c081f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:48388/0 (socket says 172.21.15.7:48388) 2023-11-28T18:43:43.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c07fff640 1 -- 172.21.15.7:0/632662564 learned_addr learned my addr 172.21.15.7:0/632662564 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c07fff640 1 -- 172.21.15.7:0/632662564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 msgr2=0x7f4c081f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c07fff640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c081f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c07fff640 1 -- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c081542e0 msgr2=0x7f4c08145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c07fff640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c081542e0 0x7f4c08145040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c07fff640 1 -- 172.21.15.7:0/632662564 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf4009420 con 0x7f4c080cc810 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.428+0000 7f4c0c886640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c081f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c07fff640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c080cc810 0x7f4c081f2a70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400e9b0 tx=0x7f4bf400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c057fa640 1 -- 172.21.15.7:0/632662564 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf400ca00 con 0x7f4c080cc810 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c057fa640 1 -- 172.21.15.7:0/632662564 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf400cb60 con 0x7f4c080cc810 2023-11-28T18:43:43.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c057fa640 1 -- 172.21.15.7:0/632662564 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4013470 con 0x7f4c080cc810 2023-11-28T18:43:43.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bfc009420 con 0x7f4c080cc810 2023-11-28T18:43:43.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c080c88d0 con 0x7f4c080cc810 2023-11-28T18:43:43.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0806c670 con 0x7f4c080cc810 2023-11-28T18:43:43.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c057fa640 1 -- 172.21.15.7:0/632662564 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4bf40135d0 con 0x7f4c080cc810 2023-11-28T18:43:43.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.432+0000 7f4c057fa640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bec0668d0 0x7f4bec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:43.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.436+0000 7f4c057fa640 1 -- 172.21.15.7:0/632662564 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f4bf4084af0 con 0x7f4c080cc810 2023-11-28T18:43:43.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.436+0000 7f4c0c886640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bec0668d0 0x7f4bec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:43.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.436+0000 7f4c0c886640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bec0668d0 0x7f4bec068d90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0af000 tx=0x7f4bfc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:43.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.440+0000 7f4c057fa640 1 -- 172.21.15.7:0/632662564 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4bf4051910 con 0x7f4c080cc810 2023-11-28T18:43:43.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.632+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c0804e890 con 0x7f4c080cc810 2023-11-28T18:43:43.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c057fa640 1 -- 172.21.15.7:0/632662564 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5766 (secure 0 0 0) 0x7f4bf4054f30 con 0x7f4c080cc810 2023-11-28T18:43:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:43.638 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:43.471588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:43.638 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 95 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bec0668d0 msgr2=0x7f4bec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bec0668d0 0x7f4bec068d90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0af000 tx=0x7f4bfc008040 comp rx=0 tx=0).stop 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c080cc810 msgr2=0x7f4c081f2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c080cc810 0x7f4c081f2a70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400e9b0 tx=0x7f4bf400ee70 comp rx=0 tx=0).stop 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 shutdown_connections 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4bec0668d0 0x7f4bec068d90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c081542e0 0x7f4c08145040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c080cc810 0x7f4c081f2a70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 --2- 172.21.15.7:0/632662564 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4c080cbb70 0x7f4c081f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 >> 172.21.15.7:0/632662564 conn(0x7f4c080717e0 msgr2=0x7f4c0815c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 shutdown_connections 2023-11-28T18:43:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:43.636+0000 7f4c0eaea640 1 -- 172.21.15.7:0/632662564 wait complete. 2023-11-28T18:43:43.660 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:43:45.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:45.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f3770e6b640 1 Processor -- start 2023-11-28T18:43:45.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f3770e6b640 1 -- start start 2023-11-28T18:43:45.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f3770e6b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f376c150b30 0x7f376c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f3770e6b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376c0c6140 con 0x7f376c0c7980 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f3770e6b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376c0c6370 con 0x7f376c1575c0 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f3770e6b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376c0c72b0 con 0x7f376c150b30 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f376a59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f376c150b30 0x7f376c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f376ad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f376c0c7980 0x7f376c0c7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52266/0 (socket says 172.21.15.7:52266) 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.820+0000 7f376ad9d640 1 -- 172.21.15.7:0/2007924531 learned_addr learned my addr 172.21.15.7:0/2007924531 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.824+0000 7f376a59c640 1 -- 172.21.15.7:0/2007924531 >> v1:172.21.15.193:6789/0 conn(0x7f376c1575c0 legacy=0x7f376c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.824+0000 7f376959a640 1 -- 172.21.15.7:0/2007924531 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948435992 0 0) 0x7f376c0c6370 con 0x7f376c1575c0 2023-11-28T18:43:45.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.824+0000 7f376a59c640 1 -- 172.21.15.7:0/2007924531 >> v1:172.21.15.7:6789/0 conn(0x7f376c0c7980 legacy=0x7f376c0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:45.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.824+0000 7f376a59c640 1 -- 172.21.15.7:0/2007924531 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c0c6630 con 0x7f376c150b30 2023-11-28T18:43:45.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.824+0000 7f376a59c640 1 --2- 172.21.15.7:0/2007924531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c0c24f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f375c00ba60 tx=0x7f375c01ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab028aa93bb34c74 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:45.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.824+0000 7f376959a640 1 -- 172.21.15.7:0/2007924531 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c0a7ae0 con 0x7f376c150b30 2023-11-28T18:43:45.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.824+0000 7f376959a640 1 -- 172.21.15.7:0/2007924531 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c0a7c40 con 0x7f376c150b30 2023-11-28T18:43:45.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.824+0000 7f376959a640 1 -- 172.21.15.7:0/2007924531 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c0af440 con 0x7f376c150b30 2023-11-28T18:43:45.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- 172.21.15.7:0/2007924531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 msgr2=0x7f376c0c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:45.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 --2- 172.21.15.7:0/2007924531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c0c24f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f375c00ba60 tx=0x7f375c01ea70 comp rx=0 tx=0).stop 2023-11-28T18:43:45.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- 172.21.15.7:0/2007924531 shutdown_connections 2023-11-28T18:43:45.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 --2- 172.21.15.7:0/2007924531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c0c24f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:45.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- 172.21.15.7:0/2007924531 >> 172.21.15.7:0/2007924531 conn(0x7f376c0662e0 msgr2=0x7f376c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:45.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- 172.21.15.7:0/2007924531 shutdown_connections 2023-11-28T18:43:45.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- 172.21.15.7:0/2007924531 wait complete. 2023-11-28T18:43:45.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 Processor -- start 2023-11-28T18:43:45.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- start start 2023-11-28T18:43:45.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f376c0c7980 0x7f376c1e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:45.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f376c1575c0 0x7f376c152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c072580 con 0x7f376c150b30 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c072000 con 0x7f376c1575c0 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3770e6b640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c0722a0 con 0x7f376c0c7980 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3769d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3769d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41398/0 (socket says 172.21.15.7:41398) 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f3769d9b640 1 -- 172.21.15.7:0/3459792595 learned_addr learned my addr 172.21.15.7:0/3459792595 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f376a59c640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f376c0c7980 0x7f376c1e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.828+0000 7f376ad9d640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f376c1575c0 0x7f376c152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3769d9b640 1 -- 172.21.15.7:0/3459792595 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f376c0c7980 msgr2=0x7f376c1e52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3769d9b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f376c0c7980 0x7f376c1e52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3769d9b640 1 -- 172.21.15.7:0/3459792595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f376c1575c0 msgr2=0x7f376c152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3769d9b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f376c1575c0 0x7f376c152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3769d9b640 1 -- 172.21.15.7:0/3459792595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c00b6c0 con 0x7f376c150b30 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3769d9b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c151f80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3760001ec0 tx=0x7f376000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f375affd640 1 -- 172.21.15.7:0/3459792595 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37600054e0 con 0x7f376c150b30 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f375affd640 1 -- 172.21.15.7:0/3459792595 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3760003600 con 0x7f376c150b30 2023-11-28T18:43:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f375affd640 1 -- 172.21.15.7:0/3459792595 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37600039b0 con 0x7f376c150b30 2023-11-28T18:43:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376c073030 con 0x7f376c150b30 2023-11-28T18:43:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376c1e5800 con 0x7f376c150b30 2023-11-28T18:43:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376c072e90 con 0x7f376c150b30 2023-11-28T18:43:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.832+0000 7f375affd640 1 -- 172.21.15.7:0/3459792595 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3760003b10 con 0x7f376c150b30 2023-11-28T18:43:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.836+0000 7f375affd640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f374c066920 0x7f374c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.836+0000 7f375affd640 1 -- 172.21.15.7:0/3459792595 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f3760083d80 con 0x7f376c150b30 2023-11-28T18:43:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.836+0000 7f376a59c640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f374c066920 0x7f374c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.836+0000 7f376a59c640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f374c066920 0x7f374c068de0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f375c0b1000 tx=0x7f375c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:45.840+0000 7f375affd640 1 -- 172.21.15.7:0/3459792595 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f376c072e90 con 0x7f376c150b30 2023-11-28T18:43:46.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.056+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f376c072e90 con 0x7f376c150b30 2023-11-28T18:43:46.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f375affd640 1 -- 172.21.15.7:0/3459792595 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f376c072e90 con 0x7f376c150b30 2023-11-28T18:43:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:46.061 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:43.484108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:46.062 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 96 2023-11-28T18:43:46.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f374c066920 msgr2=0x7f374c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f374c066920 0x7f374c068de0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f375c0b1000 tx=0x7f375c008040 comp rx=0 tx=0).stop 2023-11-28T18:43:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 msgr2=0x7f376c151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c151f80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3760001ec0 tx=0x7f376000c8e0 comp rx=0 tx=0).stop 2023-11-28T18:43:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 shutdown_connections 2023-11-28T18:43:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f374c066920 0x7f374c068de0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f376c1575c0 0x7f376c152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f376c150b30 0x7f376c151f80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 --2- 172.21.15.7:0/3459792595 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f376c0c7980 0x7f376c1e52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 >> 172.21.15.7:0/3459792595 conn(0x7f376c0662e0 msgr2=0x7f376c0c7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 shutdown_connections 2023-11-28T18:43:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:46.060+0000 7f3770e6b640 1 -- 172.21.15.7:0/3459792595 wait complete. 2023-11-28T18:43:46.085 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 96, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 96, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5528": { "addr": "172.21.15.7:6835/695953635", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6834", "nonce": 695953635, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 695953635, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5528, "incarnation": 84, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 30 }, "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:reconnect", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:43:43.484108+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5528, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:43:48.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:48.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93ecde6640 1 Processor -- start 2023-11-28T18:43:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93ecde6640 1 -- start start 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93ecde6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f93e81542e0 0x7f93e815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93ecde6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 0x7f93e80cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93ecde6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e80ca920 con 0x7f93e80cc810 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93ecde6640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e8071d50 con 0x7f93e81542e0 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93ecde6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e8072030 con 0x7f93e80cbb70 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93e659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f93e81542e0 0x7f93e815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93e659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f93e81542e0 0x7f93e815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41400/0 (socket says 172.21.15.7:41400) 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93e659c640 1 -- 172.21.15.7:0/566597968 learned_addr learned my addr 172.21.15.7:0/566597968 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93e5d9b640 1 --2- 172.21.15.7:0/566597968 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 0x7f93e80cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93e659c640 1 -- 172.21.15.7:0/566597968 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 msgr2=0x7f93e80cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93e659c640 1 --2- 172.21.15.7:0/566597968 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 0x7f93e80cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93e659c640 1 -- 172.21.15.7:0/566597968 >> v1:172.21.15.193:6789/0 conn(0x7f93e80cc810 legacy=0x7f93e8072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:48.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.236+0000 7f93e659c640 1 -- 172.21.15.7:0/566597968 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e815e550 con 0x7f93e81542e0 2023-11-28T18:43:48.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93e659c640 1 --2- 172.21.15.7:0/566597968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e815e010 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009870 tx=0x7f93d801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad407efa15d3a45 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93e559a640 1 -- 172.21.15.7:0/566597968 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d80ab650 con 0x7f93e81542e0 2023-11-28T18:43:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93e559a640 1 -- 172.21.15.7:0/566597968 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d80ab7b0 con 0x7f93e81542e0 2023-11-28T18:43:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93e559a640 1 -- 172.21.15.7:0/566597968 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d80abb40 con 0x7f93e81542e0 2023-11-28T18:43:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93ecde6640 1 -- 172.21.15.7:0/566597968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 msgr2=0x7f93e815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93ecde6640 1 --2- 172.21.15.7:0/566597968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e815e010 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009870 tx=0x7f93d801c540 comp rx=0 tx=0).stop 2023-11-28T18:43:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93ecde6640 1 -- 172.21.15.7:0/566597968 shutdown_connections 2023-11-28T18:43:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93ecde6640 1 --2- 172.21.15.7:0/566597968 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 0x7f93e80cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93ecde6640 1 --2- 172.21.15.7:0/566597968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e815e010 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93ecde6640 1 -- 172.21.15.7:0/566597968 >> 172.21.15.7:0/566597968 conn(0x7f93e80717e0 msgr2=0x7f93e80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93ecde6640 1 -- 172.21.15.7:0/566597968 shutdown_connections 2023-11-28T18:43:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.240+0000 7f93ecde6640 1 -- 172.21.15.7:0/566597968 wait complete. 2023-11-28T18:43:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 Processor -- start 2023-11-28T18:43:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 -- start start 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 0x7f93e80cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93e80cc810 0x7f93e80c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e8159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e80cff50 con 0x7f93e81542e0 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e8159740 con 0x7f93e80cc810 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e80d02d0 con 0x7f93e80cbb70 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e8159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93e80cc810 0x7f93e80c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93e80cc810 0x7f93e80c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:43028/0 (socket says 172.21.15.7:43028) 2023-11-28T18:43:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e5d9b640 1 -- 172.21.15.7:0/4029654018 learned_addr learned my addr 172.21.15.7:0/4029654018 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e8159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41408/0 (socket says 172.21.15.7:41408) 2023-11-28T18:43:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e659c640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 0x7f93e80cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e6d9d640 1 -- 172.21.15.7:0/4029654018 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 msgr2=0x7f93e80cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e6d9d640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 0x7f93e80cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e6d9d640 1 -- 172.21.15.7:0/4029654018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93e80cc810 msgr2=0x7f93e80c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e6d9d640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93e80cc810 0x7f93e80c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e6d9d640 1 -- 172.21.15.7:0/4029654018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d80094d0 con 0x7f93e81542e0 2023-11-28T18:43:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e5d9b640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93e80cc810 0x7f93e80c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93e6d9d640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e8159bd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c700 tx=0x7f93e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93d6ffd640 1 -- 172.21.15.7:0/4029654018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0003b80 con 0x7f93e81542e0 2023-11-28T18:43:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93d6ffd640 1 -- 172.21.15.7:0/4029654018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e0003ce0 con 0x7f93e81542e0 2023-11-28T18:43:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93d6ffd640 1 -- 172.21.15.7:0/4029654018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e000aa30 con 0x7f93e81542e0 2023-11-28T18:43:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e80cf050 con 0x7f93e81542e0 2023-11-28T18:43:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.244+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e81590f0 con 0x7f93e81542e0 2023-11-28T18:43:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.248+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e806c670 con 0x7f93e81542e0 2023-11-28T18:43:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.248+0000 7f93d6ffd640 1 -- 172.21.15.7:0/4029654018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f93e000ab90 con 0x7f93e81542e0 2023-11-28T18:43:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.248+0000 7f93d6ffd640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93c8066900 0x7f93c8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.248+0000 7f93d6ffd640 1 -- 172.21.15.7:0/4029654018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f93e00843e0 con 0x7f93e81542e0 2023-11-28T18:43:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.248+0000 7f93e659c640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93c8066900 0x7f93c8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.248+0000 7f93e659c640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93c8066900 0x7f93c8068dc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f93d80af000 tx=0x7f93d80b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.252+0000 7f93d6ffd640 1 -- 172.21.15.7:0/4029654018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f93e0051330 con 0x7f93e81542e0 2023-11-28T18:43:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.440+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93e804e890 con 0x7f93e81542e0 2023-11-28T18:43:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.440+0000 7f93d6ffd640 1 -- 172.21.15.7:0/4029654018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f93e0054950 con 0x7f93e81542e0 2023-11-28T18:43:48.444 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:48.445 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:43.484108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:48.445 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 96 2023-11-28T18:43:48.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93c8066900 msgr2=0x7f93c8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:48.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93c8066900 0x7f93c8068dc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f93d80af000 tx=0x7f93d80b6040 comp rx=0 tx=0).stop 2023-11-28T18:43:48.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 msgr2=0x7f93e8159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:48.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e8159bd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c700 tx=0x7f93e000cbc0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 shutdown_connections 2023-11-28T18:43:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f93c8066900 0x7f93c8068dc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e81542e0 0x7f93e8159bd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93e80cc810 0x7f93e80c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 --2- 172.21.15.7:0/4029654018 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f93e80cbb70 0x7f93e80cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 >> 172.21.15.7:0/4029654018 conn(0x7f93e80717e0 msgr2=0x7f93e815b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 shutdown_connections 2023-11-28T18:43:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:48.444+0000 7f93ecde6640 1 -- 172.21.15.7:0/4029654018 wait complete. 2023-11-28T18:43:48.466 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:43:50.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:50.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.608+0000 7fadb4557640 1 Processor -- start 2023-11-28T18:43:50.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.608+0000 7fadb4557640 1 -- start start 2023-11-28T18:43:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb4557640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fadac151cd0 0x7fadac15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb4557640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac157b90 0x7fadac0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb4557640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac157fe0 con 0x7fadac06e560 2023-11-28T18:43:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb4557640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadac070730 con 0x7fadac151cd0 2023-11-28T18:43:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb4557640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadac071e60 con 0x7fadac157b90 2023-11-28T18:43:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb1af2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fadac06e560 0x7fadac0c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52282/0 (socket says 172.21.15.7:52282) 2023-11-28T18:43:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb1af2640 1 -- 172.21.15.7:0/2164625353 learned_addr learned my addr 172.21.15.7:0/2164625353 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:43:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb22f3640 1 --2- 172.21.15.7:0/2164625353 >> v2:172.21.15.193:3300/0 conn(0x7fadac151cd0 0x7fadac15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:50.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb2af4640 1 --2- 172.21.15.7:0/2164625353 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac157b90 0x7fadac0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:50.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb12f1640 1 -- 172.21.15.7:0/2164625353 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910367175 0 0) 0x7fadac157fe0 con 0x7fadac06e560 2023-11-28T18:43:50.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb12f1640 1 -- 172.21.15.7:0/2164625353 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad98003530 con 0x7fadac06e560 2023-11-28T18:43:50.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb22f3640 1 -- 172.21.15.7:0/2164625353 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac157b90 msgr2=0x7fadac0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:50.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb22f3640 1 --2- 172.21.15.7:0/2164625353 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac157b90 0x7fadac0701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb22f3640 1 -- 172.21.15.7:0/2164625353 >> v1:172.21.15.7:6789/0 conn(0x7fadac06e560 legacy=0x7fadac0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.612+0000 7fadb22f3640 1 -- 172.21.15.7:0/2164625353 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac070930 con 0x7fadac151cd0 2023-11-28T18:43:50.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb22f3640 1 --2- 172.21.15.7:0/2164625353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac151cd0 0x7fadac15bd00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fada80097c0 tx=0x7fada801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb5fa0a238dab5ed server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb12f1640 1 -- 172.21.15.7:0/2164625353 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada80ab600 con 0x7fadac151cd0 2023-11-28T18:43:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb12f1640 1 -- 172.21.15.7:0/2164625353 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada80a7050 con 0x7fadac151cd0 2023-11-28T18:43:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb12f1640 1 -- 172.21.15.7:0/2164625353 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada80ab980 con 0x7fadac151cd0 2023-11-28T18:43:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 -- 172.21.15.7:0/2164625353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac151cd0 msgr2=0x7fadac15bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 --2- 172.21.15.7:0/2164625353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac151cd0 0x7fadac15bd00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fada80097c0 tx=0x7fada801c3f0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 -- 172.21.15.7:0/2164625353 shutdown_connections 2023-11-28T18:43:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 --2- 172.21.15.7:0/2164625353 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac157b90 0x7fadac0701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 --2- 172.21.15.7:0/2164625353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac151cd0 0x7fadac15bd00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 -- 172.21.15.7:0/2164625353 >> 172.21.15.7:0/2164625353 conn(0x7fadac0662e0 msgr2=0x7fadac0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 -- 172.21.15.7:0/2164625353 shutdown_connections 2023-11-28T18:43:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 -- 172.21.15.7:0/2164625353 wait complete. 2023-11-28T18:43:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 Processor -- start 2023-11-28T18:43:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.616+0000 7fadb4557640 1 -- start start 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac06e560 0x7fadac1439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac151cd0 0x7fadac1f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac157b90 0x7fadac156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac1f4130 con 0x7fadac151cd0 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac1f3e00 con 0x7fadac157b90 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadac157370 con 0x7fadac06e560 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac151cd0 0x7fadac1f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac151cd0 0x7fadac1f45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41416/0 (socket says 172.21.15.7:41416) 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 -- 172.21.15.7:0/1565621487 learned_addr learned my addr 172.21.15.7:0/1565621487 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb22f3640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac06e560 0x7fadac1439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb2af4640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac157b90 0x7fadac156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 -- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac06e560 msgr2=0x7fadac1439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac06e560 0x7fadac1439a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 -- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac157b90 msgr2=0x7fadac156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac157b90 0x7fadac156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 -- 172.21.15.7:0/1565621487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8009420 con 0x7fadac151cd0 2023-11-28T18:43:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb2af4640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac157b90 0x7fadac156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:43:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb1af2640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac151cd0 0x7fadac1f45c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c4d0 tx=0x7fad9c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fada2ffd640 1 -- 172.21.15.7:0/1565621487 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c005cf0 con 0x7fadac151cd0 2023-11-28T18:43:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fada2ffd640 1 -- 172.21.15.7:0/1565621487 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c005e50 con 0x7fadac151cd0 2023-11-28T18:43:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fada2ffd640 1 -- 172.21.15.7:0/1565621487 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c007270 con 0x7fadac151cd0 2023-11-28T18:43:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac157650 con 0x7fadac151cd0 2023-11-28T18:43:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac0c8a00 con 0x7fadac151cd0 2023-11-28T18:43:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac04e890 con 0x7fadac151cd0 2023-11-28T18:43:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.620+0000 7fada2ffd640 1 -- 172.21.15.7:0/1565621487 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fad9c0073d0 con 0x7fadac151cd0 2023-11-28T18:43:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.624+0000 7fada2ffd640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066900 0x7fad90068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.624+0000 7fada2ffd640 1 -- 172.21.15.7:0/1565621487 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fad9c083dc0 con 0x7fadac151cd0 2023-11-28T18:43:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.624+0000 7fadb22f3640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066900 0x7fad90068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.624+0000 7fadb22f3640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066900 0x7fad90068dc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fada80ad000 tx=0x7fada8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:50.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.628+0000 7fada2ffd640 1 -- 172.21.15.7:0/1565621487 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fad9c050bf0 con 0x7fadac151cd0 2023-11-28T18:43:50.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.828+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadac1555b0 con 0x7fadac151cd0 2023-11-28T18:43:50.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fada2ffd640 1 -- 172.21.15.7:0/1565621487 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5762 (secure 0 0 0) 0x7fad9c017070 con 0x7fadac151cd0 2023-11-28T18:43:50.833 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:50.834 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:48.477847+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 97 2023-11-28T18:43:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066900 msgr2=0x7fad90068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066900 0x7fad90068dc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fada80ad000 tx=0x7fada8008040 comp rx=0 tx=0).stop 2023-11-28T18:43:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac151cd0 msgr2=0x7fadac1f45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac151cd0 0x7fadac1f45c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c4d0 tx=0x7fad9c00c990 comp rx=0 tx=0).stop 2023-11-28T18:43:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 shutdown_connections 2023-11-28T18:43:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fad90066900 0x7fad90068dc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadac157b90 0x7fadac156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadac151cd0 0x7fadac1f45c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 --2- 172.21.15.7:0/1565621487 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadac06e560 0x7fadac1439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 >> 172.21.15.7:0/1565621487 conn(0x7fadac0662e0 msgr2=0x7fadac148500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 shutdown_connections 2023-11-28T18:43:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:50.832+0000 7fadb4557640 1 -- 172.21.15.7:0/1565621487 wait complete. 2023-11-28T18:43:50.858 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:43:52.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:53.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0d0e09640 1 Processor -- start 2023-11-28T18:43:53.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0d0e09640 1 -- start start 2023-11-28T18:43:53.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0d0e09640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0cc072950 0x7fa0cc074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:53.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0d0e09640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0c8490 con 0x7fa0cc06c920 2023-11-28T18:43:53.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0d0e09640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0c81f0 con 0x7fa0cc073070 2023-11-28T18:43:53.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0d0e09640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0cc15ab00 con 0x7fa0cc072950 2023-11-28T18:43:53.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0ca59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0cc072950 0x7fa0cc074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:53.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0ca59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0cc072950 0x7fa0cc074030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41422/0 (socket says 172.21.15.7:41422) 2023-11-28T18:43:53.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0ca59c640 1 -- 172.21.15.7:0/1100271406 learned_addr learned my addr 172.21.15.7:0/1100271406 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:53.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.012+0000 7fa0cad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa0cc06c920 0x7fa0cc1647f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52298/0 (socket says 172.21.15.7:52298) 2023-11-28T18:43:53.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0ca59c640 1 -- 172.21.15.7:0/1100271406 >> v1:172.21.15.193:6789/0 conn(0x7fa0cc073070 legacy=0x7fa0cc0c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:53.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0ca59c640 1 -- 172.21.15.7:0/1100271406 >> v1:172.21.15.7:6789/0 conn(0x7fa0cc06c920 legacy=0x7fa0cc1647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:53.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0ca59c640 1 -- 172.21.15.7:0/1100271406 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc15ade0 con 0x7fa0cc072950 2023-11-28T18:43:53.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0ca59c640 1 --2- 172.21.15.7:0/1100271406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc072950 0x7fa0cc074030 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009870 tx=0x7fa0bc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c8043191f7e758 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:53.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0c959a640 1 -- 172.21.15.7:0/1100271406 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0ab600 con 0x7fa0cc072950 2023-11-28T18:43:53.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0c959a640 1 -- 172.21.15.7:0/1100271406 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc0a7050 con 0x7fa0cc072950 2023-11-28T18:43:53.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0c959a640 1 -- 172.21.15.7:0/1100271406 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0ab980 con 0x7fa0cc072950 2023-11-28T18:43:53.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1100271406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc072950 msgr2=0x7fa0cc074030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:53.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.016+0000 7fa0d0e09640 1 --2- 172.21.15.7:0/1100271406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc072950 0x7fa0cc074030 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009870 tx=0x7fa0bc01c420 comp rx=0 tx=0).stop 2023-11-28T18:43:53.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1100271406 shutdown_connections 2023-11-28T18:43:53.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 --2- 172.21.15.7:0/1100271406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc072950 0x7fa0cc074030 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:53.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1100271406 >> 172.21.15.7:0/1100271406 conn(0x7fa0cc0cbf80 msgr2=0x7fa0cc0cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:53.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1100271406 shutdown_connections 2023-11-28T18:43:53.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1100271406 wait complete. 2023-11-28T18:43:53.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 Processor -- start 2023-11-28T18:43:53.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 -- start start 2023-11-28T18:43:53.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0cc06c920 0x7fa0cc0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:53.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0cc072950 0x7fa0cc14e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:53.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc073070 0x7fa0cc1f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:53.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc15db10 con 0x7fa0cc073070 2023-11-28T18:43:53.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0af5b0 con 0x7fa0cc06c920 2023-11-28T18:43:53.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.020+0000 7fa0d0e09640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0aff80 con 0x7fa0cc072950 2023-11-28T18:43:53.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0cad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc073070 0x7fa0cc1f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:53.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0cc06c920 0x7fa0cc0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:53.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0cc06c920 0x7fa0cc0afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:43046/0 (socket says 172.21.15.7:43046) 2023-11-28T18:43:53.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 -- 172.21.15.7:0/1549198087 learned_addr learned my addr 172.21.15.7:0/1549198087 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0cad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc073070 0x7fa0cc1f5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41426/0 (socket says 172.21.15.7:41426) 2023-11-28T18:43:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0c9d9b640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0cc072950 0x7fa0cc14e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 -- 172.21.15.7:0/1549198087 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0cc072950 msgr2=0x7fa0cc14e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0cc072950 0x7fa0cc14e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 -- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc073070 msgr2=0x7fa0cc1f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc073070 0x7fa0cc1f5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 -- 172.21.15.7:0/1549198087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0094d0 con 0x7fa0cc06c920 2023-11-28T18:43:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0cad9d640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc073070 0x7fa0cc1f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0ca59c640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0cc06c920 0x7fa0cc0afa40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc019030 tx=0x7fa0bc01ac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0baffd640 1 -- 172.21.15.7:0/1549198087 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0b6620 con 0x7fa0cc06c920 2023-11-28T18:43:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0baffd640 1 -- 172.21.15.7:0/1549198087 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc0a4e90 con 0x7fa0cc06c920 2023-11-28T18:43:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0baffd640 1 -- 172.21.15.7:0/1549198087 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0b69e0 con 0x7fa0cc06c920 2023-11-28T18:43:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc1f36b0 con 0x7fa0cc06c920 2023-11-28T18:43:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc0aec80 con 0x7fa0cc06c920 2023-11-28T18:43:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc04e890 con 0x7fa0cc06c920 2023-11-28T18:43:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.024+0000 7fa0baffd640 1 -- 172.21.15.7:0/1549198087 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa0bc0b2040 con 0x7fa0cc06c920 2023-11-28T18:43:53.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.028+0000 7fa0baffd640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa0ac0668d0 0x7fa0ac068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:53.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.028+0000 7fa0baffd640 1 -- 172.21.15.7:0/1549198087 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fa0bc11ec50 con 0x7fa0cc06c920 2023-11-28T18:43:53.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.028+0000 7fa0c9d9b640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa0ac0668d0 0x7fa0ac068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:53.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.028+0000 7fa0c9d9b640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa0ac0668d0 0x7fa0ac068d90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0c14a0 tx=0x7fa0c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.032+0000 7fa0baffd640 1 -- 172.21.15.7:0/1549198087 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa0bc0eba70 con 0x7fa0cc06c920 2023-11-28T18:43:53.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.224+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0cc0c11f0 con 0x7fa0cc06c920 2023-11-28T18:43:53.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.224+0000 7fa0baffd640 1 -- 172.21.15.7:0/1549198087 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5762 (secure 0 0 0) 0x7fa0bc0ba090 con 0x7fa0cc06c920 2023-11-28T18:43:53.228 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:53.228 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:48.477847+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:53.229 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 97 2023-11-28T18:43:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa0ac0668d0 msgr2=0x7fa0ac068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa0ac0668d0 0x7fa0ac068d90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0c14a0 tx=0x7fa0c0009040 comp rx=0 tx=0).stop 2023-11-28T18:43:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0cc06c920 msgr2=0x7fa0cc0afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0cc06c920 0x7fa0cc0afa40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc019030 tx=0x7fa0bc01ac60 comp rx=0 tx=0).stop 2023-11-28T18:43:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 shutdown_connections 2023-11-28T18:43:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa0ac0668d0 0x7fa0ac068d90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0cc073070 0x7fa0cc1f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0cc072950 0x7fa0cc14e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 --2- 172.21.15.7:0/1549198087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0cc06c920 0x7fa0cc0afa40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 >> 172.21.15.7:0/1549198087 conn(0x7fa0cc0cbf80 msgr2=0x7fa0cc0cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 shutdown_connections 2023-11-28T18:43:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:53.228+0000 7fa0d0e09640 1 -- 172.21.15.7:0/1549198087 wait complete. 2023-11-28T18:43:53.250 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:43:55.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:43:55.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73f185f640 1 Processor -- start 2023-11-28T18:43:55.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73f185f640 1 -- start start 2023-11-28T18:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73f185f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73f185f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f73ec06e560 0x7f73ec157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73f185f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f73ec0701f0 0x7f73ec0c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73f185f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ec071e60 con 0x7f73ec06e560 2023-11-28T18:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73f185f640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ec070930 con 0x7f73ec0701f0 2023-11-28T18:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73f185f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec070660 con 0x7f73ec151cd0 2023-11-28T18:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73ea7fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f73ec06e560 0x7f73ec157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73ea7fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f73ec06e560 0x7f73ec157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50410/0 (socket says 172.21.15.7:50410) 2023-11-28T18:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73ea7fc640 1 -- 172.21.15.7:0/125088434 learned_addr learned my addr 172.21.15.7:0/125088434 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73eaffd640 1 --2- 172.21.15.7:0/125088434 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.560+0000 7f73eb7fe640 1 --2- 172.21.15.7:0/125088434 >> v2:172.21.15.193:3300/0 conn(0x7f73ec0701f0 0x7f73ec0c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73ea7fc640 1 -- 172.21.15.7:0/125088434 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 msgr2=0x7f73ec15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73ea7fc640 1 --2- 172.21.15.7:0/125088434 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73ea7fc640 1 -- 172.21.15.7:0/125088434 >> v2:172.21.15.193:3300/0 conn(0x7f73ec0701f0 msgr2=0x7f73ec0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73ea7fc640 1 --2- 172.21.15.7:0/125088434 >> v2:172.21.15.193:3300/0 conn(0x7f73ec0701f0 0x7f73ec0c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73ea7fc640 1 -- 172.21.15.7:0/125088434 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec070bf0 con 0x7f73ec06e560 2023-11-28T18:43:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73ea7fc640 1 --2- 172.21.15.7:0/125088434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec157b90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0c95f0 tx=0x7f73e001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f361859b41979af server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73e9ffb640 1 -- 172.21.15.7:0/125088434 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e00ab600 con 0x7f73ec06e560 2023-11-28T18:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73e9ffb640 1 -- 172.21.15.7:0/125088434 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e00a7050 con 0x7f73ec06e560 2023-11-28T18:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.564+0000 7f73e9ffb640 1 -- 172.21.15.7:0/125088434 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e00ab980 con 0x7f73ec06e560 2023-11-28T18:43:55.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- 172.21.15.7:0/125088434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 msgr2=0x7f73ec157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 --2- 172.21.15.7:0/125088434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec157b90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0c95f0 tx=0x7f73e001c470 comp rx=0 tx=0).stop 2023-11-28T18:43:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- 172.21.15.7:0/125088434 shutdown_connections 2023-11-28T18:43:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 --2- 172.21.15.7:0/125088434 >> v2:172.21.15.193:3300/0 conn(0x7f73ec0701f0 0x7f73ec0c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 --2- 172.21.15.7:0/125088434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec157b90 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 --2- 172.21.15.7:0/125088434 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- 172.21.15.7:0/125088434 >> 172.21.15.7:0/125088434 conn(0x7f73ec0662e0 msgr2=0x7f73ec0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- 172.21.15.7:0/125088434 shutdown_connections 2023-11-28T18:43:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- 172.21.15.7:0/125088434 wait complete. 2023-11-28T18:43:55.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 Processor -- start 2023-11-28T18:43:55.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- start start 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec0701f0 0x7f73ec0c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec0c38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec1f6fa0 con 0x7f73ec06e560 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec154450 con 0x7f73ec0701f0 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73f185f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec154730 con 0x7f73ec151cd0 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73eaffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73eaffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50414/0 (socket says 172.21.15.7:50414) 2023-11-28T18:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73eaffd640 1 -- 172.21.15.7:0/4121856340 learned_addr learned my addr 172.21.15.7:0/4121856340 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73eb7fe640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec0c38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.568+0000 7f73ea7fc640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec0701f0 0x7f73ec0c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.572+0000 7f73eb7fe640 1 -- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec0701f0 msgr2=0x7f73ec0c33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.572+0000 7f73eb7fe640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec0701f0 0x7f73ec0c33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.572+0000 7f73eb7fe640 1 -- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 msgr2=0x7f73ec152dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.572+0000 7f73eb7fe640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec152dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.572+0000 7f73eb7fe640 1 -- 172.21.15.7:0/4121856340 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00094d0 con 0x7f73ec151cd0 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.572+0000 7f73eaffd640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.572+0000 7f73eb7fe640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec0c38f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f73e400eed0 tx=0x7f73e400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:55.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.588+0000 7f73d3fff640 1 -- 172.21.15.7:0/4121856340 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4010040 con 0x7f73ec151cd0 2023-11-28T18:43:55.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.588+0000 7f73d3fff640 1 -- 172.21.15.7:0/4121856340 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e4009e60 con 0x7f73ec151cd0 2023-11-28T18:43:55.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.588+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ec154a10 con 0x7f73ec151cd0 2023-11-28T18:43:55.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.588+0000 7f73d3fff640 1 -- 172.21.15.7:0/4121856340 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4014640 con 0x7f73ec151cd0 2023-11-28T18:43:55.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.588+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ec154fd0 con 0x7f73ec151cd0 2023-11-28T18:43:55.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.588+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ec1551c0 con 0x7f73ec151cd0 2023-11-28T18:43:55.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.588+0000 7f73d3fff640 1 -- 172.21.15.7:0/4121856340 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f73e40147a0 con 0x7f73ec151cd0 2023-11-28T18:43:55.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.592+0000 7f73d3fff640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73d80668d0 0x7f73d8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:43:55.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.592+0000 7f73d3fff640 1 -- 172.21.15.7:0/4121856340 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f73e4085160 con 0x7f73ec151cd0 2023-11-28T18:43:55.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.592+0000 7f73eaffd640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73d80668d0 0x7f73d8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:43:55.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.592+0000 7f73eaffd640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73d80668d0 0x7f73d8068d90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009f70 tx=0x7f73dc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:43:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.596+0000 7f73d3fff640 1 -- 172.21.15.7:0/4121856340 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f73e4052010 con 0x7f73ec151cd0 2023-11-28T18:43:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73ec04e890 con 0x7f73ec151cd0 2023-11-28T18:43:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73d3fff640 1 -- 172.21.15.7:0/4121856340 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5762 (secure 0 0 0) 0x7f73e4055630 con 0x7f73ec151cd0 2023-11-28T18:43:55.798 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:43:55.798 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:43:53.516792+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:43:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 98 2023-11-28T18:43:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73d80668d0 msgr2=0x7f73d8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73d80668d0 0x7f73d8068d90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009f70 tx=0x7f73dc009190 comp rx=0 tx=0).stop 2023-11-28T18:43:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 msgr2=0x7f73ec0c38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:43:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec0c38f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f73e400eed0 tx=0x7f73e400b480 comp rx=0 tx=0).stop 2023-11-28T18:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 shutdown_connections 2023-11-28T18:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f73d80668d0 0x7f73d8068d90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f73ec151cd0 0x7f73ec0c38f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec0701f0 0x7f73ec0c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 --2- 172.21.15.7:0/4121856340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73ec06e560 0x7f73ec152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 >> 172.21.15.7:0/4121856340 conn(0x7f73ec0662e0 msgr2=0x7f73ec1614a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 shutdown_connections 2023-11-28T18:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:43:55.796+0000 7f73f185f640 1 -- 172.21.15.7:0/4121856340 wait complete. 2023-11-28T18:43:55.821 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=2, replacing gid=5450 2023-11-28T18:43:55.821 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.f 2023-11-28T18:44:04.822 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-11-28T18:44:04.823 INFO:tasks.ceph.mds.f:Restarting daemon 2023-11-28T18:44:04.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-11-28T18:44:04.827 INFO:tasks.ceph.mds.f:Started 2023-11-28T18:44:04.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:44:04.942 INFO:tasks.ceph.mds.f.smithi193.stdout:starting mds.f at 2023-11-28T18:44:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.055+0000 7fe36af48640 1 Processor -- start 2023-11-28T18:44:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.055+0000 7fe36af48640 1 -- start start 2023-11-28T18:44:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe36af48640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c046d50 0x7fe35c0dad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe36af48640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0dbd20 con 0x7fe35c04b6b0 2023-11-28T18:44:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe36af48640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0468c0 con 0x7fe35c04b2d0 2023-11-28T18:44:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe36af48640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c0dba80 con 0x7fe35c046d50 2023-11-28T18:44:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe369f46640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fe35c04b2d0 0x7fe35c005ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:39398/0 (socket says 172.21.15.7:39398) 2023-11-28T18:44:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe369f46640 1 -- 172.21.15.7:0/372713048 learned_addr learned my addr 172.21.15.7:0/372713048 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:44:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe36a747640 1 --2- 172.21.15.7:0/372713048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c046d50 0x7fe35c0dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:05.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe36a747640 1 -- 172.21.15.7:0/372713048 >> v1:172.21.15.193:6789/0 conn(0x7fe35c04b2d0 legacy=0x7fe35c005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:05.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe36a747640 1 -- 172.21.15.7:0/372713048 >> v1:172.21.15.7:6789/0 conn(0x7fe35c04b6b0 legacy=0x7fe35c0d4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:05.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.059+0000 7fe36a747640 1 -- 172.21.15.7:0/372713048 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c045f90 con 0x7fe35c046d50 2023-11-28T18:44:05.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.063+0000 7fe36a747640 1 --2- 172.21.15.7:0/372713048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c046d50 0x7fe35c0dad10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe36404f6e0 tx=0x7fe3640977f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd163875a756a78 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:44:05.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.063+0000 7fe368f44640 1 -- 172.21.15.7:0/372713048 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3641236b0 con 0x7fe35c046d50 2023-11-28T18:44:05.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.063+0000 7fe368f44640 1 -- 172.21.15.7:0/372713048 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe364123850 con 0x7fe35c046d50 2023-11-28T18:44:05.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.063+0000 7fe368f44640 1 -- 172.21.15.7:0/372713048 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe364123c80 con 0x7fe35c046d50 2023-11-28T18:44:05.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.063+0000 7fe36af48640 1 -- 172.21.15.7:0/372713048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c046d50 msgr2=0x7fe35c0dad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:05.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.063+0000 7fe36af48640 1 --2- 172.21.15.7:0/372713048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c046d50 0x7fe35c0dad10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe36404f6e0 tx=0x7fe3640977f0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.067+0000 7fe36af48640 1 -- 172.21.15.7:0/372713048 shutdown_connections 2023-11-28T18:44:05.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.067+0000 7fe36af48640 1 --2- 172.21.15.7:0/372713048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c046d50 0x7fe35c0dad10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.067+0000 7fe36af48640 1 -- 172.21.15.7:0/372713048 >> 172.21.15.7:0/372713048 conn(0x7fe35c050160 msgr2=0x7fe35c050570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:44:05.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.067+0000 7fe36af48640 1 -- 172.21.15.7:0/372713048 shutdown_connections 2023-11-28T18:44:05.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.067+0000 7fe36af48640 1 -- 172.21.15.7:0/372713048 wait complete. 2023-11-28T18:44:05.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe36af48640 1 Processor -- start 2023-11-28T18:44:05.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe36af48640 1 -- start start 2023-11-28T18:44:05.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe36af48640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35c046d50 0x7fe35c0c0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe36af48640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c04b2d0 0x7fe35c0c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe36af48640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe35c04b6b0 0x7fe35c04e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe36af48640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c04a450 con 0x7fe35c046d50 2023-11-28T18:44:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe36af48640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c04fe70 con 0x7fe35c04b6b0 2023-11-28T18:44:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe36af48640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c04a6a0 con 0x7fe35c04b2d0 2023-11-28T18:44:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35c046d50 0x7fe35c0c0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35c046d50 0x7fe35c0c0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42560/0 (socket says 172.21.15.7:42560) 2023-11-28T18:44:05.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 -- 172.21.15.7:0/449086594 learned_addr learned my addr 172.21.15.7:0/449086594 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:44:05.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369745640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c04b2d0 0x7fe35c0c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:05.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 -- 172.21.15.7:0/449086594 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c04b2d0 msgr2=0x7fe35c0c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:05.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c04b2d0 0x7fe35c0c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 -- 172.21.15.7:0/449086594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe35c04b6b0 msgr2=0x7fe35c04e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:05.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe35c04b6b0 0x7fe35c04e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 -- 172.21.15.7:0/449086594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c178e40 con 0x7fe35c046d50 2023-11-28T18:44:05.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.071+0000 7fe369f46640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35c046d50 0x7fe35c0c0150 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe35400c500 tx=0x7fe35400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:44:05.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.075+0000 7fe35a7fc640 1 -- 172.21.15.7:0/449086594 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354005cf0 con 0x7fe35c046d50 2023-11-28T18:44:05.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.075+0000 7fe35a7fc640 1 -- 172.21.15.7:0/449086594 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe354005e50 con 0x7fe35c046d50 2023-11-28T18:44:05.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.075+0000 7fe35a7fc640 1 -- 172.21.15.7:0/449086594 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354007270 con 0x7fe35c046d50 2023-11-28T18:44:05.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.075+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c179120 con 0x7fe35c046d50 2023-11-28T18:44:05.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.075+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c0c1550 con 0x7fe35c046d50 2023-11-28T18:44:05.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.075+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35c0bfb20 con 0x7fe35c046d50 2023-11-28T18:44:05.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.079+0000 7fe35a7fc640 1 -- 172.21.15.7:0/449086594 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe3540073d0 con 0x7fe35c046d50 2023-11-28T18:44:05.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.079+0000 7fe35a7fc640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3480669d0 0x7fe348068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:05.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.079+0000 7fe35a7fc640 1 -- 172.21.15.7:0/449086594 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fe354083c90 con 0x7fe35c046d50 2023-11-28T18:44:05.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.079+0000 7fe369745640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3480669d0 0x7fe348068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:05.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.083+0000 7fe369745640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3480669d0 0x7fe348068e90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0e6670 tx=0x7fe360009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:44:05.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.083+0000 7fe35a7fc640 1 -- 172.21.15.7:0/449086594 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe35c0bfb20 con 0x7fe35c046d50 2023-11-28T18:44:05.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.483+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe35c0bfb20 con 0x7fe35c046d50 2023-11-28T18:44:05.491 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:44:05.491 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:44:04.528599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:44:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 99 2023-11-28T18:44:05.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.487+0000 7fe35a7fc640 1 -- 172.21.15.7:0/449086594 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5763 (secure 0 0 0) 0x7fe35c0bfb20 con 0x7fe35c046d50 2023-11-28T18:44:05.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3480669d0 msgr2=0x7fe348068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3480669d0 0x7fe348068e90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0e6670 tx=0x7fe360009040 comp rx=0 tx=0).stop 2023-11-28T18:44:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35c046d50 msgr2=0x7fe35c0c0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35c046d50 0x7fe35c0c0150 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe35400c500 tx=0x7fe35400c9c0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 shutdown_connections 2023-11-28T18:44:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3480669d0 0x7fe348068e90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe35c04b6b0 0x7fe35c04e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe35c04b2d0 0x7fe35c0c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 --2- 172.21.15.7:0/449086594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35c046d50 0x7fe35c0c0150 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 >> 172.21.15.7:0/449086594 conn(0x7fe35c050160 msgr2=0x7fe35c0d3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:44:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 shutdown_connections 2023-11-28T18:44:05.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:05.491+0000 7fe36af48640 1 -- 172.21.15.7:0/449086594 wait complete. 2023-11-28T18:44:05.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-11-28T18:44:07.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:44:07.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e90f2a640 1 Processor -- start 2023-11-28T18:44:07.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e90f2a640 1 -- start start 2023-11-28T18:44:07.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e90f2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:07.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e90f2a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e8c0cbb70 0x7f8e8c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:07.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e90f2a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c0ca920 con 0x7f8e8c0cc810 2023-11-28T18:44:07.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e90f2a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c071d50 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e90f2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c072030 con 0x7f8e8c1542e0 2023-11-28T18:44:07.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e8a59c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:07.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e8ad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8e8c0cc810 0x7f8e8c072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58788/0 (socket says 172.21.15.7:58788) 2023-11-28T18:44:07.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e8ad9d640 1 -- 172.21.15.7:0/550603861 learned_addr learned my addr 172.21.15.7:0/550603861 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:44:07.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e89d9b640 1 --2- 172.21.15.7:0/550603861 >> v2:172.21.15.7:3300/0 conn(0x7f8e8c0cbb70 0x7f8e8c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:07.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e8a59c640 1 -- 172.21.15.7:0/550603861 >> v2:172.21.15.7:3300/0 conn(0x7f8e8c0cbb70 msgr2=0x7f8e8c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:44:07.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e8959a640 1 -- 172.21.15.7:0/550603861 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783544116 0 0) 0x7f8e8c0ca920 con 0x7f8e8c0cc810 2023-11-28T18:44:07.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e8a59c640 1 --2- 172.21.15.7:0/550603861 >> v2:172.21.15.7:3300/0 conn(0x7f8e8c0cbb70 0x7f8e8c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:07.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e8a59c640 1 -- 172.21.15.7:0/550603861 >> v1:172.21.15.7:6789/0 conn(0x7f8e8c0cc810 legacy=0x7f8e8c072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:07.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.775+0000 7f8e8a59c640 1 -- 172.21.15.7:0/550603861 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c15e550 con 0x7f8e8c1542e0 2023-11-28T18:44:07.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.779+0000 7f8e89d9b640 1 --2- 172.21.15.7:0/550603861 >> v2:172.21.15.7:3300/0 conn(0x7f8e8c0cbb70 0x7f8e8c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:44:07.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e8a59c640 1 --2- 172.21.15.7:0/550603861 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c15e010 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0097c0 tx=0x7f8e7c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=214fdefd8032d33f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:44:07.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e8959a640 1 -- 172.21.15.7:0/550603861 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0ab680 con 0x7f8e8c1542e0 2023-11-28T18:44:07.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e8959a640 1 -- 172.21.15.7:0/550603861 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c0ab7e0 con 0x7f8e8c1542e0 2023-11-28T18:44:07.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e8959a640 1 -- 172.21.15.7:0/550603861 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0abb70 con 0x7f8e8c1542e0 2023-11-28T18:44:07.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e90f2a640 1 -- 172.21.15.7:0/550603861 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 msgr2=0x7f8e8c15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:07.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/550603861 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c15e010 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0097c0 tx=0x7f8e7c01c4a0 comp rx=0 tx=0).stop 2023-11-28T18:44:07.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e90f2a640 1 -- 172.21.15.7:0/550603861 shutdown_connections 2023-11-28T18:44:07.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/550603861 >> v2:172.21.15.7:3300/0 conn(0x7f8e8c0cbb70 0x7f8e8c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:07.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/550603861 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c15e010 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:07.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.783+0000 7f8e90f2a640 1 -- 172.21.15.7:0/550603861 >> 172.21.15.7:0/550603861 conn(0x7f8e8c0717e0 msgr2=0x7f8e8c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:44:07.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.787+0000 7f8e90f2a640 1 -- 172.21.15.7:0/550603861 shutdown_connections 2023-11-28T18:44:07.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.787+0000 7f8e90f2a640 1 -- 172.21.15.7:0/550603861 wait complete. 2023-11-28T18:44:07.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.787+0000 7f8e90f2a640 1 Processor -- start 2023-11-28T18:44:07.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.787+0000 7f8e90f2a640 1 -- start start 2023-11-28T18:44:07.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e90f2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e8c0cbb70 0x7f8e8c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:07.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e90f2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0cc810 0x7f8e8c1474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:07.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e90f2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e90f2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c06d7f0 con 0x7f8e8c0cc810 2023-11-28T18:44:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e90f2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c15b230 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e90f2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c06e390 con 0x7f8e8c1542e0 2023-11-28T18:44:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e8c0cbb70 0x7f8e8c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e8c0cbb70 0x7f8e8c06de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:41550/0 (socket says 172.21.15.7:41550) 2023-11-28T18:44:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 -- 172.21.15.7:0/2227217501 learned_addr learned my addr 172.21.15.7:0/2227217501 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:44:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8ad9d640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e89d9b640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0cc810 0x7f8e8c1474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 -- 172.21.15.7:0/2227217501 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 msgr2=0x7f8e8c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c06c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 -- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0cc810 msgr2=0x7f8e8c1474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0cc810 0x7f8e8c1474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 -- 172.21.15.7:0/2227217501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c009420 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e89d9b640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0cc810 0x7f8e8c1474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:44:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e8a59c640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e8c0cbb70 0x7f8e8c06de50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c01a0d0 tx=0x7f8e7c019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:44:07.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e7affd640 1 -- 172.21.15.7:0/2227217501 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0bf650 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e7affd640 1 -- 172.21.15.7:0/2227217501 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c0bf7b0 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e7affd640 1 -- 172.21.15.7:0/2227217501 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0bfb60 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c070a00 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.791+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c1fa110 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.795+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c06c670 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.795+0000 7f8e7affd640 1 -- 172.21.15.7:0/2227217501 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8e7c0bfcc0 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.799+0000 7f8e7affd640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e6c066950 0x7f8e6c068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:44:07.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.799+0000 7f8e7affd640 1 -- 172.21.15.7:0/2227217501 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f8e7c11fc10 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.799+0000 7f8e89d9b640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e6c066950 0x7f8e6c068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:44:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.803+0000 7f8e7affd640 1 -- 172.21.15.7:0/2227217501 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8e7c0eca40 con 0x7f8e8c0cbb70 2023-11-28T18:44:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:07.807+0000 7f8e89d9b640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e6c066950 0x7f8e6c068e10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8e80006fd0 tx=0x7f8e80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:44:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.211+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e8c1f9f70 con 0x7f8e8c0cbb70 2023-11-28T18:44:08.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.215+0000 7f8e7affd640 1 -- 172.21.15.7:0/2227217501 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6558 (secure 0 0 0) 0x7f8e8c1f9f70 con 0x7f8e8c0cbb70 2023-11-28T18:44:08.219 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:44:08.219 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5714,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:44:04.528599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5528,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5528":{"gid":5528,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":30,"addr":"172.21.15.7:6835/695953635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":695953635},{"type":"v1","addr":"172.21.15.7:6835","nonce":695953635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:44:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 100 2023-11-28T18:44:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e6c066950 msgr2=0x7f8e6c068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e6c066950 0x7f8e6c068e10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8e80006fd0 tx=0x7f8e80008040 comp rx=0 tx=0).stop 2023-11-28T18:44:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e8c0cbb70 msgr2=0x7f8e8c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:44:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e8c0cbb70 0x7f8e8c06de50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c01a0d0 tx=0x7f8e7c019240 comp rx=0 tx=0).stop 2023-11-28T18:44:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 shutdown_connections 2023-11-28T18:44:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e6c066950 0x7f8e6c068e10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e8c1542e0 0x7f8e8c06c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0cc810 0x7f8e8c1474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 --2- 172.21.15.7:0/2227217501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e8c0cbb70 0x7f8e8c06de50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:44:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.219+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 >> 172.21.15.7:0/2227217501 conn(0x7f8e8c0717e0 msgr2=0x7f8e8c147d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:44:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.223+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 shutdown_connections 2023-11-28T18:44:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:44:08.223+0000 7f8e90f2a640 1 -- 172.21.15.7:0/2227217501 wait complete. 2023-11-28T18:44:08.265 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-11-28T18:44:08.266 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2023-11-28T18:44:11.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:44:11.180 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:44:41.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:44:41.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:45:11.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:45:11.325 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:45:16.684 INFO:tasks.workunit.client.0.smithi007.stdout:Syncing()...2 sec 2023-11-28T18:45:16.685 INFO:tasks.workunit.client.0.smithi007.stdout:FFSB benchmark finished at: Tue Nov 28 18:45:16 2023 2023-11-28T18:45:16.686 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.686 INFO:tasks.workunit.client.0.smithi007.stdout:Results: 2023-11-28T18:45:16.686 INFO:tasks.workunit.client.0.smithi007.stdout:Benchmark took 314.05 sec 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout:Total Results 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout:=============== 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout: ======= ============ ========= ======= =========== ========== 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout: write : 1297920 4132.81 100.000% 100.000% 16.1MB/sec 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout:- 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout:4132.81 Transactions per Second 2023-11-28T18:45:16.687 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.688 INFO:tasks.workunit.client.0.smithi007.stdout:Throughput Results 2023-11-28T18:45:16.688 INFO:tasks.workunit.client.0.smithi007.stdout:=================== 2023-11-28T18:45:16.688 INFO:tasks.workunit.client.0.smithi007.stdout:Write Throughput: 16.1MB/sec 2023-11-28T18:45:16.699 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.699 INFO:tasks.workunit.client.0.smithi007.stdout:System Call Latency statistics in millisecs 2023-11-28T18:45:16.699 INFO:tasks.workunit.client.0.smithi007.stdout:===== 2023-11-28T18:45:16.699 INFO:tasks.workunit.client.0.smithi007.stdout: Min Avg Max Total Calls 2023-11-28T18:45:16.699 INFO:tasks.workunit.client.0.smithi007.stdout: ======== ======== ======== ============ 2023-11-28T18:45:16.699 INFO:tasks.workunit.client.0.smithi007.stdout:[ open] 1.019000 1844.624232 40327.246094 1014 2023-11-28T18:45:16.699 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[0] 5000.000000 - 10000.000000 : 5 2023-11-28T18:45:16.699 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[2] 1000.000000 - 2000.000000 : 1 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[3] 500.000000 - 1000.000000 : 15 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[4] 200.000000 - 500.000000 : 6 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[5] 100.000000 - 200.000000 : 11 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[6] 50.000000 - 100.000000 : 40 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[7] 20.000000 - 50.000000 : 438 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[8] 10.000000 - 20.000000 : 348 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[9] 5.000000 - 10.000000 : 62 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[10] 2.000000 - 5.000000 : 23 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[11] 1.000000 - 2.000000 : 5 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2023-11-28T18:45:16.700 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout:[ write] 0.159000 6.010874 22396.039062 1297920 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[2] 1000.000000 - 2000.000000 : 15 2023-11-28T18:45:16.701 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[3] 500.000000 - 1000.000000 : 90 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[4] 200.000000 - 500.000000 : 255 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[5] 100.000000 - 200.000000 : 1250 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[6] 50.000000 - 100.000000 : 3057 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[7] 20.000000 - 50.000000 : 39595 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[8] 10.000000 - 20.000000 : 150879 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[9] 5.000000 - 10.000000 : 349557 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[10] 2.000000 - 5.000000 : 411642 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[11] 1.000000 - 2.000000 : 121947 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[12] 0.500000 - 1.000000 : 79099 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[13] 0.200000 - 0.500000 : 136087 2023-11-28T18:45:16.702 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[14] 0.100000 - 0.200000 : 4446 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout:[ lseek] 0.000000 0.001757 4.849000 1297920 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2023-11-28T18:45:16.703 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2023-11-28T18:45:16.704 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2023-11-28T18:45:16.704 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2023-11-28T18:45:16.704 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2023-11-28T18:45:16.704 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2023-11-28T18:45:16.704 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[10] 2.000000 - 5.000000 : 12 2023-11-28T18:45:16.704 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[11] 1.000000 - 2.000000 : 30 2023-11-28T18:45:16.704 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[12] 0.500000 - 1.000000 : 86 2023-11-28T18:45:16.704 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[13] 0.200000 - 0.500000 : 289 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[14] 0.100000 - 0.200000 : 317 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[15] 0.050000 - 0.100000 : 440 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[16] 0.020000 - 0.050000 : 1319 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[17] 0.010000 - 0.020000 : 1429 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[18] 0.000000 - 0.010000 : 1293998 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout:[ close] 0.037000 2.418923 72.746002 1014 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-11-28T18:45:16.705 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[6] 50.000000 - 100.000000 : 1 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[7] 20.000000 - 50.000000 : 14 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[8] 10.000000 - 20.000000 : 36 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[9] 5.000000 - 10.000000 : 77 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[10] 2.000000 - 5.000000 : 201 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[11] 1.000000 - 2.000000 : 176 2023-11-28T18:45:16.706 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[12] 0.500000 - 1.000000 : 63 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[13] 0.200000 - 0.500000 : 83 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[14] 0.100000 - 0.200000 : 92 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[15] 0.050000 - 0.100000 : 242 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[16] 0.020000 - 0.050000 : 29 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout: 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout:0.9% User Time 2023-11-28T18:45:16.707 INFO:tasks.workunit.client.0.smithi007.stdout:27.5% System Time 2023-11-28T18:45:16.708 INFO:tasks.workunit.client.0.smithi007.stdout:28.4% CPU Utilization 2023-11-28T18:45:16.708 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd .. 2023-11-28T18:45:16.713 INFO:tasks.workunit.client.0.smithi007.stderr:+ rm -r tmp ffsb 2023-11-28T18:45:16.753 INFO:tasks.ceph.mds.a.smithi007.stderr:./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f4e91fb3640 time 2023-11-28T18:45:16.756096+0000 2023-11-28T18:45:16.754 INFO:tasks.ceph.mds.a.smithi007.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T18:45:16.754 INFO:tasks.ceph.mds.a.smithi007.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T18:45:16.754 INFO:tasks.ceph.mds.a.smithi007.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f4e97f6fe96] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 7: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 2023-11-28T18:45:16.755 INFO:tasks.ceph.mds.a.smithi007.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr:*** Caught signal (Aborted) ** 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr: in thread 7f4e91fb3640 thread_name:ms_dispatch 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-11-28T18:45:16.754+0000 7f4e91fb3640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f4e91fb3640 time 2023-11-28T18:45:16.756096+0000 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T18:45:16.756 INFO:tasks.ceph.mds.a.smithi007.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f4e97f6fe96] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 7: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 2023-11-28T18:45:16.757 INFO:tasks.ceph.mds.a.smithi007.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 2023-11-28T18:45:16.758 INFO:tasks.ceph.mds.a.smithi007.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 2023-11-28T18:45:16.758 INFO:tasks.ceph.mds.a.smithi007.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 2023-11-28T18:45:16.758 INFO:tasks.ceph.mds.a.smithi007.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 2023-11-28T18:45:16.758 INFO:tasks.ceph.mds.a.smithi007.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-28T18:45:16.758 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.758 INFO:tasks.ceph.mds.a.smithi007.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4e97642520] 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 2: pthread_kill() 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 3: raise() 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 4: abort() 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f4e97f6fef0] 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 2023-11-28T18:45:16.759 INFO:tasks.ceph.mds.a.smithi007.stderr: 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-28T18:45:16.760 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-11-28T18:45:16.754+0000 7f4e91fb3640 -1 *** Caught signal (Aborted) ** 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: in thread 7f4e91fb3640 thread_name:ms_dispatch 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4e97642520] 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 2: pthread_kill() 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 3: raise() 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 4: abort() 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f4e97f6fef0] 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 2023-11-28T18:45:16.761 INFO:tasks.ceph.mds.a.smithi007.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 2023-11-28T18:45:16.762 INFO:tasks.ceph.mds.a.smithi007.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 2023-11-28T18:45:16.763 INFO:tasks.ceph.mds.a.smithi007.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 2023-11-28T18:45:16.763 INFO:tasks.ceph.mds.a.smithi007.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-28T18:45:16.763 INFO:tasks.ceph.mds.a.smithi007.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T18:45:16.763 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.782 INFO:tasks.ceph.mds.a.smithi007.stderr: -1> 2023-11-28T18:45:16.754+0000 7f4e91fb3640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f4e91fb3640 time 2023-11-28T18:45:16.756096+0000 2023-11-28T18:45:16.782 INFO:tasks.ceph.mds.a.smithi007.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T18:45:16.782 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.782 INFO:tasks.ceph.mds.a.smithi007.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T18:45:16.782 INFO:tasks.ceph.mds.a.smithi007.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f4e97f6fe96] 2023-11-28T18:45:16.782 INFO:tasks.ceph.mds.a.smithi007.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 2023-11-28T18:45:16.782 INFO:tasks.ceph.mds.a.smithi007.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 2023-11-28T18:45:16.782 INFO:tasks.ceph.mds.a.smithi007.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 7: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 2023-11-28T18:45:16.783 INFO:tasks.ceph.mds.a.smithi007.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 2023-11-28T18:45:16.784 INFO:tasks.ceph.mds.a.smithi007.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 2023-11-28T18:45:16.784 INFO:tasks.ceph.mds.a.smithi007.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-28T18:45:16.784 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 0> 2023-11-28T18:45:16.754+0000 7f4e91fb3640 -1 *** Caught signal (Aborted) ** 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: in thread 7f4e91fb3640 thread_name:ms_dispatch 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4e97642520] 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 2: pthread_kill() 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 3: raise() 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 4: abort() 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f4e97f6fef0] 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 2023-11-28T18:45:16.785 INFO:tasks.ceph.mds.a.smithi007.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 2023-11-28T18:45:16.786 INFO:tasks.ceph.mds.a.smithi007.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 2023-11-28T18:45:16.787 INFO:tasks.ceph.mds.a.smithi007.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 2023-11-28T18:45:16.787 INFO:tasks.ceph.mds.a.smithi007.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-28T18:45:16.787 INFO:tasks.ceph.mds.a.smithi007.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T18:45:16.787 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.793 INFO:tasks.ceph.mds.a.smithi007.stderr: -1> 2023-11-28T18:45:16.754+0000 7f4e91fb3640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f4e91fb3640 time 2023-11-28T18:45:16.756096+0000 2023-11-28T18:45:16.793 INFO:tasks.ceph.mds.a.smithi007.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T18:45:16.793 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.794 INFO:tasks.ceph.mds.a.smithi007.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T18:45:16.794 INFO:tasks.ceph.mds.a.smithi007.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f4e97f6fe96] 2023-11-28T18:45:16.794 INFO:tasks.ceph.mds.a.smithi007.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 2023-11-28T18:45:16.794 INFO:tasks.ceph.mds.a.smithi007.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 2023-11-28T18:45:16.794 INFO:tasks.ceph.mds.a.smithi007.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 2023-11-28T18:45:16.794 INFO:tasks.ceph.mds.a.smithi007.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 2023-11-28T18:45:16.794 INFO:tasks.ceph.mds.a.smithi007.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 7: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.795 INFO:tasks.ceph.mds.a.smithi007.stderr: 0> 2023-11-28T18:45:16.754+0000 7f4e91fb3640 -1 *** Caught signal (Aborted) ** 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: in thread 7f4e91fb3640 thread_name:ms_dispatch 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4e97642520] 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 2: pthread_kill() 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 3: raise() 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 4: abort() 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f4e97f6fef0] 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 2023-11-28T18:45:16.796 INFO:tasks.ceph.mds.a.smithi007.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T18:45:16.797 INFO:tasks.ceph.mds.a.smithi007.stderr: 2023-11-28T18:45:17.050 INFO:tasks.ceph.mds.a.smithi007.stderr:daemon-helper: command crashed with signal 6 2023-11-28T18:45:19.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2023-11-28T18:45:26.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2023-11-28T18:45:33.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~14s 2023-11-28T18:45:40.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2023-11-28T18:45:41.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:45:41.396 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:45:44.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:44.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf3d1c4640 1 Processor -- start 2023-11-28T18:45:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf3d1c4640 1 -- start start 2023-11-28T18:45:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf3d1c4640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7faf38151cd0 0x7faf3815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf3d1c4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faf38157b90 0x7faf380701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf3d1c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf38071e60 con 0x7faf3806e560 2023-11-28T18:45:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf3d1c4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf38070730 con 0x7faf38157b90 2023-11-28T18:45:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf3d1c4640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf38070930 con 0x7faf38151cd0 2023-11-28T18:45:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf2ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faf3806e560 0x7faf380c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36962/0 (socket says 172.21.15.7:36962) 2023-11-28T18:45:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf2ffff640 1 -- 172.21.15.7:0/2698726689 learned_addr learned my addr 172.21.15.7:0/2698726689 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:45:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf3759e640 1 --2- 172.21.15.7:0/2698726689 >> v2:172.21.15.7:3300/0 conn(0x7faf38157b90 0x7faf380701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.422+0000 7faf36d9d640 1 --2- 172.21.15.7:0/2698726689 >> v2:172.21.15.193:3300/0 conn(0x7faf38151cd0 0x7faf3815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3759e640 1 -- 172.21.15.7:0/2698726689 >> v2:172.21.15.193:3300/0 conn(0x7faf38151cd0 msgr2=0x7faf3815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3759e640 1 --2- 172.21.15.7:0/2698726689 >> v2:172.21.15.193:3300/0 conn(0x7faf38151cd0 0x7faf3815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3759e640 1 -- 172.21.15.7:0/2698726689 >> v1:172.21.15.7:6789/0 conn(0x7faf3806e560 legacy=0x7faf380c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3759e640 1 -- 172.21.15.7:0/2698726689 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf38157fa0 con 0x7faf38157b90 2023-11-28T18:45:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3759e640 1 --2- 172.21.15.7:0/2698726689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38157b90 0x7faf380701f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7faf3806fee0 tx=0x7faf3001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e8eb5e19c57d59 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:44.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3659c640 1 -- 172.21.15.7:0/2698726689 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf300a5830 con 0x7faf38157b90 2023-11-28T18:45:44.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3659c640 1 -- 172.21.15.7:0/2698726689 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf300a5990 con 0x7faf38157b90 2023-11-28T18:45:44.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3659c640 1 -- 172.21.15.7:0/2698726689 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf300a5d20 con 0x7faf38157b90 2023-11-28T18:45:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3d1c4640 1 -- 172.21.15.7:0/2698726689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38157b90 msgr2=0x7faf380701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/2698726689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38157b90 0x7faf380701f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7faf3806fee0 tx=0x7faf3001c880 comp rx=0 tx=0).stop 2023-11-28T18:45:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3d1c4640 1 -- 172.21.15.7:0/2698726689 shutdown_connections 2023-11-28T18:45:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/2698726689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38157b90 0x7faf380701f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/2698726689 >> v2:172.21.15.193:3300/0 conn(0x7faf38151cd0 0x7faf3815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.426+0000 7faf3d1c4640 1 -- 172.21.15.7:0/2698726689 >> 172.21.15.7:0/2698726689 conn(0x7faf380662e0 msgr2=0x7faf380666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 -- 172.21.15.7:0/2698726689 shutdown_connections 2023-11-28T18:45:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 -- 172.21.15.7:0/2698726689 wait complete. 2023-11-28T18:45:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 Processor -- start 2023-11-28T18:45:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 -- start start 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf3806e560 0x7faf380cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38151cd0 0x7faf38155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf38157b90 0x7faf38148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf380c7b80 con 0x7faf38151cd0 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf381f3010 con 0x7faf3806e560 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3d1c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf381f2460 con 0x7faf38157b90 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38151cd0 0x7faf38155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38151cd0 0x7faf38155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42104/0 (socket says 172.21.15.7:42104) 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 -- 172.21.15.7:0/721036849 learned_addr learned my addr 172.21.15.7:0/721036849 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf36d9d640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf3806e560 0x7faf380cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf3759e640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf38157b90 0x7faf38148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 -- 172.21.15.7:0/721036849 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf38157b90 msgr2=0x7faf38148e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf38157b90 0x7faf38148e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 -- 172.21.15.7:0/721036849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf3806e560 msgr2=0x7faf380cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf3806e560 0x7faf380cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 -- 172.21.15.7:0/721036849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30009a70 con 0x7faf38151cd0 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.430+0000 7faf2ffff640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38151cd0 0x7faf38155c00 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7faf2000c500 tx=0x7faf2000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf2f7fe640 1 -- 172.21.15.7:0/721036849 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf20005cf0 con 0x7faf38151cd0 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf2f7fe640 1 -- 172.21.15.7:0/721036849 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf20005e50 con 0x7faf38151cd0 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf2f7fe640 1 -- 172.21.15.7:0/721036849 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf200072f0 con 0x7faf38151cd0 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf381f3430 con 0x7faf38151cd0 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf381f2670 con 0x7faf38151cd0 2023-11-28T18:45:44.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3804e890 con 0x7faf38151cd0 2023-11-28T18:45:44.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf2f7fe640 1 -- 172.21.15.7:0/721036849 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7faf20007450 con 0x7faf38151cd0 2023-11-28T18:45:44.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf2f7fe640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faf180668d0 0x7faf18068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf2f7fe640 1 -- 172.21.15.7:0/721036849 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7faf20083d70 con 0x7faf38151cd0 2023-11-28T18:45:44.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.434+0000 7faf36d9d640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faf180668d0 0x7faf18068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.438+0000 7faf36d9d640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faf180668d0 0x7faf18068d90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faf28007a60 tx=0x7faf28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:44.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.438+0000 7faf2f7fe640 1 -- 172.21.15.7:0/721036849 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faf20050bd0 con 0x7faf38151cd0 2023-11-28T18:45:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf381555b0 con 0x7faf38151cd0 2023-11-28T18:45:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf2f7fe640 1 -- 172.21.15.7:0/721036849 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5780 (secure 0 0 0) 0x7faf20017020 con 0x7faf38151cd0 2023-11-28T18:45:44.635 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:44.635 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:43.574850+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 104 2023-11-28T18:45:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faf180668d0 msgr2=0x7faf18068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faf180668d0 0x7faf18068d90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faf28007a60 tx=0x7faf28008040 comp rx=0 tx=0).stop 2023-11-28T18:45:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38151cd0 msgr2=0x7faf38155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38151cd0 0x7faf38155c00 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7faf2000c500 tx=0x7faf2000c9c0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 shutdown_connections 2023-11-28T18:45:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faf180668d0 0x7faf18068d90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faf38157b90 0x7faf38148e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf38151cd0 0x7faf38155c00 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 --2- 172.21.15.7:0/721036849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf3806e560 0x7faf380cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.634+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 >> 172.21.15.7:0/721036849 conn(0x7faf380662e0 msgr2=0x7faf381570d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.638+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 shutdown_connections 2023-11-28T18:45:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.638+0000 7faf3d1c4640 1 -- 172.21.15.7:0/721036849 wait complete. 2023-11-28T18:45:44.657 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-11-28T18:45:44.658 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T18:45:44.658 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:45:44.672 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T18:45:44.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:44.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16c1afe640 1 Processor -- start 2023-11-28T18:45:44.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16c1afe640 1 -- start start 2023-11-28T18:45:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16c1afe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f16bc150b30 0x7f16bc0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16c1afe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16bc1575c0 0x7f16bc0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16c1afe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 0x7f16bc0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16c1afe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc0c72b0 con 0x7f16bc1575c0 2023-11-28T18:45:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16c1afe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc070c00 con 0x7f16bc150b30 2023-11-28T18:45:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16c1afe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0c6140 con 0x7f16bc0c4020 2023-11-28T18:45:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16baffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16bc1575c0 0x7f16bc0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.858+0000 7f16baffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16bc1575c0 0x7f16bc0c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42114/0 (socket says 172.21.15.7:42114) 2023-11-28T18:45:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16baffd640 1 -- 172.21.15.7:0/1245317470 learned_addr learned my addr 172.21.15.7:0/1245317470 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16bb7fe640 1 --2- 172.21.15.7:0/1245317470 >> v2:172.21.15.193:3300/0 conn(0x7f16bc150b30 0x7f16bc0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16baffd640 1 -- 172.21.15.7:0/1245317470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 msgr2=0x7f16bc0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16baffd640 1 --2- 172.21.15.7:0/1245317470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 0x7f16bc0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16baffd640 1 -- 172.21.15.7:0/1245317470 >> v2:172.21.15.193:3300/0 conn(0x7f16bc150b30 msgr2=0x7f16bc0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16baffd640 1 --2- 172.21.15.7:0/1245317470 >> v2:172.21.15.193:3300/0 conn(0x7f16bc150b30 0x7f16bc0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16baffd640 1 -- 172.21.15.7:0/1245317470 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc0c6370 con 0x7f16bc1575c0 2023-11-28T18:45:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16bb7fe640 1 --2- 172.21.15.7:0/1245317470 >> v2:172.21.15.193:3300/0 conn(0x7f16bc150b30 0x7f16bc0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:45:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16baffd640 1 --2- 172.21.15.7:0/1245317470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc1575c0 0x7f16bc0c3ae0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f16bc06ec80 tx=0x7f16b001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f1369c0eac4b6a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16ba7fc640 1 -- 172.21.15.7:0/1245317470 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b00a89b0 con 0x7f16bc1575c0 2023-11-28T18:45:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16ba7fc640 1 -- 172.21.15.7:0/1245317470 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b00a8b10 con 0x7f16bc1575c0 2023-11-28T18:45:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.862+0000 7f16ba7fc640 1 -- 172.21.15.7:0/1245317470 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b00a8ea0 con 0x7f16bc1575c0 2023-11-28T18:45:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- 172.21.15.7:0/1245317470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc1575c0 msgr2=0x7f16bc0c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 --2- 172.21.15.7:0/1245317470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc1575c0 0x7f16bc0c3ae0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f16bc06ec80 tx=0x7f16b001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- 172.21.15.7:0/1245317470 shutdown_connections 2023-11-28T18:45:44.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 --2- 172.21.15.7:0/1245317470 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 0x7f16bc0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 --2- 172.21.15.7:0/1245317470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc1575c0 0x7f16bc0c3ae0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 --2- 172.21.15.7:0/1245317470 >> v2:172.21.15.193:3300/0 conn(0x7f16bc150b30 0x7f16bc0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- 172.21.15.7:0/1245317470 >> 172.21.15.7:0/1245317470 conn(0x7f16bc0662e0 msgr2=0x7f16bc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- 172.21.15.7:0/1245317470 shutdown_connections 2023-11-28T18:45:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- 172.21.15.7:0/1245317470 wait complete. 2023-11-28T18:45:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 Processor -- start 2023-11-28T18:45:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- start start 2023-11-28T18:45:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 0x7f16bc1f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc150b30 0x7f16bc1620e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16bc1575c0 0x7f16bc1f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc14f660 con 0x7f16bc150b30 2023-11-28T18:45:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc160d60 con 0x7f16bc1575c0 2023-11-28T18:45:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16c1afe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc161040 con 0x7f16bc0c4020 2023-11-28T18:45:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc150b30 0x7f16bc1620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc150b30 0x7f16bc1620e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42128/0 (socket says 172.21.15.7:42128) 2023-11-28T18:45:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16baffd640 1 -- 172.21.15.7:0/3528837432 learned_addr learned my addr 172.21.15.7:0/3528837432 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16bbfff640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16bc1575c0 0x7f16bc1f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16bb7fe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 0x7f16bc1f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16baffd640 1 -- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 msgr2=0x7f16bc1f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16baffd640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 0x7f16bc1f3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16baffd640 1 -- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16bc1575c0 msgr2=0x7f16bc1f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:44.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16baffd640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16bc1575c0 0x7f16bc1f50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:44.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16baffd640 1 -- 172.21.15.7:0/3528837432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b00094d0 con 0x7f16bc150b30 2023-11-28T18:45:44.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16bb7fe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 0x7f16bc1f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:45:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.866+0000 7f16bbfff640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16bc1575c0 0x7f16bc1f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:45:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16baffd640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc150b30 0x7f16bc1620e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f16b00b3040 tx=0x7f16b001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16a3fff640 1 -- 172.21.15.7:0/3528837432 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b00b9620 con 0x7f16bc150b30 2023-11-28T18:45:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16a3fff640 1 -- 172.21.15.7:0/3528837432 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b00b97a0 con 0x7f16bc150b30 2023-11-28T18:45:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc161320 con 0x7f16bc150b30 2023-11-28T18:45:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16a3fff640 1 -- 172.21.15.7:0/3528837432 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b00b9b50 con 0x7f16bc150b30 2023-11-28T18:45:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc1618e0 con 0x7f16bc150b30 2023-11-28T18:45:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc04e890 con 0x7f16bc150b30 2023-11-28T18:45:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16a3fff640 1 -- 172.21.15.7:0/3528837432 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f16b00b7070 con 0x7f16bc150b30 2023-11-28T18:45:44.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16a3fff640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f16a80668d0 0x7f16a8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:44.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16a3fff640 1 -- 172.21.15.7:0/3528837432 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f16b0122a00 con 0x7f16bc150b30 2023-11-28T18:45:44.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.870+0000 7f16bb7fe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f16a80668d0 0x7f16a8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:44.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.874+0000 7f16bb7fe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f16a80668d0 0x7f16a8068d90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f16ac008930 tx=0x7f16ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:44.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:44.874+0000 7f16a3fff640 1 -- 172.21.15.7:0/3528837432 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f16b00ef690 con 0x7f16bc150b30 2023-11-28T18:45:45.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.062+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16bc0c2d80 con 0x7f16bc150b30 2023-11-28T18:45:45.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16a3fff640 1 -- 172.21.15.7:0/3528837432 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5780 (secure 0 0 0) 0x7f16b00b5090 con 0x7f16bc150b30 2023-11-28T18:45:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:45.065 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:43.574850+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:45.066 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 104 2023-11-28T18:45:45.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f16a80668d0 msgr2=0x7f16a8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:45.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f16a80668d0 0x7f16a8068d90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f16ac008930 tx=0x7f16ac008040 comp rx=0 tx=0).stop 2023-11-28T18:45:45.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc150b30 msgr2=0x7f16bc1620e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:45.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc150b30 0x7f16bc1620e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f16b00b3040 tx=0x7f16b001ba90 comp rx=0 tx=0).stop 2023-11-28T18:45:45.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 shutdown_connections 2023-11-28T18:45:45.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f16a80668d0 0x7f16a8068d90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:45.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16bc1575c0 0x7f16bc1f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:45.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16bc150b30 0x7f16bc1620e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:45.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 --2- 172.21.15.7:0/3528837432 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f16bc0c4020 0x7f16bc1f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:45.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 >> 172.21.15.7:0/3528837432 conn(0x7f16bc0662e0 msgr2=0x7f16bc0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:45.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 shutdown_connections 2023-11-28T18:45:45.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:45.066+0000 7f16c1afe640 1 -- 172.21.15.7:0/3528837432 wait complete. 2023-11-28T18:45:45.087 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:45:46.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~27s 2023-11-28T18:45:47.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:47.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.242+0000 7f60876dd640 1 Processor -- start 2023-11-28T18:45:47.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.242+0000 7f60876dd640 1 -- start start 2023-11-28T18:45:47.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f60876dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 0x7f60800c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:47.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f60876dd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f608015a3a0 0x7f60800c1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:47.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f60876dd640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608006ccc0 con 0x7f60800c7d60 2023-11-28T18:45:47.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f60876dd640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608015a7c0 con 0x7f608015a3a0 2023-11-28T18:45:47.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f60876dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608006cf90 con 0x7f6080150840 2023-11-28T18:45:47.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6085c7a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f60800c7d60 0x7f608006c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36976/0 (socket says 172.21.15.7:36976) 2023-11-28T18:45:47.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6085c7a640 1 -- 172.21.15.7:0/2180825326 learned_addr learned my addr 172.21.15.7:0/2180825326 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:45:47.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6084c78640 1 --2- 172.21.15.7:0/2180825326 >> v2:172.21.15.7:3300/0 conn(0x7f608015a3a0 0x7f60800c1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:47.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6085479640 1 --2- 172.21.15.7:0/2180825326 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 0x7f60800c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:47.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6077fff640 1 -- 172.21.15.7:0/2180825326 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359080307 0 0) 0x7f608006ccc0 con 0x7f60800c7d60 2023-11-28T18:45:47.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6077fff640 1 -- 172.21.15.7:0/2180825326 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f606c003530 con 0x7f60800c7d60 2023-11-28T18:45:47.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6084c78640 1 -- 172.21.15.7:0/2180825326 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 msgr2=0x7f60800c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:47.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6084c78640 1 --2- 172.21.15.7:0/2180825326 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 0x7f60800c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6084c78640 1 -- 172.21.15.7:0/2180825326 >> v1:172.21.15.7:6789/0 conn(0x7f60800c7d60 legacy=0x7f608006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:47.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.246+0000 7f6084c78640 1 -- 172.21.15.7:0/2180825326 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608006d270 con 0x7f608015a3a0 2023-11-28T18:45:47.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f6084c78640 1 --2- 172.21.15.7:0/2180825326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f60800c1700 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f60700097c0 tx=0x7f607001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43cea41257b1ef5a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:47.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f6077fff640 1 -- 172.21.15.7:0/2180825326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60700a8970 con 0x7f608015a3a0 2023-11-28T18:45:47.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f6077fff640 1 -- 172.21.15.7:0/2180825326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60700a8ad0 con 0x7f608015a3a0 2023-11-28T18:45:47.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f6077fff640 1 -- 172.21.15.7:0/2180825326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60700a8e60 con 0x7f608015a3a0 2023-11-28T18:45:47.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 -- 172.21.15.7:0/2180825326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 msgr2=0x7f60800c1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:47.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 --2- 172.21.15.7:0/2180825326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f60800c1700 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f60700097c0 tx=0x7f607001ec70 comp rx=0 tx=0).stop 2023-11-28T18:45:47.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 -- 172.21.15.7:0/2180825326 shutdown_connections 2023-11-28T18:45:47.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 --2- 172.21.15.7:0/2180825326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f60800c1700 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 --2- 172.21.15.7:0/2180825326 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 0x7f60800c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 -- 172.21.15.7:0/2180825326 >> 172.21.15.7:0/2180825326 conn(0x7f60800cbf80 msgr2=0x7f60800cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:47.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 -- 172.21.15.7:0/2180825326 shutdown_connections 2023-11-28T18:45:47.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 -- 172.21.15.7:0/2180825326 wait complete. 2023-11-28T18:45:47.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 Processor -- start 2023-11-28T18:45:47.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.250+0000 7f60876dd640 1 -- start start 2023-11-28T18:45:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60800c7d60 0x7f60801e1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 0x7f60801e4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f6080158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60801f31e0 con 0x7f608015a3a0 2023-11-28T18:45:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60801e1690 con 0x7f60800c7d60 2023-11-28T18:45:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60801541e0 con 0x7f6080150840 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f6080158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f6080158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42158/0 (socket says 172.21.15.7:42158) 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 -- 172.21.15.7:0/2179502454 learned_addr learned my addr 172.21.15.7:0/2179502454 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085479640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60800c7d60 0x7f60801e1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6084c78640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 0x7f60801e4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 -- 172.21.15.7:0/2179502454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 msgr2=0x7f60801e4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 0x7f60801e4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 -- 172.21.15.7:0/2179502454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60800c7d60 msgr2=0x7f60801e1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60800c7d60 0x7f60801e1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 -- 172.21.15.7:0/2179502454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6070009420 con 0x7f608015a3a0 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6085c7a640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f6080158860 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f607800cb40 tx=0x7f607800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:47.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6075ffb640 1 -- 172.21.15.7:0/2179502454 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607800aca0 con 0x7f608015a3a0 2023-11-28T18:45:47.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6075ffb640 1 -- 172.21.15.7:0/2179502454 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607800ae00 con 0x7f608015a3a0 2023-11-28T18:45:47.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6075ffb640 1 -- 172.21.15.7:0/2179502454 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078012620 con 0x7f608015a3a0 2023-11-28T18:45:47.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60801544a0 con 0x7f608015a3a0 2023-11-28T18:45:47.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60801f2b20 con 0x7f608015a3a0 2023-11-28T18:45:47.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60801f3320 con 0x7f608015a3a0 2023-11-28T18:45:47.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.254+0000 7f6075ffb640 1 -- 172.21.15.7:0/2179502454 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6078007070 con 0x7f608015a3a0 2023-11-28T18:45:47.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.258+0000 7f6075ffb640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f60640668d0 0x7f6064068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:47.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.258+0000 7f6075ffb640 1 -- 172.21.15.7:0/2179502454 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f60780849d0 con 0x7f608015a3a0 2023-11-28T18:45:47.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.258+0000 7f6085479640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f60640668d0 0x7f6064068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:47.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.258+0000 7f6085479640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f60640668d0 0x7f6064068d90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f607c009af0 tx=0x7f607c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:47.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.262+0000 7f6075ffb640 1 -- 172.21.15.7:0/2179502454 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f60801f3320 con 0x7f608015a3a0 2023-11-28T18:45:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.454+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60801f3320 con 0x7f608015a3a0 2023-11-28T18:45:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.454+0000 7f6075ffb640 1 -- 172.21.15.7:0/2179502454 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5780 (secure 0 0 0) 0x7f60801f3320 con 0x7f608015a3a0 2023-11-28T18:45:47.456 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:47.456 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:43.574850+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 104 2023-11-28T18:45:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f60640668d0 msgr2=0x7f6064068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f60640668d0 0x7f6064068d90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f607c009af0 tx=0x7f607c008040 comp rx=0 tx=0).stop 2023-11-28T18:45:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 msgr2=0x7f6080158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f6080158860 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f607800cb40 tx=0x7f607800a060 comp rx=0 tx=0).stop 2023-11-28T18:45:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 shutdown_connections 2023-11-28T18:45:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f60640668d0 0x7f6064068d90 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f608015a3a0 0x7f6080158860 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6080150840 0x7f60801e4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 --2- 172.21.15.7:0/2179502454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60800c7d60 0x7f60801e1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 >> 172.21.15.7:0/2179502454 conn(0x7f60800cbf80 msgr2=0x7f6080159990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 shutdown_connections 2023-11-28T18:45:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:47.458+0000 7f60876dd640 1 -- 172.21.15.7:0/2179502454 wait complete. 2023-11-28T18:45:47.476 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:45:49.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:49.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8d13f8640 1 Processor -- start 2023-11-28T18:45:49.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8d13f8640 1 -- start start 2023-11-28T18:45:49.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8d13f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc150b30 0x7fa8cc0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8d13f8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa8cc1575c0 0x7fa8cc0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8d13f8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa8cc0c4020 0x7fa8cc0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8d13f8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8cc0c72b0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8d13f8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8cc070c00 con 0x7fa8cc0c4020 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8d13f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc0c6140 con 0x7fa8cc150b30 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8c3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa8cc1575c0 0x7fa8cc0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8c3fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa8cc1575c0 0x7fa8cc0c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42170/0 (socket says 172.21.15.7:42170) 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8c3fff640 1 -- 172.21.15.7:0/1310285708 learned_addr learned my addr 172.21.15.7:0/1310285708 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8cb7fe640 1 --2- 172.21.15.7:0/1310285708 >> v2:172.21.15.193:3300/0 conn(0x7fa8cc0c4020 0x7fa8cc0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8caffd640 1 --2- 172.21.15.7:0/1310285708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc150b30 0x7fa8cc0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8c3fff640 1 -- 172.21.15.7:0/1310285708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc150b30 msgr2=0x7fa8cc0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8c3fff640 1 --2- 172.21.15.7:0/1310285708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc150b30 0x7fa8cc0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8c3fff640 1 -- 172.21.15.7:0/1310285708 >> v2:172.21.15.193:3300/0 conn(0x7fa8cc0c4020 msgr2=0x7fa8cc0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8c3fff640 1 --2- 172.21.15.7:0/1310285708 >> v2:172.21.15.193:3300/0 conn(0x7fa8cc0c4020 0x7fa8cc0c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8c3fff640 1 -- 172.21.15.7:0/1310285708 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc0c6370 con 0x7fa8cc1575c0 2023-11-28T18:45:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.634+0000 7fa8caffd640 1 --2- 172.21.15.7:0/1310285708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc150b30 0x7fa8cc0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:45:49.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8c3fff640 1 --2- 172.21.15.7:0/1310285708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc0c3ae0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0c95f0 tx=0x7fa8b401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb34644a6c86396 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:49.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8ca7fc640 1 -- 172.21.15.7:0/1310285708 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40a89b0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8ca7fc640 1 -- 172.21.15.7:0/1310285708 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b40a8b10 con 0x7fa8cc1575c0 2023-11-28T18:45:49.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8ca7fc640 1 -- 172.21.15.7:0/1310285708 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40a8ea0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 -- 172.21.15.7:0/1310285708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 msgr2=0x7fa8cc0c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/1310285708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc0c3ae0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0c95f0 tx=0x7fa8b401ecf0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 -- 172.21.15.7:0/1310285708 shutdown_connections 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/1310285708 >> v2:172.21.15.193:3300/0 conn(0x7fa8cc0c4020 0x7fa8cc0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/1310285708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc0c3ae0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/1310285708 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc150b30 0x7fa8cc0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 -- 172.21.15.7:0/1310285708 >> 172.21.15.7:0/1310285708 conn(0x7fa8cc0662e0 msgr2=0x7fa8cc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 -- 172.21.15.7:0/1310285708 shutdown_connections 2023-11-28T18:45:49.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.638+0000 7fa8d13f8640 1 -- 172.21.15.7:0/1310285708 wait complete. 2023-11-28T18:45:49.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 Processor -- start 2023-11-28T18:45:49.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 -- start start 2023-11-28T18:45:49.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc0c4020 0x7fa8cc1ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:49.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8cc150b30 0x7fa8cc14bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:49.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:49.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc14c7e0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc14aff0 con 0x7fa8cc150b30 2023-11-28T18:45:49.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc153df0 con 0x7fa8cc0c4020 2023-11-28T18:45:49.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8cb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:49.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8c3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8cc150b30 0x7fa8cc14bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:49.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8caffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc0c4020 0x7fa8cc1ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:49.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8c3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8cc150b30 0x7fa8cc14bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:59716/0 (socket says 172.21.15.7:59716) 2023-11-28T18:45:49.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8cb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42180/0 (socket says 172.21.15.7:42180) 2023-11-28T18:45:49.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8c3fff640 1 -- 172.21.15.7:0/3732365321 learned_addr learned my addr 172.21.15.7:0/3732365321 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8cb7fe640 1 -- 172.21.15.7:0/3732365321 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc0c4020 msgr2=0x7fa8cc1ee910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8cb7fe640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc0c4020 0x7fa8cc1ee910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8cb7fe640 1 -- 172.21.15.7:0/3732365321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8cc150b30 msgr2=0x7fa8cc14bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8cb7fe640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8cc150b30 0x7fa8cc14bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8cb7fe640 1 -- 172.21.15.7:0/3732365321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b40094d0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8cb7fe640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc155c60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4007d40 tx=0x7fa8c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8c37fe640 1 -- 172.21.15.7:0/3732365321 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40039a0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8c37fe640 1 -- 172.21.15.7:0/3732365321 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c4003b20 con 0x7fa8cc1575c0 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8c37fe640 1 -- 172.21.15.7:0/3732365321 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4004030 con 0x7fa8cc1575c0 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc1540d0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc14b4e0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.642+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc04e890 con 0x7fa8cc1575c0 2023-11-28T18:45:49.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.646+0000 7fa8c37fe640 1 -- 172.21.15.7:0/3732365321 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa8c4004190 con 0x7fa8cc1575c0 2023-11-28T18:45:49.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.646+0000 7fa8c37fe640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8ac0668d0 0x7fa8ac068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:49.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.646+0000 7fa8c37fe640 1 -- 172.21.15.7:0/3732365321 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fa8c4084790 con 0x7fa8cc1575c0 2023-11-28T18:45:49.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.646+0000 7fa8caffd640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8ac0668d0 0x7fa8ac068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:49.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.646+0000 7fa8caffd640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8ac0668d0 0x7fa8ac068d90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00ab60 tx=0x7fa8bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:49.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.650+0000 7fa8c37fe640 1 -- 172.21.15.7:0/3732365321 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa8c40515f0 con 0x7fa8cc1575c0 2023-11-28T18:45:49.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.838+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8cc154790 con 0x7fa8cc1575c0 2023-11-28T18:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.838+0000 7fa8c37fe640 1 -- 172.21.15.7:0/3732365321 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fa8c4054c10 con 0x7fa8cc1575c0 2023-11-28T18:45:49.840 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:49.841 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:48.589265+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 105 2023-11-28T18:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8ac0668d0 msgr2=0x7fa8ac068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8ac0668d0 0x7fa8ac068d90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00ab60 tx=0x7fa8bc008040 comp rx=0 tx=0).stop 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 msgr2=0x7fa8cc155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc155c60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4007d40 tx=0x7fa8c400e4b0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 shutdown_connections 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa8ac0668d0 0x7fa8ac068d90 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8cc1575c0 0x7fa8cc155c60 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8cc150b30 0x7fa8cc14bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 --2- 172.21.15.7:0/3732365321 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa8cc0c4020 0x7fa8cc1ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 >> 172.21.15.7:0/3732365321 conn(0x7fa8cc0662e0 msgr2=0x7fa8cc06d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 shutdown_connections 2023-11-28T18:45:49.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:49.842+0000 7fa8d13f8640 1 -- 172.21.15.7:0/3732365321 wait complete. 2023-11-28T18:45:49.863 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:45:51.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:52.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f402b89a640 1 Processor -- start 2023-11-28T18:45:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f402b89a640 1 -- start start 2023-11-28T18:45:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f402b89a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4024150840 0x7f40240c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f402b89a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40240c1700 0x7f402406c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f402b89a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40240c1b50 con 0x7f402415a3a0 2023-11-28T18:45:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f402b89a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f402406ccc0 con 0x7f40240c1700 2023-11-28T18:45:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f402b89a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f402415a7c0 con 0x7f4024150840 2023-11-28T18:45:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f4029e37640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40240c1700 0x7f402406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f4028e35640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f402415a3a0 0x7f40240c7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36992/0 (socket says 172.21.15.7:36992) 2023-11-28T18:45:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.018+0000 7f4028e35640 1 -- 172.21.15.7:0/2999599837 learned_addr learned my addr 172.21.15.7:0/2999599837 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:45:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f4029636640 1 --2- 172.21.15.7:0/2999599837 >> v2:172.21.15.193:3300/0 conn(0x7f4024150840 0x7f40240c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f4029e37640 1 -- 172.21.15.7:0/2999599837 >> v2:172.21.15.193:3300/0 conn(0x7f4024150840 msgr2=0x7f40240c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:52.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f4029e37640 1 --2- 172.21.15.7:0/2999599837 >> v2:172.21.15.193:3300/0 conn(0x7f4024150840 0x7f40240c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f4029e37640 1 -- 172.21.15.7:0/2999599837 >> v1:172.21.15.7:6789/0 conn(0x7f402415a3a0 legacy=0x7f40240c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:52.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f4029e37640 1 -- 172.21.15.7:0/2999599837 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402406cf90 con 0x7f40240c1700 2023-11-28T18:45:52.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f4029e37640 1 --2- 172.21.15.7:0/2999599837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f402406c080 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f402406f480 tx=0x7f401c021310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae4b77e10055151 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:52.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f401bfff640 1 -- 172.21.15.7:0/2999599837 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c0b1600 con 0x7f40240c1700 2023-11-28T18:45:52.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f401bfff640 1 -- 172.21.15.7:0/2999599837 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c0ad050 con 0x7f40240c1700 2023-11-28T18:45:52.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f401bfff640 1 -- 172.21.15.7:0/2999599837 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c0b1980 con 0x7f40240c1700 2023-11-28T18:45:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f402b89a640 1 -- 172.21.15.7:0/2999599837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 msgr2=0x7f402406c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.022+0000 7f402b89a640 1 --2- 172.21.15.7:0/2999599837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f402406c080 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f402406f480 tx=0x7f401c021310 comp rx=0 tx=0).stop 2023-11-28T18:45:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 -- 172.21.15.7:0/2999599837 shutdown_connections 2023-11-28T18:45:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 --2- 172.21.15.7:0/2999599837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f402406c080 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 --2- 172.21.15.7:0/2999599837 >> v2:172.21.15.193:3300/0 conn(0x7f4024150840 0x7f40240c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 -- 172.21.15.7:0/2999599837 >> 172.21.15.7:0/2999599837 conn(0x7f40240cbf80 msgr2=0x7f40240cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 -- 172.21.15.7:0/2999599837 shutdown_connections 2023-11-28T18:45:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 -- 172.21.15.7:0/2999599837 wait complete. 2023-11-28T18:45:52.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 Processor -- start 2023-11-28T18:45:52.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 -- start start 2023-11-28T18:45:52.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f40240cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4024150840 0x7f40240c3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f402415a3a0 0x7f40240c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4024151d40 con 0x7f40240c1700 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40240c53c0 con 0x7f4024150840 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f402b89a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40240c5bf0 con 0x7f402415a3a0 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f40240cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f40240cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42194/0 (socket says 172.21.15.7:42194) 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 -- 172.21.15.7:0/55272343 learned_addr learned my addr 172.21.15.7:0/55272343 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4028e35640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4024150840 0x7f40240c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 -- 172.21.15.7:0/55272343 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f402415a3a0 msgr2=0x7f40240c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f402415a3a0 0x7f40240c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 -- 172.21.15.7:0/55272343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4024150840 msgr2=0x7f40240c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4024150840 0x7f40240c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 -- 172.21.15.7:0/55272343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c009a70 con 0x7f40240c1700 2023-11-28T18:45:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4028e35640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4024150840 0x7f40240c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:45:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.026+0000 7f4029636640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f40240cc5c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f402000d800 tx=0x7f402000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f4019ffb640 1 -- 172.21.15.7:0/55272343 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402000b6c0 con 0x7f40240c1700 2023-11-28T18:45:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f4019ffb640 1 -- 172.21.15.7:0/55272343 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402000b820 con 0x7f40240c1700 2023-11-28T18:45:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40240c5ed0 con 0x7f40240c1700 2023-11-28T18:45:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f4019ffb640 1 -- 172.21.15.7:0/55272343 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402000bb90 con 0x7f40240c1700 2023-11-28T18:45:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40241569c0 con 0x7f40240c1700 2023-11-28T18:45:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f402404e890 con 0x7f40240c1700 2023-11-28T18:45:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f4019ffb640 1 -- 172.21.15.7:0/55272343 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f402000bcf0 con 0x7f40240c1700 2023-11-28T18:45:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f4019ffb640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f40080668d0 0x7f4008068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f4019ffb640 1 -- 172.21.15.7:0/55272343 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f4020084ef0 con 0x7f40240c1700 2023-11-28T18:45:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.030+0000 7f4028e35640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f40080668d0 0x7f4008068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.034+0000 7f4028e35640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f40080668d0 0x7f4008068d90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f40240ca900 tx=0x7f4014009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.034+0000 7f4019ffb640 1 -- 172.21.15.7:0/55272343 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4020051c10 con 0x7f40240c1700 2023-11-28T18:45:52.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40240c7b40 con 0x7f40240c1700 2023-11-28T18:45:52.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f4019ffb640 1 -- 172.21.15.7:0/55272343 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f4020011340 con 0x7f40240c1700 2023-11-28T18:45:52.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:52.226 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:48.589265+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 105 2023-11-28T18:45:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f40080668d0 msgr2=0x7f4008068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f40080668d0 0x7f4008068d90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f40240ca900 tx=0x7f4014009040 comp rx=0 tx=0).stop 2023-11-28T18:45:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 msgr2=0x7f40240cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f40240cc5c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f402000d800 tx=0x7f402000dcc0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 shutdown_connections 2023-11-28T18:45:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f40080668d0 0x7f4008068d90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f402415a3a0 0x7f40240c3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4024150840 0x7f40240c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 --2- 172.21.15.7:0/55272343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40240c1700 0x7f40240cc5c0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 >> 172.21.15.7:0/55272343 conn(0x7f40240cbf80 msgr2=0x7f4024072060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 shutdown_connections 2023-11-28T18:45:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:52.226+0000 7f402b89a640 1 -- 172.21.15.7:0/55272343 wait complete. 2023-11-28T18:45:52.248 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:45:53.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~34s 2023-11-28T18:45:54.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:54.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d71c4640 1 Processor -- start 2023-11-28T18:45:54.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d71c4640 1 -- start start 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d71c4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22d0151cd0 0x7f22d015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d71c4640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f22d0157b90 0x7f22d00701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d71c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d0071e60 con 0x7f22d006e560 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d71c4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d0070730 con 0x7f22d0151cd0 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d71c4640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d0070930 con 0x7f22d0157b90 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d4f60640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22d0151cd0 0x7f22d015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22cffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f22d006e560 0x7f22d00c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57872/0 (socket says 172.21.15.7:57872) 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22cffff640 1 -- 172.21.15.7:0/3823603426 learned_addr learned my addr 172.21.15.7:0/3823603426 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d5761640 1 --2- 172.21.15.7:0/3823603426 >> v2:172.21.15.193:3300/0 conn(0x7f22d0157b90 0x7f22d00701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d4f60640 1 -- 172.21.15.7:0/3823603426 >> v2:172.21.15.193:3300/0 conn(0x7f22d0157b90 msgr2=0x7f22d00701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d4f60640 1 --2- 172.21.15.7:0/3823603426 >> v2:172.21.15.193:3300/0 conn(0x7f22d0157b90 0x7f22d00701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d4f60640 1 -- 172.21.15.7:0/3823603426 >> v1:172.21.15.7:6789/0 conn(0x7f22d006e560 legacy=0x7f22d00c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:54.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d4f60640 1 -- 172.21.15.7:0/3823603426 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d0157fa0 con 0x7f22d0151cd0 2023-11-28T18:45:54.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22cf7fe640 1 -- 172.21.15.7:0/3823603426 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557947609 0 0) 0x7f22d0071e60 con 0x7f22d006e560 2023-11-28T18:45:54.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.406+0000 7f22d5761640 1 --2- 172.21.15.7:0/3823603426 >> v2:172.21.15.193:3300/0 conn(0x7f22d0157b90 0x7f22d00701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:45:54.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d4f60640 1 --2- 172.21.15.7:0/3823603426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d015bd00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f22d00c5d70 tx=0x7f22c401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba985fcbe94a3f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:54.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22cf7fe640 1 -- 172.21.15.7:0/3823603426 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c40ab600 con 0x7f22d0151cd0 2023-11-28T18:45:54.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22cf7fe640 1 -- 172.21.15.7:0/3823603426 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c40a7050 con 0x7f22d0151cd0 2023-11-28T18:45:54.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22cf7fe640 1 -- 172.21.15.7:0/3823603426 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c40ab980 con 0x7f22d0151cd0 2023-11-28T18:45:54.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d71c4640 1 -- 172.21.15.7:0/3823603426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 msgr2=0x7f22d015bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:54.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d71c4640 1 --2- 172.21.15.7:0/3823603426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d015bd00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f22d00c5d70 tx=0x7f22c401c3d0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d71c4640 1 -- 172.21.15.7:0/3823603426 shutdown_connections 2023-11-28T18:45:54.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d71c4640 1 --2- 172.21.15.7:0/3823603426 >> v2:172.21.15.193:3300/0 conn(0x7f22d0157b90 0x7f22d00701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d71c4640 1 --2- 172.21.15.7:0/3823603426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d015bd00 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d71c4640 1 -- 172.21.15.7:0/3823603426 >> 172.21.15.7:0/3823603426 conn(0x7f22d00662e0 msgr2=0x7f22d00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:54.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d71c4640 1 -- 172.21.15.7:0/3823603426 shutdown_connections 2023-11-28T18:45:54.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.410+0000 7f22d71c4640 1 -- 172.21.15.7:0/3823603426 wait complete. 2023-11-28T18:45:54.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 Processor -- start 2023-11-28T18:45:54.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 -- start start 2023-11-28T18:45:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d006e560 0x7f22d0145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d0146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d0157b90 0x7f22d01570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d00c7dc0 con 0x7f22d0151cd0 2023-11-28T18:45:54.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d0156050 con 0x7f22d006e560 2023-11-28T18:45:54.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d0156310 con 0x7f22d0157b90 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d0146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d0146c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48730/0 (socket says 172.21.15.7:48730) 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 -- 172.21.15.7:0/1248759649 learned_addr learned my addr 172.21.15.7:0/1248759649 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d4f60640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d006e560 0x7f22d0145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d5761640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d0157b90 0x7f22d01570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 -- 172.21.15.7:0/1248759649 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d0157b90 msgr2=0x7f22d01570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d0157b90 0x7f22d01570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 -- 172.21.15.7:0/1248759649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d006e560 msgr2=0x7f22d0145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d006e560 0x7f22d0145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 -- 172.21.15.7:0/1248759649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c4009420 con 0x7f22d0151cd0 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cffff640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d0146c00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0011f0 tx=0x7f22bc00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cd7fa640 1 -- 172.21.15.7:0/1248759649 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc0074e0 con 0x7f22d0151cd0 2023-11-28T18:45:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cd7fa640 1 -- 172.21.15.7:0/1248759649 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22bc007640 con 0x7f22d0151cd0 2023-11-28T18:45:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22cd7fa640 1 -- 172.21.15.7:0/1248759649 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc004a10 con 0x7f22d0151cd0 2023-11-28T18:45:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d01565f0 con 0x7f22d0151cd0 2023-11-28T18:45:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d0156bb0 con 0x7f22d0151cd0 2023-11-28T18:45:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.414+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d004e890 con 0x7f22d0151cd0 2023-11-28T18:45:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.418+0000 7f22cd7fa640 1 -- 172.21.15.7:0/1248759649 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f22bc004b70 con 0x7f22d0151cd0 2023-11-28T18:45:54.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.418+0000 7f22cd7fa640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b40668d0 0x7f22b4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:54.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.418+0000 7f22cd7fa640 1 -- 172.21.15.7:0/1248759649 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f22bc083b00 con 0x7f22d0151cd0 2023-11-28T18:45:54.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.418+0000 7f22d4f60640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b40668d0 0x7f22b4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:54.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.418+0000 7f22d4f60640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b40668d0 0x7f22b4068d90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f22c40ad000 tx=0x7f22c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:54.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.422+0000 7f22cd7fa640 1 -- 172.21.15.7:0/1248759649 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f22bc085e60 con 0x7f22d0151cd0 2023-11-28T18:45:54.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.618+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22d0142070 con 0x7f22d0151cd0 2023-11-28T18:45:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.618+0000 7f22cd7fa640 1 -- 172.21.15.7:0/1248759649 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f22bc018020 con 0x7f22d0151cd0 2023-11-28T18:45:54.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:54.620 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:48.589265+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 105 2023-11-28T18:45:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b40668d0 msgr2=0x7f22b4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b40668d0 0x7f22b4068d90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f22c40ad000 tx=0x7f22c4008040 comp rx=0 tx=0).stop 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 msgr2=0x7f22d0146c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d0146c00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0011f0 tx=0x7f22bc00c9b0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 shutdown_connections 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b40668d0 0x7f22b4068d90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d0157b90 0x7f22d01570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d0151cd0 0x7f22d0146c00 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 --2- 172.21.15.7:0/1248759649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d006e560 0x7f22d0145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 >> 172.21.15.7:0/1248759649 conn(0x7f22d00662e0 msgr2=0x7f22d00caa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 shutdown_connections 2023-11-28T18:45:54.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:54.622+0000 7f22d71c4640 1 -- 172.21.15.7:0/1248759649 wait complete. 2023-11-28T18:45:54.642 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:45:56.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44c53df640 1 Processor -- start 2023-11-28T18:45:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44c53df640 1 -- start start 2023-11-28T18:45:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44c53df640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f44c0072950 0x7f44c0074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44c53df640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44c0073070 0x7f44c00c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44c53df640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c00c8490 con 0x7f44c00c1ef0 2023-11-28T18:45:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44c53df640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c00c81f0 con 0x7f44c0073070 2023-11-28T18:45:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44c53df640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c015ab00 con 0x7f44c0072950 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44b67fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44c0073070 0x7f44c00c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44bf7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f44c00c1ef0 0x7f44c006c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57882/0 (socket says 172.21.15.7:57882) 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44bf7fe640 1 -- 172.21.15.7:0/2507545690 learned_addr learned my addr 172.21.15.7:0/2507545690 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44beffd640 1 --2- 172.21.15.7:0/2507545690 >> v2:172.21.15.193:3300/0 conn(0x7f44c0072950 0x7f44c0074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44b67fc640 1 -- 172.21.15.7:0/2507545690 >> v2:172.21.15.193:3300/0 conn(0x7f44c0072950 msgr2=0x7f44c0074030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44b67fc640 1 --2- 172.21.15.7:0/2507545690 >> v2:172.21.15.193:3300/0 conn(0x7f44c0072950 0x7f44c0074030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44b67fc640 1 -- 172.21.15.7:0/2507545690 >> v1:172.21.15.7:6789/0 conn(0x7f44c00c1ef0 legacy=0x7f44c006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44b67fc640 1 -- 172.21.15.7:0/2507545690 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c015ade0 con 0x7f44c0073070 2023-11-28T18:45:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.798+0000 7f44be7fc640 1 -- 172.21.15.7:0/2507545690 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714700522 0 0) 0x7f44c00c8490 con 0x7f44c00c1ef0 2023-11-28T18:45:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44b67fc640 1 --2- 172.21.15.7:0/2507545690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0073070 0x7f44c00c19b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f44c00c9b10 tx=0x7f44a801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aada91cf3bd05fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44be7fc640 1 -- 172.21.15.7:0/2507545690 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a80a5720 con 0x7f44c0073070 2023-11-28T18:45:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44be7fc640 1 -- 172.21.15.7:0/2507545690 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a80a5880 con 0x7f44c0073070 2023-11-28T18:45:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44be7fc640 1 -- 172.21.15.7:0/2507545690 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a80a5c10 con 0x7f44c0073070 2023-11-28T18:45:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44c53df640 1 -- 172.21.15.7:0/2507545690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0073070 msgr2=0x7f44c00c19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44c53df640 1 --2- 172.21.15.7:0/2507545690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0073070 0x7f44c00c19b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f44c00c9b10 tx=0x7f44a801c880 comp rx=0 tx=0).stop 2023-11-28T18:45:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44c53df640 1 -- 172.21.15.7:0/2507545690 shutdown_connections 2023-11-28T18:45:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44c53df640 1 --2- 172.21.15.7:0/2507545690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0073070 0x7f44c00c19b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44c53df640 1 --2- 172.21.15.7:0/2507545690 >> v2:172.21.15.193:3300/0 conn(0x7f44c0072950 0x7f44c0074030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44c53df640 1 -- 172.21.15.7:0/2507545690 >> 172.21.15.7:0/2507545690 conn(0x7f44c00cbf80 msgr2=0x7f44c00cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44c53df640 1 -- 172.21.15.7:0/2507545690 shutdown_connections 2023-11-28T18:45:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.802+0000 7f44c53df640 1 -- 172.21.15.7:0/2507545690 wait complete. 2023-11-28T18:45:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 Processor -- start 2023-11-28T18:45:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 -- start start 2023-11-28T18:45:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0072950 0x7f44c0152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f44c0073070 0x7f44c01f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c00c1ef0 0x7f44c01f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c015b4b0 con 0x7f44c0072950 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c015b880 con 0x7f44c00c1ef0 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0158ce0 con 0x7f44c0073070 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0072950 0x7f44c0152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0072950 0x7f44c0152b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48744/0 (socket says 172.21.15.7:48744) 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 -- 172.21.15.7:0/1025605938 learned_addr learned my addr 172.21.15.7:0/1025605938 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44b67fc640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f44c0073070 0x7f44c01f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44bf7fe640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c00c1ef0 0x7f44c01f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 -- 172.21.15.7:0/1025605938 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f44c0073070 msgr2=0x7f44c01f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f44c0073070 0x7f44c01f41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 -- 172.21.15.7:0/1025605938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c00c1ef0 msgr2=0x7f44c01f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c00c1ef0 0x7f44c01f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 -- 172.21.15.7:0/1025605938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a8009a70 con 0x7f44c0072950 2023-11-28T18:45:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44beffd640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0072950 0x7f44c0152b10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f44b000e930 tx=0x7f44b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44b7fff640 1 -- 172.21.15.7:0/1025605938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0009690 con 0x7f44c0072950 2023-11-28T18:45:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44b7fff640 1 -- 172.21.15.7:0/1025605938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b000ca00 con 0x7f44c0072950 2023-11-28T18:45:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44b7fff640 1 -- 172.21.15.7:0/1025605938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b000cd70 con 0x7f44c0072950 2023-11-28T18:45:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c0158780 con 0x7f44c0072950 2023-11-28T18:45:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c01f3710 con 0x7f44c0072950 2023-11-28T18:45:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.806+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c004e890 con 0x7f44c0072950 2023-11-28T18:45:56.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.810+0000 7f44b7fff640 1 -- 172.21.15.7:0/1025605938 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f44b0013470 con 0x7f44c0072950 2023-11-28T18:45:56.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.810+0000 7f44b7fff640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f44a0066900 0x7f44a0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:56.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.810+0000 7f44b7fff640 1 -- 172.21.15.7:0/1025605938 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f44b0084e60 con 0x7f44c0072950 2023-11-28T18:45:56.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.810+0000 7f44b67fc640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f44a0066900 0x7f44a0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:56.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.810+0000 7f44b67fc640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f44a0066900 0x7f44a0068dc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f44a80af000 tx=0x7f44a80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:56.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:56.814+0000 7f44b7fff640 1 -- 172.21.15.7:0/1025605938 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f44b0051cd0 con 0x7f44c0072950 2023-11-28T18:45:57.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.002+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44c006b600 con 0x7f44c0072950 2023-11-28T18:45:57.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44b7fff640 1 -- 172.21.15.7:0/1025605938 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5775 (secure 0 0 0) 0x7f44b0017090 con 0x7f44c0072950 2023-11-28T18:45:57.005 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:57.005 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:54.534142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:57.006 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 106 2023-11-28T18:45:57.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f44a0066900 msgr2=0x7f44a0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:57.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f44a0066900 0x7f44a0068dc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f44a80af000 tx=0x7f44a80a7070 comp rx=0 tx=0).stop 2023-11-28T18:45:57.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0072950 msgr2=0x7f44c0152b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:57.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0072950 0x7f44c0152b10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f44b000e930 tx=0x7f44b000edf0 comp rx=0 tx=0).stop 2023-11-28T18:45:57.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 shutdown_connections 2023-11-28T18:45:57.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f44a0066900 0x7f44a0068dc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:57.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c00c1ef0 0x7f44c01f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:57.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f44c0073070 0x7f44c01f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:57.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 --2- 172.21.15.7:0/1025605938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44c0072950 0x7f44c0152b10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:57.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 >> 172.21.15.7:0/1025605938 conn(0x7f44c00cbf80 msgr2=0x7f44c0159fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:57.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 shutdown_connections 2023-11-28T18:45:57.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:57.006+0000 7f44c53df640 1 -- 172.21.15.7:0/1025605938 wait complete. 2023-11-28T18:45:57.028 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-11-28T18:45:59.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.182+0000 7f55682d6640 1 Processor -- start 2023-11-28T18:45:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.182+0000 7f55682d6640 1 -- start start 2023-11-28T18:45:59.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f55682d6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5560072950 0x7f5560074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f55682d6640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f556006c920 0x7f556006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f55682d6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55600c8490 con 0x7f5560073070 2023-11-28T18:45:59.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f55682d6640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556006d2e0 con 0x7f556006c920 2023-11-28T18:45:59.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f55682d6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55600c81f0 con 0x7f5560072950 2023-11-28T18:45:59.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565871640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5560073070 0x7f55600c19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57892/0 (socket says 172.21.15.7:57892) 2023-11-28T18:45:59.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565871640 1 -- 172.21.15.7:0/4199707660 learned_addr learned my addr 172.21.15.7:0/4199707660 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:45:59.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566873640 1 --2- 172.21.15.7:0/4199707660 >> v2:172.21.15.193:3300/0 conn(0x7f556006c920 0x7f556006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566072640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5560072950 0x7f5560074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061108929 0 0) 0x7f55600c8490 con 0x7f5560073070 2023-11-28T18:45:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f554c003530 con 0x7f5560073070 2023-11-28T18:45:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3157798986 0 0) 0x7f554c003530 con 0x7f5560073070 2023-11-28T18:45:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5560072950 msgr2=0x7f5560074030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5560072950 0x7f5560074030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 >> v2:172.21.15.193:3300/0 conn(0x7f556006c920 msgr2=0x7f556006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 --2- 172.21.15.7:0/4199707660 >> v2:172.21.15.193:3300/0 conn(0x7f556006c920 0x7f556006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556015ab00 con 0x7f5560073070 2023-11-28T18:45:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f5550001680 con 0x7f5560073070 2023-11-28T18:45:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 >> v1:172.21.15.7:6789/0 conn(0x7f5560073070 legacy=0x7f55600c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f554c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5560072950 0x7f554c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f554c004d10 0x7f554c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556006d2e0 con 0x7f554c003980 2023-11-28T18:45:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600c81f0 con 0x7f554c004d10 2023-11-28T18:45:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c00ba10 con 0x7f5560072950 2023-11-28T18:45:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566072640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f554c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566873640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f554c004d10 0x7f554c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5565871640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5560072950 0x7f554c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566072640 1 -- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5560072950 msgr2=0x7f554c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566072640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5560072950 0x7f554c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566072640 1 -- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f554c004d10 msgr2=0x7f554c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566072640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f554c004d10 0x7f554c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.186+0000 7f5566072640 1 -- 172.21.15.7:0/4199707660 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c003530 con 0x7f554c003980 2023-11-28T18:45:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f5566072640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f554c003da0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f555c009ae0 tx=0x7f555c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6fcdfaaeb9bcd9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c0a9420 con 0x7f554c003980 2023-11-28T18:45:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c0a9580 con 0x7f554c003980 2023-11-28T18:45:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f5565070640 1 -- 172.21.15.7:0/4199707660 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c0a9930 con 0x7f554c003980 2023-11-28T18:45:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f55682d6640 1 -- 172.21.15.7:0/4199707660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 msgr2=0x7f554c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f55682d6640 1 --2- 172.21.15.7:0/4199707660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f554c003da0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f555c009ae0 tx=0x7f555c01ca50 comp rx=0 tx=0).stop 2023-11-28T18:45:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f5566873640 1 -- 172.21.15.7:0/4199707660 reap_dead start 2023-11-28T18:45:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f55682d6640 1 -- 172.21.15.7:0/4199707660 shutdown_connections 2023-11-28T18:45:59.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f55682d6640 1 -- 172.21.15.7:0/4199707660 >> 172.21.15.7:0/4199707660 conn(0x7f55600cbf80 msgr2=0x7f55600cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:59.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f55682d6640 1 -- 172.21.15.7:0/4199707660 shutdown_connections 2023-11-28T18:45:59.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f55682d6640 1 -- 172.21.15.7:0/4199707660 wait complete. 2023-11-28T18:45:59.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f55682d6640 1 Processor -- start 2023-11-28T18:45:59.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.190+0000 7f55682d6640 1 -- start start 2023-11-28T18:45:59.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f55600c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f556015d5d0 0x7f55600c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f556015db90 0x7f556014c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556015a0a0 con 0x7f554c003980 2023-11-28T18:45:59.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560151fe0 con 0x7f556015db90 2023-11-28T18:45:59.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55601f2830 con 0x7f556015d5d0 2023-11-28T18:45:59.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f55600c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f55600c32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48764/0 (socket says 172.21.15.7:48764) 2023-11-28T18:45:59.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 -- 172.21.15.7:0/627995296 learned_addr learned my addr 172.21.15.7:0/627995296 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:59.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5565871640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f556015d5d0 0x7f55600c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566873640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f556015db90 0x7f556014c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 -- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f556015d5d0 msgr2=0x7f55600c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f556015d5d0 0x7f55600c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 -- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f556015db90 msgr2=0x7f556014c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f556015db90 0x7f556014c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 -- 172.21.15.7:0/627995296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560152230 con 0x7f554c003980 2023-11-28T18:45:59.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5565871640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f556015d5d0 0x7f55600c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:45:59.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5566072640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f55600c32b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f555c0ab040 tx=0x7f555c0197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:59.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5556ffd640 1 -- 172.21.15.7:0/627995296 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c0b6070 con 0x7f554c003980 2023-11-28T18:45:59.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5556ffd640 1 -- 172.21.15.7:0/627995296 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c0a96f0 con 0x7f554c003980 2023-11-28T18:45:59.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5556ffd640 1 -- 172.21.15.7:0/627995296 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c0a9ac0 con 0x7f554c003980 2023-11-28T18:45:59.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55601f2ab0 con 0x7f554c003980 2023-11-28T18:45:59.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55601f3070 con 0x7f554c003980 2023-11-28T18:45:59.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556004e890 con 0x7f554c003980 2023-11-28T18:45:59.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.194+0000 7f5556ffd640 1 -- 172.21.15.7:0/627995296 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f555c0a9c20 con 0x7f554c003980 2023-11-28T18:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.198+0000 7f5556ffd640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5544066900 0x7f5544068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.198+0000 7f5556ffd640 1 -- 172.21.15.7:0/627995296 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f555c11e500 con 0x7f554c003980 2023-11-28T18:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.198+0000 7f5565871640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5544066900 0x7f5544068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.198+0000 7f5565871640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5544066900 0x7f5544068dc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f55601ed170 tx=0x7f5550009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:59.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.202+0000 7f5556ffd640 1 -- 172.21.15.7:0/627995296 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f555c0eb190 con 0x7f554c003980 2023-11-28T18:45:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5560163de0 con 0x7f554c003980 2023-11-28T18:45:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f5556ffd640 1 -- 172.21.15.7:0/627995296 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f555c0bf090 con 0x7f554c003980 2023-11-28T18:45:59.392 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:59.392 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:45:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5544066900 msgr2=0x7f5544068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5544066900 0x7f5544068dc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f55601ed170 tx=0x7f5550009040 comp rx=0 tx=0).stop 2023-11-28T18:45:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 msgr2=0x7f55600c32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f55600c32b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f555c0ab040 tx=0x7f555c0197f0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 shutdown_connections 2023-11-28T18:45:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5544066900 0x7f5544068dc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f556015db90 0x7f556014c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f556015d5d0 0x7f55600c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 --2- 172.21.15.7:0/627995296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c003980 0x7f55600c32b0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 >> 172.21.15.7:0/627995296 conn(0x7f55600cbf80 msgr2=0x7f55600ca6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 shutdown_connections 2023-11-28T18:45:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.390+0000 7f55682d6640 1 -- 172.21.15.7:0/627995296 wait complete. 2023-11-28T18:45:59.419 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported laggy/crashed since: 2023-11-28T18:45:57.508800+0000 2023-11-28T18:45:59.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-11-28T18:45:59.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:45:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.566+0000 7fd448fdf640 1 Processor -- start 2023-11-28T18:45:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.566+0000 7fd448fdf640 1 -- start start 2023-11-28T18:45:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.566+0000 7fd448fdf640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fd444150b30 0x7fd4440c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.566+0000 7fd448fdf640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd4440c7980 0x7fd4440c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.566+0000 7fd448fdf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4440c6140 con 0x7fd4441575c0 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.566+0000 7fd448fdf640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4440c72b0 con 0x7fd4440c7980 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.566+0000 7fd448fdf640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd444070c00 con 0x7fd444150b30 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.566+0000 7fd442d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd4440c7980 0x7fd4440c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd442d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd4440c7980 0x7fd4440c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48768/0 (socket says 172.21.15.7:48768) 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd442d9d640 1 -- 172.21.15.7:0/2470330803 learned_addr learned my addr 172.21.15.7:0/2470330803 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd44259c640 1 --2- 172.21.15.7:0/2470330803 >> v2:172.21.15.193:3300/0 conn(0x7fd444150b30 0x7fd4440c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd442d9d640 1 -- 172.21.15.7:0/2470330803 >> v2:172.21.15.193:3300/0 conn(0x7fd444150b30 msgr2=0x7fd4440c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd442d9d640 1 --2- 172.21.15.7:0/2470330803 >> v2:172.21.15.193:3300/0 conn(0x7fd444150b30 0x7fd4440c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd442d9d640 1 -- 172.21.15.7:0/2470330803 >> v1:172.21.15.193:6789/0 conn(0x7fd4441575c0 legacy=0x7fd4440c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd442d9d640 1 -- 172.21.15.7:0/2470330803 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4440c8310 con 0x7fd4440c7980 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd442d9d640 1 --2- 172.21.15.7:0/2470330803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4440c7980 0x7fd4440c7dd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd44406ec80 tx=0x7fd43c0213d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fd4eaf1a7dfc72 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:59.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd441d9b640 1 -- 172.21.15.7:0/2470330803 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0b1400 con 0x7fd4440c7980 2023-11-28T18:45:59.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd441d9b640 1 -- 172.21.15.7:0/2470330803 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43c0ad050 con 0x7fd4440c7980 2023-11-28T18:45:59.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.570+0000 7fd441d9b640 1 -- 172.21.15.7:0/2470330803 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0b1780 con 0x7fd4440c7980 2023-11-28T18:45:59.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- 172.21.15.7:0/2470330803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4440c7980 msgr2=0x7fd4440c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 --2- 172.21.15.7:0/2470330803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4440c7980 0x7fd4440c7dd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd44406ec80 tx=0x7fd43c0213d0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- 172.21.15.7:0/2470330803 shutdown_connections 2023-11-28T18:45:59.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 --2- 172.21.15.7:0/2470330803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4440c7980 0x7fd4440c7dd0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 --2- 172.21.15.7:0/2470330803 >> v2:172.21.15.193:3300/0 conn(0x7fd444150b30 0x7fd4440c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- 172.21.15.7:0/2470330803 >> 172.21.15.7:0/2470330803 conn(0x7fd4440662e0 msgr2=0x7fd4440666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:59.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- 172.21.15.7:0/2470330803 shutdown_connections 2023-11-28T18:45:59.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- 172.21.15.7:0/2470330803 wait complete. 2023-11-28T18:45:59.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 Processor -- start 2023-11-28T18:45:59.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- start start 2023-11-28T18:45:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4440c7980 0x7fd444151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd444150b30 0x7fd4441eba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4441575c0 0x7fd444153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd444153c90 con 0x7fd444150b30 2023-11-28T18:45:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441eb5d0 con 0x7fd4441575c0 2023-11-28T18:45:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd448fdf640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd444154350 con 0x7fd4440c7980 2023-11-28T18:45:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd43bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd444150b30 0x7fd4441eba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd43bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd444150b30 0x7fd4441eba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48778/0 (socket says 172.21.15.7:48778) 2023-11-28T18:45:59.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd43bfff640 1 -- 172.21.15.7:0/3324424996 learned_addr learned my addr 172.21.15.7:0/3324424996 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:45:59.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd44259c640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4440c7980 0x7fd444151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd442d9d640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4441575c0 0x7fd444153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd43bfff640 1 -- 172.21.15.7:0/3324424996 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4440c7980 msgr2=0x7fd444151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd43bfff640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4440c7980 0x7fd444151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd43bfff640 1 -- 172.21.15.7:0/3324424996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4441575c0 msgr2=0x7fd444153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd43bfff640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4441575c0 0x7fd444153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.574+0000 7fd43bfff640 1 -- 172.21.15.7:0/3324424996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43c009b30 con 0x7fd444150b30 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd43bfff640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd444150b30 0x7fd4441eba60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00c3f0 tx=0x7fd42c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd43affd640 1 -- 172.21.15.7:0/3324424996 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c0052d0 con 0x7fd444150b30 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd43affd640 1 -- 172.21.15.7:0/3324424996 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c005430 con 0x7fd444150b30 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd43affd640 1 -- 172.21.15.7:0/3324424996 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c003840 con 0x7fd444150b30 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd444147ef0 con 0x7fd444150b30 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4441eefc0 con 0x7fd444150b30 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4441eb760 con 0x7fd444150b30 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd43affd640 1 -- 172.21.15.7:0/3324424996 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd42c0039a0 con 0x7fd444150b30 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd43affd640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd424066950 0x7fd424068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:45:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd43affd640 1 -- 172.21.15.7:0/3324424996 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fd42c083da0 con 0x7fd444150b30 2023-11-28T18:45:59.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.578+0000 7fd44259c640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd424066950 0x7fd424068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:45:59.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.582+0000 7fd44259c640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd424066950 0x7fd424068e10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd434007a60 tx=0x7fd434008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:45:59.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.582+0000 7fd43affd640 1 -- 172.21.15.7:0/3324424996 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd42c050c80 con 0x7fd444150b30 2023-11-28T18:45:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd444150f10 con 0x7fd444150b30 2023-11-28T18:45:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd43affd640 1 -- 172.21.15.7:0/3324424996 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7fd42c0542a0 con 0x7fd444150b30 2023-11-28T18:45:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:45:59.775 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:45:59.775 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:45:59.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd424066950 msgr2=0x7fd424068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd424066950 0x7fd424068e10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd434007a60 tx=0x7fd434008040 comp rx=0 tx=0).stop 2023-11-28T18:45:59.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd444150b30 msgr2=0x7fd4441eba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:45:59.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd444150b30 0x7fd4441eba60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00c3f0 tx=0x7fd42c00c8b0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 shutdown_connections 2023-11-28T18:45:59.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd424066950 0x7fd424068e10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4441575c0 0x7fd444153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd444150b30 0x7fd4441eba60 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 --2- 172.21.15.7:0/3324424996 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd4440c7980 0x7fd444151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:45:59.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 >> 172.21.15.7:0/3324424996 conn(0x7fd4440662e0 msgr2=0x7fd44414f160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:45:59.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 shutdown_connections 2023-11-28T18:45:59.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:45:59.774+0000 7fd448fdf640 1 -- 172.21.15.7:0/3324424996 wait complete. 2023-11-28T18:45:59.797 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:45:57.508800+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:46:00.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~41s 2023-11-28T18:46:01.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:01.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a30aea640 1 Processor -- start 2023-11-28T18:46:01.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a30aea640 1 -- start start 2023-11-28T18:46:01.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a30aea640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2a2c070e70 0x7f2a2c15a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:01.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a30aea640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a2c071250 0x7f2a2c0c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:01.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a30aea640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2c0c48e0 con 0x7f2a2c0ccbb0 2023-11-28T18:46:01.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a30aea640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2c15e330 con 0x7f2a2c071250 2023-11-28T18:46:01.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a30aea640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2c15ad80 con 0x7f2a2c070e70 2023-11-28T18:46:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a23fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a2c071250 0x7f2a2c0c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a23fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a2c071250 0x7f2a2c0c4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48780/0 (socket says 172.21.15.7:48780) 2023-11-28T18:46:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a23fff640 1 -- 172.21.15.7:0/203884214 learned_addr learned my addr 172.21.15.7:0/203884214 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a2a59c640 1 --2- 172.21.15.7:0/203884214 >> v2:172.21.15.193:3300/0 conn(0x7f2a2c070e70 0x7f2a2c15a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a23fff640 1 -- 172.21.15.7:0/203884214 >> v2:172.21.15.193:3300/0 conn(0x7f2a2c070e70 msgr2=0x7f2a2c15a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:01.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a23fff640 1 --2- 172.21.15.7:0/203884214 >> v2:172.21.15.193:3300/0 conn(0x7f2a2c070e70 0x7f2a2c15a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:01.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a23fff640 1 -- 172.21.15.7:0/203884214 >> v1:172.21.15.193:6789/0 conn(0x7f2a2c0ccbb0 legacy=0x7f2a2c153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:01.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.954+0000 7f2a23fff640 1 -- 172.21.15.7:0/203884214 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2c15d7b0 con 0x7f2a2c071250 2023-11-28T18:46:01.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a23fff640 1 --2- 172.21.15.7:0/203884214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c071250 0x7f2a2c0c4d70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0cc660 tx=0x7f2a1401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c85e615c3a8c27 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:01.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a29d9b640 1 -- 172.21.15.7:0/203884214 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140a89f0 con 0x7f2a2c071250 2023-11-28T18:46:01.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a29d9b640 1 -- 172.21.15.7:0/203884214 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a140a8b50 con 0x7f2a2c071250 2023-11-28T18:46:01.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a29d9b640 1 -- 172.21.15.7:0/203884214 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140b0440 con 0x7f2a2c071250 2023-11-28T18:46:01.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a30aea640 1 -- 172.21.15.7:0/203884214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c071250 msgr2=0x7f2a2c0c4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:01.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a30aea640 1 --2- 172.21.15.7:0/203884214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c071250 0x7f2a2c0c4d70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0cc660 tx=0x7f2a1401ecf0 comp rx=0 tx=0).stop 2023-11-28T18:46:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a30aea640 1 -- 172.21.15.7:0/203884214 shutdown_connections 2023-11-28T18:46:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a30aea640 1 --2- 172.21.15.7:0/203884214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c071250 0x7f2a2c0c4d70 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a30aea640 1 --2- 172.21.15.7:0/203884214 >> v2:172.21.15.193:3300/0 conn(0x7f2a2c070e70 0x7f2a2c15a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a30aea640 1 -- 172.21.15.7:0/203884214 >> 172.21.15.7:0/203884214 conn(0x7f2a2c0662e0 msgr2=0x7f2a2c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a30aea640 1 -- 172.21.15.7:0/203884214 shutdown_connections 2023-11-28T18:46:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.958+0000 7f2a30aea640 1 -- 172.21.15.7:0/203884214 wait complete. 2023-11-28T18:46:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 Processor -- start 2023-11-28T18:46:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 -- start start 2023-11-28T18:46:01.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c070e70 0x7f2a2c151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:01.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a2c071250 0x7f2a2c0b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:01.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a2c0ccbb0 0x7f2a2c1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:01.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c14be10 con 0x7f2a2c070e70 2023-11-28T18:46:01.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c0ce840 con 0x7f2a2c0ccbb0 2023-11-28T18:46:01.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c14ba70 con 0x7f2a2c071250 2023-11-28T18:46:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c070e70 0x7f2a2c151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c070e70 0x7f2a2c151570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48782/0 (socket says 172.21.15.7:48782) 2023-11-28T18:46:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 -- 172.21.15.7:0/975700147 learned_addr learned my addr 172.21.15.7:0/975700147 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 -- 172.21.15.7:0/975700147 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a2c071250 msgr2=0x7f2a2c0b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a2c071250 0x7f2a2c0b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 -- 172.21.15.7:0/975700147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a2c0ccbb0 msgr2=0x7f2a2c1f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a2c0ccbb0 0x7f2a2c1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 -- 172.21.15.7:0/975700147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a140094d0 con 0x7f2a2c070e70 2023-11-28T18:46:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a2a59c640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c070e70 0x7f2a2c151570 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00d5f0 tx=0x7f2a1c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a22ffd640 1 -- 172.21.15.7:0/975700147 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c0125f0 con 0x7f2a2c070e70 2023-11-28T18:46:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a22ffd640 1 -- 172.21.15.7:0/975700147 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c012750 con 0x7f2a2c070e70 2023-11-28T18:46:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a22ffd640 1 -- 172.21.15.7:0/975700147 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c012b00 con 0x7f2a2c070e70 2023-11-28T18:46:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c1ef960 con 0x7f2a2c070e70 2023-11-28T18:46:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c1efc30 con 0x7f2a2c070e70 2023-11-28T18:46:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.962+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c04e890 con 0x7f2a2c070e70 2023-11-28T18:46:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.966+0000 7f2a22ffd640 1 -- 172.21.15.7:0/975700147 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2a1c012c60 con 0x7f2a2c070e70 2023-11-28T18:46:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.966+0000 7f2a22ffd640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a0c0668d0 0x7f2a0c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.966+0000 7f2a22ffd640 1 -- 172.21.15.7:0/975700147 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f2a1c084ef0 con 0x7f2a2c070e70 2023-11-28T18:46:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.966+0000 7f2a23fff640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a0c0668d0 0x7f2a0c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.966+0000 7f2a23fff640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a0c0668d0 0x7f2a0c068d90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2a140b2000 tx=0x7f2a14004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:01.970+0000 7f2a22ffd640 1 -- 172.21.15.7:0/975700147 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2a1c051c10 con 0x7f2a2c070e70 2023-11-28T18:46:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.158+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a2c158ea0 con 0x7f2a2c070e70 2023-11-28T18:46:02.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.158+0000 7f2a22ffd640 1 -- 172.21.15.7:0/975700147 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f2a1c055230 con 0x7f2a2c070e70 2023-11-28T18:46:02.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:02.160 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:02.160 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:02.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a0c0668d0 msgr2=0x7f2a0c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:02.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a0c0668d0 0x7f2a0c068d90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2a140b2000 tx=0x7f2a14004740 comp rx=0 tx=0).stop 2023-11-28T18:46:02.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c070e70 msgr2=0x7f2a2c151570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:02.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c070e70 0x7f2a2c151570 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00d5f0 tx=0x7f2a1c00dab0 comp rx=0 tx=0).stop 2023-11-28T18:46:02.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 shutdown_connections 2023-11-28T18:46:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2a0c0668d0 0x7f2a0c068d90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a2c0ccbb0 0x7f2a2c1f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a2c071250 0x7f2a2c0b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 --2- 172.21.15.7:0/975700147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2c070e70 0x7f2a2c151570 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 >> 172.21.15.7:0/975700147 conn(0x7f2a2c0662e0 msgr2=0x7f2a2c157a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 shutdown_connections 2023-11-28T18:46:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:02.162+0000 7f2a30aea640 1 -- 172.21.15.7:0/975700147 wait complete. 2023-11-28T18:46:02.181 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:04.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.334+0000 7f975539d640 1 Processor -- start 2023-11-28T18:46:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.334+0000 7f975539d640 1 -- start start 2023-11-28T18:46:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f975539d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f975015d350 0x7f975006a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f975539d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f975006ab30 0x7f97500c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f975539d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97500c8750 con 0x7f97501540a0 2023-11-28T18:46:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f975539d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97500c8a30 con 0x7f975006ab30 2023-11-28T18:46:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f975539d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97500731c0 con 0x7f975015d350 2023-11-28T18:46:04.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f974f7fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f975006ab30 0x7f97500c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:04.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f9747fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f975015d350 0x7f975006a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f974effd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f97501540a0 0x7f9750072ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:53864/0 (socket says 172.21.15.7:53864) 2023-11-28T18:46:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f974effd640 1 -- 172.21.15.7:0/2738186862 learned_addr learned my addr 172.21.15.7:0/2738186862 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f974f7fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f975006ab30 0x7f97500c8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:58810/0 (socket says 172.21.15.7:58810) 2023-11-28T18:46:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f9747fff640 1 -- 172.21.15.7:0/2738186862 >> v2:172.21.15.193:3300/0 conn(0x7f975006ab30 msgr2=0x7f97500c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f9747fff640 1 --2- 172.21.15.7:0/2738186862 >> v2:172.21.15.193:3300/0 conn(0x7f975006ab30 0x7f97500c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f9747fff640 1 -- 172.21.15.7:0/2738186862 >> v1:172.21.15.193:6789/0 conn(0x7f97501540a0 legacy=0x7f9750072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f9747fff640 1 -- 172.21.15.7:0/2738186862 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975015d840 con 0x7f975015d350 2023-11-28T18:46:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.338+0000 7f974e7fc640 1 -- 172.21.15.7:0/2738186862 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911231383 0 0) 0x7f97500c8750 con 0x7f97501540a0 2023-11-28T18:46:04.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f9747fff640 1 --2- 172.21.15.7:0/2738186862 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f975015d350 0x7f975006a5f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f97380097f0 tx=0x7f973801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608447ef971118d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f974e7fc640 1 -- 172.21.15.7:0/2738186862 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380ab600 con 0x7f975015d350 2023-11-28T18:46:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f974e7fc640 1 -- 172.21.15.7:0/2738186862 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97380a7050 con 0x7f975015d350 2023-11-28T18:46:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f974e7fc640 1 -- 172.21.15.7:0/2738186862 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380ab980 con 0x7f975015d350 2023-11-28T18:46:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 -- 172.21.15.7:0/2738186862 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f975015d350 msgr2=0x7f975006a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 --2- 172.21.15.7:0/2738186862 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f975015d350 0x7f975006a5f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f97380097f0 tx=0x7f973801c480 comp rx=0 tx=0).stop 2023-11-28T18:46:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 -- 172.21.15.7:0/2738186862 shutdown_connections 2023-11-28T18:46:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 --2- 172.21.15.7:0/2738186862 >> v2:172.21.15.193:3300/0 conn(0x7f975006ab30 0x7f97500c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 --2- 172.21.15.7:0/2738186862 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f975015d350 0x7f975006a5f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 -- 172.21.15.7:0/2738186862 >> 172.21.15.7:0/2738186862 conn(0x7f9750073cd0 msgr2=0x7f97500662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 -- 172.21.15.7:0/2738186862 shutdown_connections 2023-11-28T18:46:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 -- 172.21.15.7:0/2738186862 wait complete. 2023-11-28T18:46:04.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 Processor -- start 2023-11-28T18:46:04.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.342+0000 7f975539d640 1 -- start start 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f975006ab30 0x7f9750158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97501540a0 0x7f9750159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f975015d350 0x7f9750159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750160ac0 con 0x7f975015d350 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97501f0d60 con 0x7f975006ab30 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97501586f0 con 0x7f97501540a0 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f975015d350 0x7f9750159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f975006ab30 0x7f9750158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f975015d350 0x7f9750159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47022/0 (socket says 172.21.15.7:47022) 2023-11-28T18:46:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 -- 172.21.15.7:0/3870795748 learned_addr learned my addr 172.21.15.7:0/3870795748 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f9747fff640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97501540a0 0x7f9750159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 -- 172.21.15.7:0/3870795748 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97501540a0 msgr2=0x7f9750159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97501540a0 0x7f9750159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 -- 172.21.15.7:0/3870795748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f975006ab30 msgr2=0x7f9750158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f975006ab30 0x7f9750158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 -- 172.21.15.7:0/3870795748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738009420 con 0x7f975015d350 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974effd640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f975006ab30 0x7f9750158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f974f7fe640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f975015d350 0x7f9750159d80 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f974800efc0 tx=0x7f974800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f97477fe640 1 -- 172.21.15.7:0/3870795748 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748014480 con 0x7f975015d350 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f97477fe640 1 -- 172.21.15.7:0/3870795748 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974800f040 con 0x7f975015d350 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f97477fe640 1 -- 172.21.15.7:0/3870795748 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748014820 con 0x7f975015d350 2023-11-28T18:46:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97501f3cd0 con 0x7f975015d350 2023-11-28T18:46:04.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9750157dc0 con 0x7f975015d350 2023-11-28T18:46:04.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.346+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975004e890 con 0x7f975015d350 2023-11-28T18:46:04.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.350+0000 7f97477fe640 1 -- 172.21.15.7:0/3870795748 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9748014980 con 0x7f975015d350 2023-11-28T18:46:04.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.350+0000 7f97477fe640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97300668d0 0x7f9730068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:04.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.350+0000 7f97477fe640 1 -- 172.21.15.7:0/3870795748 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f9748085310 con 0x7f975015d350 2023-11-28T18:46:04.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.350+0000 7f974effd640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97300668d0 0x7f9730068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:04.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.350+0000 7f974effd640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97300668d0 0x7f9730068d90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9740007fd0 tx=0x7f9740007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:04.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.354+0000 7f97477fe640 1 -- 172.21.15.7:0/3870795748 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9748052160 con 0x7f975015d350 2023-11-28T18:46:04.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.546+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9750158830 con 0x7f975015d350 2023-11-28T18:46:04.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f97477fe640 1 -- 172.21.15.7:0/3870795748 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f9748055780 con 0x7f975015d350 2023-11-28T18:46:04.549 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:04.549 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:04.550 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97300668d0 msgr2=0x7f9730068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97300668d0 0x7f9730068d90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9740007fd0 tx=0x7f9740007d60 comp rx=0 tx=0).stop 2023-11-28T18:46:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f975015d350 msgr2=0x7f9750159d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f975015d350 0x7f9750159d80 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f974800efc0 tx=0x7f974800c490 comp rx=0 tx=0).stop 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 shutdown_connections 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97300668d0 0x7f9730068d90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f975015d350 0x7f9750159d80 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97501540a0 0x7f9750159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 --2- 172.21.15.7:0/3870795748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f975006ab30 0x7f9750158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 >> 172.21.15.7:0/3870795748 conn(0x7f9750073cd0 msgr2=0x7f975015b130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 shutdown_connections 2023-11-28T18:46:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:04.550+0000 7f975539d640 1 -- 172.21.15.7:0/3870795748 wait complete. 2023-11-28T18:46:04.571 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:06.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:06.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.726+0000 7f28b852a640 1 Processor -- start 2023-11-28T18:46:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.726+0000 7f28b852a640 1 -- start start 2023-11-28T18:46:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b852a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f28b0151cd0 0x7f28b015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b852a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28b0157b90 0x7f28b00701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b852a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b0071e60 con 0x7f28b006e560 2023-11-28T18:46:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b852a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b0070730 con 0x7f28b0157b90 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b852a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b0070930 con 0x7f28b0151cd0 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b6ac7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28b0157b90 0x7f28b00701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b6ac7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28b0157b90 0x7f28b00701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47024/0 (socket says 172.21.15.7:47024) 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b6ac7640 1 -- 172.21.15.7:0/1364930106 learned_addr learned my addr 172.21.15.7:0/1364930106 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b62c6640 1 --2- 172.21.15.7:0/1364930106 >> v2:172.21.15.193:3300/0 conn(0x7f28b0151cd0 0x7f28b015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b6ac7640 1 -- 172.21.15.7:0/1364930106 >> v2:172.21.15.193:3300/0 conn(0x7f28b0151cd0 msgr2=0x7f28b015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b6ac7640 1 --2- 172.21.15.7:0/1364930106 >> v2:172.21.15.193:3300/0 conn(0x7f28b0151cd0 0x7f28b015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b6ac7640 1 -- 172.21.15.7:0/1364930106 >> v1:172.21.15.193:6789/0 conn(0x7f28b006e560 legacy=0x7f28b00c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.730+0000 7f28b6ac7640 1 -- 172.21.15.7:0/1364930106 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b0157fa0 con 0x7f28b0157b90 2023-11-28T18:46:06.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b6ac7640 1 --2- 172.21.15.7:0/1364930106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b00701f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f28b006fee0 tx=0x7f28a801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2490caf0a9b463d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b52c4640 1 -- 172.21.15.7:0/1364930106 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80a56f0 con 0x7f28b0157b90 2023-11-28T18:46:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b52c4640 1 -- 172.21.15.7:0/1364930106 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a80a5850 con 0x7f28b0157b90 2023-11-28T18:46:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b52c4640 1 -- 172.21.15.7:0/1364930106 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80a5be0 con 0x7f28b0157b90 2023-11-28T18:46:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 -- 172.21.15.7:0/1364930106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 msgr2=0x7f28b00701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 --2- 172.21.15.7:0/1364930106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b00701f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f28b006fee0 tx=0x7f28a801c940 comp rx=0 tx=0).stop 2023-11-28T18:46:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 -- 172.21.15.7:0/1364930106 shutdown_connections 2023-11-28T18:46:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 --2- 172.21.15.7:0/1364930106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b00701f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 --2- 172.21.15.7:0/1364930106 >> v2:172.21.15.193:3300/0 conn(0x7f28b0151cd0 0x7f28b015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 -- 172.21.15.7:0/1364930106 >> 172.21.15.7:0/1364930106 conn(0x7f28b00662e0 msgr2=0x7f28b00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 -- 172.21.15.7:0/1364930106 shutdown_connections 2023-11-28T18:46:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 -- 172.21.15.7:0/1364930106 wait complete. 2023-11-28T18:46:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 Processor -- start 2023-11-28T18:46:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.734+0000 7f28b852a640 1 -- start start 2023-11-28T18:46:06.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28b006e560 0x7f28b00ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:06.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28b0151cd0 0x7f28b0159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:06.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b0143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:06.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b00cac80 con 0x7f28b0157b90 2023-11-28T18:46:06.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b015a690 con 0x7f28b0151cd0 2023-11-28T18:46:06.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b015a950 con 0x7f28b006e560 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b0143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b0143820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47026/0 (socket says 172.21.15.7:47026) 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 -- 172.21.15.7:0/394602325 learned_addr learned my addr 172.21.15.7:0/394602325 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b5ac5640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28b0151cd0 0x7f28b0159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b62c6640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28b006e560 0x7f28b00ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 -- 172.21.15.7:0/394602325 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28b006e560 msgr2=0x7f28b00ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28b006e560 0x7f28b00ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 -- 172.21.15.7:0/394602325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28b0151cd0 msgr2=0x7f28b0159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28b0151cd0 0x7f28b0159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 -- 172.21.15.7:0/394602325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8009b30 con 0x7f28b0157b90 2023-11-28T18:46:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b6ac7640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b0143820 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f28a80af040 tx=0x7f28a80196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28a6ffd640 1 -- 172.21.15.7:0/394602325 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80b8670 con 0x7f28b0157b90 2023-11-28T18:46:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28a6ffd640 1 -- 172.21.15.7:0/394602325 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a80b87f0 con 0x7f28b0157b90 2023-11-28T18:46:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28a6ffd640 1 -- 172.21.15.7:0/394602325 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80b8ba0 con 0x7f28b0157b90 2023-11-28T18:46:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b015ac30 con 0x7f28b0157b90 2023-11-28T18:46:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b00c9d40 con 0x7f28b0157b90 2023-11-28T18:46:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b00c9f30 con 0x7f28b0157b90 2023-11-28T18:46:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.738+0000 7f28a6ffd640 1 -- 172.21.15.7:0/394602325 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f28a80b8d00 con 0x7f28b0157b90 2023-11-28T18:46:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.742+0000 7f28a6ffd640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28940668d0 0x7f2894068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.742+0000 7f28a6ffd640 1 -- 172.21.15.7:0/394602325 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f28a811fc30 con 0x7f28b0157b90 2023-11-28T18:46:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.742+0000 7f28b62c6640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28940668d0 0x7f2894068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.742+0000 7f28b62c6640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28940668d0 0x7f2894068d90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f28ac007a60 tx=0x7f28ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:06.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.746+0000 7f28a6ffd640 1 -- 172.21.15.7:0/394602325 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f28a80eca90 con 0x7f28b0157b90 2023-11-28T18:46:06.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.930+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28b004e890 con 0x7f28b0157b90 2023-11-28T18:46:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28a6ffd640 1 -- 172.21.15.7:0/394602325 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f28a8009510 con 0x7f28b0157b90 2023-11-28T18:46:06.933 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:06.933 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28940668d0 msgr2=0x7f2894068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28940668d0 0x7f2894068d90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f28ac007a60 tx=0x7f28ac008040 comp rx=0 tx=0).stop 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 msgr2=0x7f28b0143820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b0143820 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f28a80af040 tx=0x7f28a80196e0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 shutdown_connections 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28940668d0 0x7f2894068d90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28b0157b90 0x7f28b0143820 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28b0151cd0 0x7f28b0159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 --2- 172.21.15.7:0/394602325 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28b006e560 0x7f28b00ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 >> 172.21.15.7:0/394602325 conn(0x7f28b00662e0 msgr2=0x7f28b00724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 shutdown_connections 2023-11-28T18:46:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:06.934+0000 7f28b852a640 1 -- 172.21.15.7:0/394602325 wait complete. 2023-11-28T18:46:06.954 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:06.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~47s 2023-11-28T18:46:08.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:09.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f382b772640 1 Processor -- start 2023-11-28T18:46:09.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f382b772640 1 -- start start 2023-11-28T18:46:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f382b772640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f38240c6ca0 0x7f3824161f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f382b772640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38240c2970 con 0x7f382415e300 2023-11-28T18:46:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f382b772640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382415e820 con 0x7f38240c68c0 2023-11-28T18:46:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f382b772640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f382415eb00 con 0x7f38240c6ca0 2023-11-28T18:46:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f3829d0f640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f382415e300 0x7f3824157eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44370/0 (socket says 172.21.15.7:44370) 2023-11-28T18:46:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f3829d0f640 1 -- 172.21.15.7:0/1420693284 learned_addr learned my addr 172.21.15.7:0/1420693284 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.110+0000 7f3828d0d640 1 --2- 172.21.15.7:0/1420693284 >> v2:172.21.15.193:3300/0 conn(0x7f38240c6ca0 0x7f3824161f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:09.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f381bfff640 1 -- 172.21.15.7:0/1420693284 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177108259 0 0) 0x7f38240c2970 con 0x7f382415e300 2023-11-28T18:46:09.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f381bfff640 1 -- 172.21.15.7:0/1420693284 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3810003530 con 0x7f382415e300 2023-11-28T18:46:09.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f3828d0d640 1 -- 172.21.15.7:0/1420693284 >> v1:172.21.15.193:6789/0 conn(0x7f38240c68c0 legacy=0x7f38240c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:09.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f381bfff640 1 -- 172.21.15.7:0/1420693284 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570899174 0 0) 0x7f382415e820 con 0x7f38240c68c0 2023-11-28T18:46:09.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f3828d0d640 1 -- 172.21.15.7:0/1420693284 >> v1:172.21.15.7:6789/0 conn(0x7f382415e300 legacy=0x7f3824157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:09.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f3828d0d640 1 -- 172.21.15.7:0/1420693284 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382415ede0 con 0x7f38240c6ca0 2023-11-28T18:46:09.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f3828d0d640 1 --2- 172.21.15.7:0/1420693284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38240c6ca0 0x7f3824161f90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f38140097c0 tx=0x7f381401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=811d77dc66dc8993 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:09.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f381bfff640 1 -- 172.21.15.7:0/1420693284 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38140a8970 con 0x7f38240c6ca0 2023-11-28T18:46:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f381bfff640 1 -- 172.21.15.7:0/1420693284 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38140a8ad0 con 0x7f38240c6ca0 2023-11-28T18:46:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.114+0000 7f381bfff640 1 -- 172.21.15.7:0/1420693284 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38140a8e60 con 0x7f38240c6ca0 2023-11-28T18:46:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- 172.21.15.7:0/1420693284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38240c6ca0 msgr2=0x7f3824161f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 --2- 172.21.15.7:0/1420693284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38240c6ca0 0x7f3824161f90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f38140097c0 tx=0x7f381401ec70 comp rx=0 tx=0).stop 2023-11-28T18:46:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- 172.21.15.7:0/1420693284 shutdown_connections 2023-11-28T18:46:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 --2- 172.21.15.7:0/1420693284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38240c6ca0 0x7f3824161f90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- 172.21.15.7:0/1420693284 >> 172.21.15.7:0/1420693284 conn(0x7f38240662e0 msgr2=0x7f38240666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- 172.21.15.7:0/1420693284 shutdown_connections 2023-11-28T18:46:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- 172.21.15.7:0/1420693284 wait complete. 2023-11-28T18:46:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 Processor -- start 2023-11-28T18:46:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- start start 2023-11-28T18:46:09.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38240c68c0 0x7f38241f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38240c6ca0 0x7f38241f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f382415e300 0x7f3824152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3824160540 con 0x7f38240c68c0 2023-11-28T18:46:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38241616d0 con 0x7f38240c6ca0 2023-11-28T18:46:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382b772640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38241542c0 con 0x7f382415e300 2023-11-28T18:46:09.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38240c68c0 0x7f38241f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:09.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38240c68c0 0x7f38241f4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47036/0 (socket says 172.21.15.7:47036) 2023-11-28T18:46:09.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 -- 172.21.15.7:0/2283865857 learned_addr learned my addr 172.21.15.7:0/2283865857 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 -- 172.21.15.7:0/2283865857 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f382415e300 msgr2=0x7f3824152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f382415e300 0x7f3824152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:09.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 -- 172.21.15.7:0/2283865857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38240c6ca0 msgr2=0x7f38241f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:09.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38240c6ca0 0x7f38241f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:09.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 -- 172.21.15.7:0/2283865857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814009420 con 0x7f38240c68c0 2023-11-28T18:46:09.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.118+0000 7f382950e640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38240c68c0 0x7f38241f4d60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f382000c4d0 tx=0x7f382000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:09.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f3819ffb640 1 -- 172.21.15.7:0/2283865857 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820005480 con 0x7f38240c68c0 2023-11-28T18:46:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f3819ffb640 1 -- 172.21.15.7:0/2283865857 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3820003600 con 0x7f38240c68c0 2023-11-28T18:46:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f3819ffb640 1 -- 172.21.15.7:0/2283865857 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38200039b0 con 0x7f38240c68c0 2023-11-28T18:46:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38241e9e90 con 0x7f38240c68c0 2023-11-28T18:46:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3824153fe0 con 0x7f38240c68c0 2023-11-28T18:46:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3824161a20 con 0x7f38240c68c0 2023-11-28T18:46:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f3819ffb640 1 -- 172.21.15.7:0/2283865857 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3820003b10 con 0x7f38240c68c0 2023-11-28T18:46:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f3819ffb640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38080668d0 0x7f3808068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:09.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f3819ffb640 1 -- 172.21.15.7:0/2283865857 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f3820083f00 con 0x7f38240c68c0 2023-11-28T18:46:09.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.122+0000 7f3828d0d640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38080668d0 0x7f3808068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:09.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.126+0000 7f3828d0d640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38080668d0 0x7f3808068d90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f38140b2000 tx=0x7f3814004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:09.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.126+0000 7f3819ffb640 1 -- 172.21.15.7:0/2283865857 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3820050d70 con 0x7f38240c68c0 2023-11-28T18:46:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f382404e890 con 0x7f38240c68c0 2023-11-28T18:46:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f3819ffb640 1 -- 172.21.15.7:0/2283865857 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f3820017070 con 0x7f38240c68c0 2023-11-28T18:46:09.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:09.319 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38080668d0 msgr2=0x7f3808068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38080668d0 0x7f3808068d90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f38140b2000 tx=0x7f3814004740 comp rx=0 tx=0).stop 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38240c68c0 msgr2=0x7f38241f4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38240c68c0 0x7f38241f4d60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f382000c4d0 tx=0x7f382000c990 comp rx=0 tx=0).stop 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 shutdown_connections 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38080668d0 0x7f3808068d90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f382415e300 0x7f3824152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38240c6ca0 0x7f38241f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 --2- 172.21.15.7:0/2283865857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38240c68c0 0x7f38241f4d60 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.318+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 >> 172.21.15.7:0/2283865857 conn(0x7f38240662e0 msgr2=0x7f382415d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.322+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 shutdown_connections 2023-11-28T18:46:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:09.322+0000 7f382b772640 1 -- 172.21.15.7:0/2283865857 wait complete. 2023-11-28T18:46:09.341 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:11.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:11.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:46:11.467 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:46:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6fcdb8640 1 Processor -- start 2023-11-28T18:46:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6fcdb8640 1 -- start start 2023-11-28T18:46:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6fcdb8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff6f81540a0 0x7ff6f8072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6fcdb8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6f815d350 0x7ff6f806a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6fcdb8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f80c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6fcdb8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80c8750 con 0x7ff6f815d350 2023-11-28T18:46:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6fcdb8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80c8a30 con 0x7ff6f81540a0 2023-11-28T18:46:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6fcdb8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f815d840 con 0x7ff6f806ab30 2023-11-28T18:46:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6effff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6f815d350 0x7ff6f806a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:11.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6effff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6f815d350 0x7ff6f806a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47042/0 (socket says 172.21.15.7:47042) 2023-11-28T18:46:11.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.498+0000 7ff6effff640 1 -- 172.21.15.7:0/3139936380 learned_addr learned my addr 172.21.15.7:0/3139936380 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:11.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6f6d9d640 1 --2- 172.21.15.7:0/3139936380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f80c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:11.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6f659c640 1 --2- 172.21.15.7:0/3139936380 >> v2:172.21.15.193:3300/0 conn(0x7ff6f81540a0 0x7ff6f8072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6effff640 1 -- 172.21.15.7:0/3139936380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 msgr2=0x7ff6f80c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6effff640 1 --2- 172.21.15.7:0/3139936380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f80c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6effff640 1 -- 172.21.15.7:0/3139936380 >> v2:172.21.15.193:3300/0 conn(0x7ff6f81540a0 msgr2=0x7ff6f8072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6effff640 1 --2- 172.21.15.7:0/3139936380 >> v2:172.21.15.193:3300/0 conn(0x7ff6f81540a0 0x7ff6f8072ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6effff640 1 -- 172.21.15.7:0/3139936380 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f815db20 con 0x7ff6f815d350 2023-11-28T18:46:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6f6d9d640 1 --2- 172.21.15.7:0/3139936380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f80c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6f659c640 1 --2- 172.21.15.7:0/3139936380 >> v2:172.21.15.193:3300/0 conn(0x7ff6f81540a0 0x7ff6f8072ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6effff640 1 --2- 172.21.15.7:0/3139936380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f815d350 0x7ff6f806a5f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80ce820 tx=0x7ff6e001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db45d0469bc5faef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6f5d9b640 1 -- 172.21.15.7:0/3139936380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00ab600 con 0x7ff6f815d350 2023-11-28T18:46:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6f5d9b640 1 -- 172.21.15.7:0/3139936380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e00a7050 con 0x7ff6f815d350 2023-11-28T18:46:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.502+0000 7ff6f5d9b640 1 -- 172.21.15.7:0/3139936380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00ab980 con 0x7ff6f815d350 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/3139936380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f815d350 msgr2=0x7ff6f806a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/3139936380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f815d350 0x7ff6f806a5f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80ce820 tx=0x7ff6e001c500 comp rx=0 tx=0).stop 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/3139936380 shutdown_connections 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/3139936380 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f80c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/3139936380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f815d350 0x7ff6f806a5f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/3139936380 >> v2:172.21.15.193:3300/0 conn(0x7ff6f81540a0 0x7ff6f8072ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/3139936380 >> 172.21.15.7:0/3139936380 conn(0x7ff6f8073cd0 msgr2=0x7ff6f80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/3139936380 shutdown_connections 2023-11-28T18:46:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/3139936380 wait complete. 2023-11-28T18:46:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 Processor -- start 2023-11-28T18:46:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- start start 2023-11-28T18:46:11.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f814d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:11.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f81540a0 0x7ff6f814b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:11.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6f815d350 0x7ff6f814b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:11.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80cc350 con 0x7ff6f81540a0 2023-11-28T18:46:11.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f814eac0 con 0x7ff6f815d350 2023-11-28T18:46:11.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6fcdb8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f814ee90 con 0x7ff6f806ab30 2023-11-28T18:46:11.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6effff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f81540a0 0x7ff6f814b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6effff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f81540a0 0x7ff6f814b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47058/0 (socket says 172.21.15.7:47058) 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6effff640 1 -- 172.21.15.7:0/1605730339 learned_addr learned my addr 172.21.15.7:0/1605730339 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6f659c640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f814d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6f6d9d640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6f815d350 0x7ff6f814b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6effff640 1 -- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 msgr2=0x7ff6f814d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6effff640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f814d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6effff640 1 -- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6f815d350 msgr2=0x7ff6f814b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6effff640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6f815d350 0x7ff6f814b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6effff640 1 -- 172.21.15.7:0/1605730339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e00094d0 con 0x7ff6f81540a0 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.506+0000 7ff6f659c640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f814d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6effff640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f81540a0 0x7ff6f814b470 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00ae040 tx=0x7ff6f815ad90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6eeffd640 1 -- 172.21.15.7:0/1605730339 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00b6420 con 0x7ff6f81540a0 2023-11-28T18:46:11.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6eeffd640 1 -- 172.21.15.7:0/1605730339 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e00b6580 con 0x7ff6f81540a0 2023-11-28T18:46:11.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6eeffd640 1 -- 172.21.15.7:0/1605730339 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00b6930 con 0x7ff6f81540a0 2023-11-28T18:46:11.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f814af70 con 0x7ff6f81540a0 2023-11-28T18:46:11.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f80cc820 con 0x7ff6f81540a0 2023-11-28T18:46:11.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f804e890 con 0x7ff6f81540a0 2023-11-28T18:46:11.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6eeffd640 1 -- 172.21.15.7:0/1605730339 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff6e00b2040 con 0x7ff6f81540a0 2023-11-28T18:46:11.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6eeffd640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6d80668d0 0x7ff6d8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:11.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6eeffd640 1 -- 172.21.15.7:0/1605730339 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff6e011eff0 con 0x7ff6f81540a0 2023-11-28T18:46:11.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.510+0000 7ff6f659c640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6d80668d0 0x7ff6d8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:11.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.514+0000 7ff6f659c640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6d80668d0 0x7ff6d8068d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009970 tx=0x7ff6e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:11.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.514+0000 7ff6eeffd640 1 -- 172.21.15.7:0/1605730339 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff6e00ebdc0 con 0x7ff6f81540a0 2023-11-28T18:46:11.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.710+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f81537c0 con 0x7ff6f81540a0 2023-11-28T18:46:11.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.710+0000 7ff6eeffd640 1 -- 172.21.15.7:0/1605730339 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7ff6e00ef3e0 con 0x7ff6f81540a0 2023-11-28T18:46:11.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:11.711 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:11.712 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:11.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6d80668d0 msgr2=0x7ff6d8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:11.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6d80668d0 0x7ff6d8068d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009970 tx=0x7ff6e8008040 comp rx=0 tx=0).stop 2023-11-28T18:46:11.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f81540a0 msgr2=0x7ff6f814b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:11.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f81540a0 0x7ff6f814b470 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00ae040 tx=0x7ff6f815ad90 comp rx=0 tx=0).stop 2023-11-28T18:46:11.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 shutdown_connections 2023-11-28T18:46:11.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6d80668d0 0x7ff6d8068d90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6f815d350 0x7ff6f814b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6f81540a0 0x7ff6f814b470 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 --2- 172.21.15.7:0/1605730339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6f806ab30 0x7ff6f814d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:11.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 >> 172.21.15.7:0/1605730339 conn(0x7ff6f8073cd0 msgr2=0x7ff6f80c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:11.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 shutdown_connections 2023-11-28T18:46:11.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:11.714+0000 7ff6fcdb8640 1 -- 172.21.15.7:0/1605730339 wait complete. 2023-11-28T18:46:11.734 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:45:57.508800+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:46:13.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~54s 2023-11-28T18:46:13.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60b55b8640 1 Processor -- start 2023-11-28T18:46:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60b55b8640 1 -- start start 2023-11-28T18:46:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60b55b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00ca4d0 0x7f60b015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60b55b8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b015e380 con 0x7f60b00cf6c0 2023-11-28T18:46:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60b55b8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b015d7e0 con 0x7f60b01540e0 2023-11-28T18:46:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60b55b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b015ade0 con 0x7f60b00ca4d0 2023-11-28T18:46:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60af7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f60b00cf6c0 0x7f60b0164760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59880/0 (socket says 172.21.15.7:59880) 2023-11-28T18:46:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60af7fe640 1 -- 172.21.15.7:0/4072358115 learned_addr learned my addr 172.21.15.7:0/4072358115 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.890+0000 7f60a7fff640 1 --2- 172.21.15.7:0/4072358115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00ca4d0 0x7f60b015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60ae7fc640 1 -- 172.21.15.7:0/4072358115 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740106416 0 0) 0x7f60a80014a0 con 0x7f60b00cf6c0 2023-11-28T18:46:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60ae7fc640 1 -- 172.21.15.7:0/4072358115 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f609c003530 con 0x7f60b00cf6c0 2023-11-28T18:46:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60ae7fc640 1 -- 172.21.15.7:0/4072358115 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852922598 0 0) 0x7f60b015d7e0 con 0x7f60b01540e0 2023-11-28T18:46:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60ae7fc640 1 -- 172.21.15.7:0/4072358115 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a80014a0 con 0x7f60b01540e0 2023-11-28T18:46:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60ae7fc640 1 -- 172.21.15.7:0/4072358115 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882817540 0 0) 0x7f609c003530 con 0x7f60b00cf6c0 2023-11-28T18:46:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60a7fff640 1 -- 172.21.15.7:0/4072358115 >> v1:172.21.15.193:6789/0 conn(0x7f60b01540e0 legacy=0x7f60b015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60a7fff640 1 -- 172.21.15.7:0/4072358115 >> v1:172.21.15.7:6789/0 conn(0x7f60b00cf6c0 legacy=0x7f60b0164760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60a7fff640 1 -- 172.21.15.7:0/4072358115 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b00ca8f0 con 0x7f60b00ca4d0 2023-11-28T18:46:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60a7fff640 1 --2- 172.21.15.7:0/4072358115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00ca4d0 0x7f60b015a8a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f60980097f0 tx=0x7f609801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c8afaa67e70589e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60ae7fc640 1 -- 172.21.15.7:0/4072358115 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8001b90 con 0x7f60b00ca4d0 2023-11-28T18:46:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60ae7fc640 1 -- 172.21.15.7:0/4072358115 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60980a7050 con 0x7f60b00ca4d0 2023-11-28T18:46:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.894+0000 7f60ae7fc640 1 -- 172.21.15.7:0/4072358115 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60980ab620 con 0x7f60b00ca4d0 2023-11-28T18:46:13.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- 172.21.15.7:0/4072358115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00ca4d0 msgr2=0x7f60b015a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:13.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 --2- 172.21.15.7:0/4072358115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00ca4d0 0x7f60b015a8a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f60980097f0 tx=0x7f609801c3e0 comp rx=0 tx=0).stop 2023-11-28T18:46:13.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- 172.21.15.7:0/4072358115 shutdown_connections 2023-11-28T18:46:13.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 --2- 172.21.15.7:0/4072358115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00ca4d0 0x7f60b015a8a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:13.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- 172.21.15.7:0/4072358115 >> 172.21.15.7:0/4072358115 conn(0x7f60b00662e0 msgr2=0x7f60b00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:13.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- 172.21.15.7:0/4072358115 shutdown_connections 2023-11-28T18:46:13.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- 172.21.15.7:0/4072358115 wait complete. 2023-11-28T18:46:13.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 Processor -- start 2023-11-28T18:46:13.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- start start 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60b00ca4d0 0x7f60b0144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00cf6c0 0x7f60b0144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60b01540e0 0x7f60b00c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b015ac10 con 0x7f60b01540e0 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b01454e0 con 0x7f60b00ca4d0 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60b55b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b015a650 con 0x7f60b00cf6c0 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60af7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60b01540e0 0x7f60b00c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60af7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60b01540e0 0x7f60b00c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47542/0 (socket says 172.21.15.7:47542) 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60af7fe640 1 -- 172.21.15.7:0/2326320107 learned_addr learned my addr 172.21.15.7:0/2326320107 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:13.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60a7fff640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00cf6c0 0x7f60b0144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.898+0000 7f60aeffd640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60b00ca4d0 0x7f60b0144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60af7fe640 1 -- 172.21.15.7:0/2326320107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00cf6c0 msgr2=0x7f60b0144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60af7fe640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00cf6c0 0x7f60b0144e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60af7fe640 1 -- 172.21.15.7:0/2326320107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60b00ca4d0 msgr2=0x7f60b0144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60af7fe640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60b00ca4d0 0x7f60b0144950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60af7fe640 1 -- 172.21.15.7:0/2326320107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6098009420 con 0x7f60b01540e0 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60af7fe640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60b01540e0 0x7f60b00c6a10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f60a8001bd0 tx=0x7f60a800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60a77fe640 1 -- 172.21.15.7:0/2326320107 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8012600 con 0x7f60b01540e0 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60a77fe640 1 -- 172.21.15.7:0/2326320107 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a8012760 con 0x7f60b01540e0 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b015a930 con 0x7f60b01540e0 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60a77fe640 1 -- 172.21.15.7:0/2326320107 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8012b30 con 0x7f60b01540e0 2023-11-28T18:46:13.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b0146770 con 0x7f60b01540e0 2023-11-28T18:46:13.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b004e890 con 0x7f60b01540e0 2023-11-28T18:46:13.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60a77fe640 1 -- 172.21.15.7:0/2326320107 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f60a8012c90 con 0x7f60b01540e0 2023-11-28T18:46:13.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60a77fe640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6090066900 0x7f6090068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:13.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.902+0000 7f60a77fe640 1 -- 172.21.15.7:0/2326320107 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f60a8085040 con 0x7f60b01540e0 2023-11-28T18:46:13.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.906+0000 7f60aeffd640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6090066900 0x7f6090068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:13.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.906+0000 7f60aeffd640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6090066900 0x7f6090068dc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f60a00011f0 tx=0x7f60a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:13.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:13.906+0000 7f60a77fe640 1 -- 172.21.15.7:0/2326320107 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f60a80b6280 con 0x7f60b01540e0 2023-11-28T18:46:14.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.098+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60b015e380 con 0x7f60b01540e0 2023-11-28T18:46:14.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.098+0000 7f60a77fe640 1 -- 172.21.15.7:0/2326320107 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f60b015e380 con 0x7f60b01540e0 2023-11-28T18:46:14.099 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:14.099 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:14.099 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:14.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6090066900 msgr2=0x7f6090068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:14.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6090066900 0x7f6090068dc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f60a00011f0 tx=0x7f60a0009040 comp rx=0 tx=0).stop 2023-11-28T18:46:14.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60b01540e0 msgr2=0x7f60b00c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:14.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60b01540e0 0x7f60b00c6a10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f60a8001bd0 tx=0x7f60a800a430 comp rx=0 tx=0).stop 2023-11-28T18:46:14.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 shutdown_connections 2023-11-28T18:46:14.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6090066900 0x7f6090068dc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:14.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60b01540e0 0x7f60b00c6a10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:14.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f60b00cf6c0 0x7f60b0144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:14.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 --2- 172.21.15.7:0/2326320107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60b00ca4d0 0x7f60b0144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:14.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 >> 172.21.15.7:0/2326320107 conn(0x7f60b00662e0 msgr2=0x7f60b00cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:14.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 shutdown_connections 2023-11-28T18:46:14.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:14.102+0000 7f60b55b8640 1 -- 172.21.15.7:0/2326320107 wait complete. 2023-11-28T18:46:14.121 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:16.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:16.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.274+0000 7fca15a17640 1 Processor -- start 2023-11-28T18:46:16.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca15a17640 1 -- start start 2023-11-28T18:46:16.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca15a17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fca1006e560 0x7fca1015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:16.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca15a17640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca10157b90 0x7fca100701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:16.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca15a17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca10071e60 con 0x7fca10151cd0 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca15a17640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca10070730 con 0x7fca10157b90 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca15a17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca10070930 con 0x7fca1006e560 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca10157b90 0x7fca100701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0effd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fca10151cd0 0x7fca100c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59894/0 (socket says 172.21.15.7:59894) 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0effd640 1 -- 172.21.15.7:0/4286026891 learned_addr learned my addr 172.21.15.7:0/4286026891 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0e7fc640 1 --2- 172.21.15.7:0/4286026891 >> v2:172.21.15.193:3300/0 conn(0x7fca1006e560 0x7fca1015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0f7fe640 1 -- 172.21.15.7:0/4286026891 >> v2:172.21.15.193:3300/0 conn(0x7fca1006e560 msgr2=0x7fca1015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0f7fe640 1 --2- 172.21.15.7:0/4286026891 >> v2:172.21.15.193:3300/0 conn(0x7fca1006e560 0x7fca1015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0f7fe640 1 -- 172.21.15.7:0/4286026891 >> v1:172.21.15.7:6789/0 conn(0x7fca10151cd0 legacy=0x7fca100c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0f7fe640 1 -- 172.21.15.7:0/4286026891 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10157fa0 con 0x7fca10157b90 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.278+0000 7fca0dffb640 1 -- 172.21.15.7:0/4286026891 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625463144 0 0) 0x7fca10071e60 con 0x7fca10151cd0 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca0f7fe640 1 --2- 172.21.15.7:0/4286026891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10157b90 0x7fca100701f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fca100671b0 tx=0x7fca0801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88aedce4fdbe19e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca0dffb640 1 -- 172.21.15.7:0/4286026891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca080a5830 con 0x7fca10157b90 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca0dffb640 1 -- 172.21.15.7:0/4286026891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca080a5990 con 0x7fca10157b90 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca0dffb640 1 -- 172.21.15.7:0/4286026891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca080a5d20 con 0x7fca10157b90 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 -- 172.21.15.7:0/4286026891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10157b90 msgr2=0x7fca100701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 --2- 172.21.15.7:0/4286026891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10157b90 0x7fca100701f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fca100671b0 tx=0x7fca0801c880 comp rx=0 tx=0).stop 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 -- 172.21.15.7:0/4286026891 shutdown_connections 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 --2- 172.21.15.7:0/4286026891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10157b90 0x7fca100701f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 --2- 172.21.15.7:0/4286026891 >> v2:172.21.15.193:3300/0 conn(0x7fca1006e560 0x7fca1015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 -- 172.21.15.7:0/4286026891 >> 172.21.15.7:0/4286026891 conn(0x7fca100662e0 msgr2=0x7fca100666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 -- 172.21.15.7:0/4286026891 shutdown_connections 2023-11-28T18:46:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 -- 172.21.15.7:0/4286026891 wait complete. 2023-11-28T18:46:16.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.282+0000 7fca15a17640 1 Processor -- start 2023-11-28T18:46:16.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 -- start start 2023-11-28T18:46:16.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca1006e560 0x7fca100c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10151cd0 0x7fca100c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca10157b90 0x7fca100c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca10150b70 con 0x7fca10151cd0 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca101f6650 con 0x7fca1006e560 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1015a980 con 0x7fca10157b90 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10151cd0 0x7fca100c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10151cd0 0x7fca100c5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47556/0 (socket says 172.21.15.7:47556) 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 -- 172.21.15.7:0/1762968982 learned_addr learned my addr 172.21.15.7:0/1762968982 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0effd640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca1006e560 0x7fca100c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0f7fe640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca10157b90 0x7fca100c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 -- 172.21.15.7:0/1762968982 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca10157b90 msgr2=0x7fca100c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca10157b90 0x7fca100c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 -- 172.21.15.7:0/1762968982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca1006e560 msgr2=0x7fca100c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca1006e560 0x7fca100c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 -- 172.21.15.7:0/1762968982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08009a70 con 0x7fca10151cd0 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca0e7fc640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10151cd0 0x7fca100c5bc0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fca0400e930 tx=0x7fca0400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fc9f7fff640 1 -- 172.21.15.7:0/1762968982 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04009690 con 0x7fca10151cd0 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fc9f7fff640 1 -- 172.21.15.7:0/1762968982 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0400ca00 con 0x7fca10151cd0 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1015ac20 con 0x7fca10151cd0 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fc9f7fff640 1 -- 172.21.15.7:0/1762968982 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0400cd70 con 0x7fca10151cd0 2023-11-28T18:46:16.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca101f5d20 con 0x7fca10151cd0 2023-11-28T18:46:16.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.286+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca101f67c0 con 0x7fca10151cd0 2023-11-28T18:46:16.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.290+0000 7fc9f7fff640 1 -- 172.21.15.7:0/1762968982 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fca04013470 con 0x7fca10151cd0 2023-11-28T18:46:16.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.290+0000 7fc9f7fff640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc9fc0668d0 0x7fc9fc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:16.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.290+0000 7fc9f7fff640 1 -- 172.21.15.7:0/1762968982 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fca04084cb0 con 0x7fca10151cd0 2023-11-28T18:46:16.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.290+0000 7fca0effd640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc9fc0668d0 0x7fc9fc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:16.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.290+0000 7fca0effd640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc9fc0668d0 0x7fc9fc068d90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fca00000f70 tx=0x7fca00007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:16.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.294+0000 7fc9f7fff640 1 -- 172.21.15.7:0/1762968982 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fca04051980 con 0x7fca10151cd0 2023-11-28T18:46:16.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.482+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca100c26a0 con 0x7fca10151cd0 2023-11-28T18:46:16.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fc9f7fff640 1 -- 172.21.15.7:0/1762968982 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7fca04017090 con 0x7fca10151cd0 2023-11-28T18:46:16.487 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:16.487 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc9fc0668d0 msgr2=0x7fc9fc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc9fc0668d0 0x7fc9fc068d90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fca00000f70 tx=0x7fca00007b60 comp rx=0 tx=0).stop 2023-11-28T18:46:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10151cd0 msgr2=0x7fca100c5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10151cd0 0x7fca100c5bc0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fca0400e930 tx=0x7fca0400edf0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 shutdown_connections 2023-11-28T18:46:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc9fc0668d0 0x7fc9fc068d90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca10157b90 0x7fca100c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10151cd0 0x7fca100c5bc0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 --2- 172.21.15.7:0/1762968982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca1006e560 0x7fca100c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 >> 172.21.15.7:0/1762968982 conn(0x7fca100662e0 msgr2=0x7fca1014a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 shutdown_connections 2023-11-28T18:46:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:16.486+0000 7fca15a17640 1 -- 172.21.15.7:0/1762968982 wait complete. 2023-11-28T18:46:16.507 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:18.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc597679640 1 Processor -- start 2023-11-28T18:46:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc597679640 1 -- start start 2023-11-28T18:46:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc597679640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc5901542e0 0x7fc59015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc597679640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5900ca920 con 0x7fc590072450 2023-11-28T18:46:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc597679640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59015e550 con 0x7fc5900cbb70 2023-11-28T18:46:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc597679640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc590071d50 con 0x7fc5901542e0 2023-11-28T18:46:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc595c16640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc590072450 0x7fc590164990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59896/0 (socket says 172.21.15.7:59896) 2023-11-28T18:46:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc595c16640 1 -- 172.21.15.7:0/1558729455 learned_addr learned my addr 172.21.15.7:0/1558729455 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc595415640 1 --2- 172.21.15.7:0/1558729455 >> v2:172.21.15.193:3300/0 conn(0x7fc5901542e0 0x7fc59015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:18.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc587fff640 1 -- 172.21.15.7:0/1558729455 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535403833 0 0) 0x7fc5900ca920 con 0x7fc590072450 2023-11-28T18:46:18.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc587fff640 1 -- 172.21.15.7:0/1558729455 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc57c003530 con 0x7fc590072450 2023-11-28T18:46:18.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc595415640 1 -- 172.21.15.7:0/1558729455 >> v1:172.21.15.193:6789/0 conn(0x7fc5900cbb70 legacy=0x7fc5900cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:18.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc595415640 1 -- 172.21.15.7:0/1558729455 >> v1:172.21.15.7:6789/0 conn(0x7fc590072450 legacy=0x7fc590164990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:18.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.666+0000 7fc595415640 1 -- 172.21.15.7:0/1558729455 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5900c4cd0 con 0x7fc5901542e0 2023-11-28T18:46:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc595415640 1 --2- 172.21.15.7:0/1558729455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 0x7fc59015e010 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0097c0 tx=0x7fc58c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3944dc16ca8821b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc587fff640 1 -- 172.21.15.7:0/1558729455 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0ab680 con 0x7fc5901542e0 2023-11-28T18:46:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc587fff640 1 -- 172.21.15.7:0/1558729455 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c0ab7e0 con 0x7fc5901542e0 2023-11-28T18:46:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc587fff640 1 -- 172.21.15.7:0/1558729455 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0abb70 con 0x7fc5901542e0 2023-11-28T18:46:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc597679640 1 -- 172.21.15.7:0/1558729455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 msgr2=0x7fc59015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc597679640 1 --2- 172.21.15.7:0/1558729455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 0x7fc59015e010 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0097c0 tx=0x7fc58c01c480 comp rx=0 tx=0).stop 2023-11-28T18:46:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc597679640 1 -- 172.21.15.7:0/1558729455 shutdown_connections 2023-11-28T18:46:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc597679640 1 --2- 172.21.15.7:0/1558729455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 0x7fc59015e010 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc597679640 1 -- 172.21.15.7:0/1558729455 >> 172.21.15.7:0/1558729455 conn(0x7fc5900717e0 msgr2=0x7fc5900662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc597679640 1 -- 172.21.15.7:0/1558729455 shutdown_connections 2023-11-28T18:46:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.670+0000 7fc597679640 1 -- 172.21.15.7:0/1558729455 wait complete. 2023-11-28T18:46:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 Processor -- start 2023-11-28T18:46:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 -- start start 2023-11-28T18:46:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc590072450 0x7fc590142dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5900cbb70 0x7fc5901f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 0x7fc5901f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc590072030 con 0x7fc5900cbb70 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59015d970 con 0x7fc5901542e0 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5901f7280 con 0x7fc590072450 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5900cbb70 0x7fc5901f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5900cbb70 0x7fc5901f31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47558/0 (socket says 172.21.15.7:47558) 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 -- 172.21.15.7:0/1559252911 learned_addr learned my addr 172.21.15.7:0/1559252911 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc595415640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc590072450 0x7fc590142dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc595c16640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 0x7fc5901f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 -- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc590072450 msgr2=0x7fc590142dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc590072450 0x7fc590142dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 -- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 msgr2=0x7fc5901f36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 0x7fc5901f36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 -- 172.21.15.7:0/1559252911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c009420 con 0x7fc5900cbb70 2023-11-28T18:46:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc595c16640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 0x7fc5901f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc594c14640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5900cbb70 0x7fc5901f31b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc58000c500 tx=0x7fc58000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc585ffb640 1 -- 172.21.15.7:0/1559252911 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc580005220 con 0x7fc5900cbb70 2023-11-28T18:46:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc585ffb640 1 -- 172.21.15.7:0/1559252911 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc580005380 con 0x7fc5900cbb70 2023-11-28T18:46:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5901f6420 con 0x7fc5900cbb70 2023-11-28T18:46:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc585ffb640 1 -- 172.21.15.7:0/1559252911 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc580007270 con 0x7fc5900cbb70 2023-11-28T18:46:18.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.674+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5901f69e0 con 0x7fc5900cbb70 2023-11-28T18:46:18.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.678+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5901f3c30 con 0x7fc5900cbb70 2023-11-28T18:46:18.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.678+0000 7fc585ffb640 1 -- 172.21.15.7:0/1559252911 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc5800073d0 con 0x7fc5900cbb70 2023-11-28T18:46:18.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.678+0000 7fc585ffb640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc5740668d0 0x7fc574068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:18.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.678+0000 7fc585ffb640 1 -- 172.21.15.7:0/1559252911 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fc5800841c0 con 0x7fc5900cbb70 2023-11-28T18:46:18.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.678+0000 7fc595415640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc5740668d0 0x7fc574068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:18.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.678+0000 7fc595415640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc5740668d0 0x7fc574068d90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0af000 tx=0x7fc58c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:18.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.682+0000 7fc585ffb640 1 -- 172.21.15.7:0/1559252911 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc580050ef0 con 0x7fc5900cbb70 2023-11-28T18:46:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.870+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc59006c670 con 0x7fc5900cbb70 2023-11-28T18:46:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc585ffb640 1 -- 172.21.15.7:0/1559252911 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7fc580054510 con 0x7fc5900cbb70 2023-11-28T18:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc5740668d0 msgr2=0x7fc574068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc5740668d0 0x7fc574068d90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0af000 tx=0x7fc58c0a7000 comp rx=0 tx=0).stop 2023-11-28T18:46:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5900cbb70 msgr2=0x7fc5901f31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5900cbb70 0x7fc5901f31b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc58000c500 tx=0x7fc58000c9c0 comp rx=0 tx=0).stop 2023-11-28T18:46:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 shutdown_connections 2023-11-28T18:46:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc5740668d0 0x7fc574068d90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901542e0 0x7fc5901f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:18.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5900cbb70 0x7fc5901f31b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:18.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 --2- 172.21.15.7:0/1559252911 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc590072450 0x7fc590142dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:18.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 >> 172.21.15.7:0/1559252911 conn(0x7fc5900717e0 msgr2=0x7fc590146cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:18.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 shutdown_connections 2023-11-28T18:46:18.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:18.874+0000 7fc597679640 1 -- 172.21.15.7:0/1559252911 wait complete. 2023-11-28T18:46:18.895 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:20.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~61s 2023-11-28T18:46:20.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:21.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e2f1d640 1 Processor -- start 2023-11-28T18:46:21.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e2f1d640 1 -- start start 2023-11-28T18:46:21.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e2f1d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f01dc1540e0 0x7f01dc0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:21.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e2f1d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01dc15a8a0 0x7f01dc15dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:21.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e2f1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 0x7f01dc0cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e2f1d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01dc15d7e0 con 0x7f01dc15a8a0 2023-11-28T18:46:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e2f1d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01dc0cab80 con 0x7f01dc1540e0 2023-11-28T18:46:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e2f1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc0cb850 con 0x7f01dc0cf6c0 2023-11-28T18:46:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01dbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01dc15a8a0 0x7f01dc15dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01dbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01dc15a8a0 0x7f01dc15dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47574/0 (socket says 172.21.15.7:47574) 2023-11-28T18:46:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01dbfff640 1 -- 172.21.15.7:0/312965532 learned_addr learned my addr 172.21.15.7:0/312965532 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e14ba640 1 --2- 172.21.15.7:0/312965532 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 0x7f01dc0cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:21.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01e0cb9640 1 --2- 172.21.15.7:0/312965532 >> v2:172.21.15.193:3300/0 conn(0x7f01dc1540e0 0x7f01dc0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:21.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01dbfff640 1 -- 172.21.15.7:0/312965532 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 msgr2=0x7f01dc0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:21.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01dbfff640 1 --2- 172.21.15.7:0/312965532 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 0x7f01dc0cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01dbfff640 1 -- 172.21.15.7:0/312965532 >> v2:172.21.15.193:3300/0 conn(0x7f01dc1540e0 msgr2=0x7f01dc0ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:21.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01dbfff640 1 --2- 172.21.15.7:0/312965532 >> v2:172.21.15.193:3300/0 conn(0x7f01dc1540e0 0x7f01dc0ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.050+0000 7f01dbfff640 1 -- 172.21.15.7:0/312965532 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc0cbb30 con 0x7f01dc15a8a0 2023-11-28T18:46:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01dbfff640 1 --2- 172.21.15.7:0/312965532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc15dc70 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0cf690 tx=0x7f01c801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30de90d6847456ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01db7fe640 1 -- 172.21.15.7:0/312965532 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c80ac630 con 0x7f01dc15a8a0 2023-11-28T18:46:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01db7fe640 1 -- 172.21.15.7:0/312965532 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c80ac790 con 0x7f01dc15a8a0 2023-11-28T18:46:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01db7fe640 1 -- 172.21.15.7:0/312965532 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c80acb20 con 0x7f01dc15a8a0 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 -- 172.21.15.7:0/312965532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 msgr2=0x7f01dc15dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/312965532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc15dc70 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0cf690 tx=0x7f01c801c500 comp rx=0 tx=0).stop 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 -- 172.21.15.7:0/312965532 shutdown_connections 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/312965532 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 0x7f01dc0cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/312965532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc15dc70 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/312965532 >> v2:172.21.15.193:3300/0 conn(0x7f01dc1540e0 0x7f01dc0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 -- 172.21.15.7:0/312965532 >> 172.21.15.7:0/312965532 conn(0x7f01dc0662e0 msgr2=0x7f01dc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 -- 172.21.15.7:0/312965532 shutdown_connections 2023-11-28T18:46:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.054+0000 7f01e2f1d640 1 -- 172.21.15.7:0/312965532 wait complete. 2023-11-28T18:46:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 Processor -- start 2023-11-28T18:46:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 -- start start 2023-11-28T18:46:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 0x7f01dc157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01dc1540e0 0x7f01dc145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc1f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc15b6a0 con 0x7f01dc15a8a0 2023-11-28T18:46:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc0b4dd0 con 0x7f01dc1540e0 2023-11-28T18:46:21.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc0b50b0 con 0x7f01dc0cf6c0 2023-11-28T18:46:21.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc1f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:21.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc1f5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47584/0 (socket says 172.21.15.7:47584) 2023-11-28T18:46:21.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 -- 172.21.15.7:0/138137906 learned_addr learned my addr 172.21.15.7:0/138137906 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:21.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e0cb9640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 0x7f01dc157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:21.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01dbfff640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01dc1540e0 0x7f01dc145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:21.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 -- 172.21.15.7:0/138137906 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 msgr2=0x7f01dc157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:21.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 0x7f01dc157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 -- 172.21.15.7:0/138137906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01dc1540e0 msgr2=0x7f01dc145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01dc1540e0 0x7f01dc145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 -- 172.21.15.7:0/138137906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c80094d0 con 0x7f01dc15a8a0 2023-11-28T18:46:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e14ba640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc1f5c60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f01d4007e60 tx=0x7f01d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01d97fa640 1 -- 172.21.15.7:0/138137906 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d4014620 con 0x7f01dc15a8a0 2023-11-28T18:46:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01d97fa640 1 -- 172.21.15.7:0/138137906 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d400ee90 con 0x7f01dc15a8a0 2023-11-28T18:46:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01d97fa640 1 -- 172.21.15.7:0/138137906 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d40149e0 con 0x7f01dc15a8a0 2023-11-28T18:46:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01dc0b5390 con 0x7f01dc15a8a0 2023-11-28T18:46:21.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.058+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01dc0cfe80 con 0x7f01dc15a8a0 2023-11-28T18:46:21.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.062+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01dc1f5980 con 0x7f01dc15a8a0 2023-11-28T18:46:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.062+0000 7f01d97fa640 1 -- 172.21.15.7:0/138137906 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f01d4014b40 con 0x7f01dc15a8a0 2023-11-28T18:46:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.062+0000 7f01d97fa640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01c0066920 0x7f01c0068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.062+0000 7f01d97fa640 1 -- 172.21.15.7:0/138137906 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f01d4085020 con 0x7f01dc15a8a0 2023-11-28T18:46:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.062+0000 7f01e0cb9640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01c0066920 0x7f01c0068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.062+0000 7f01e0cb9640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01c0066920 0x7f01c0068de0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f01d0007a60 tx=0x7f01d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:21.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.066+0000 7f01d97fa640 1 -- 172.21.15.7:0/138137906 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01dc1f5980 con 0x7f01dc15a8a0 2023-11-28T18:46:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.254+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01dc1f5980 con 0x7f01dc15a8a0 2023-11-28T18:46:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01d97fa640 1 -- 172.21.15.7:0/138137906 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f01dc1f5980 con 0x7f01dc15a8a0 2023-11-28T18:46:21.257 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:21.257 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:21.257 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01c0066920 msgr2=0x7f01c0068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01c0066920 0x7f01c0068de0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f01d0007a60 tx=0x7f01d0008040 comp rx=0 tx=0).stop 2023-11-28T18:46:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 msgr2=0x7f01dc1f5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc1f5c60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f01d4007e60 tx=0x7f01d400e4b0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 shutdown_connections 2023-11-28T18:46:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f01c0066920 0x7f01c0068de0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01dc15a8a0 0x7f01dc1f5c60 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01dc1540e0 0x7f01dc145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 --2- 172.21.15.7:0/138137906 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01dc0cf6c0 0x7f01dc157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:21.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 >> 172.21.15.7:0/138137906 conn(0x7f01dc0662e0 msgr2=0x7f01dc159610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:21.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 shutdown_connections 2023-11-28T18:46:21.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:21.258+0000 7f01e2f1d640 1 -- 172.21.15.7:0/138137906 wait complete. 2023-11-28T18:46:21.279 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:23.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f50404ea640 1 Processor -- start 2023-11-28T18:46:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f50404ea640 1 -- start start 2023-11-28T18:46:23.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f50404ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:23.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f50404ea640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503815e380 con 0x7f50381540e0 2023-11-28T18:46:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f50404ea640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503815d7e0 con 0x7f50380ca4d0 2023-11-28T18:46:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f50404ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50380ca8f0 con 0x7f503815a8a0 2023-11-28T18:46:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503e286640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f50381540e0 0x7f503815dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59904/0 (socket says 172.21.15.7:59904) 2023-11-28T18:46:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503e286640 1 -- 172.21.15.7:0/4198796459 learned_addr learned my addr 172.21.15.7:0/4198796459 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503ea87640 1 --2- 172.21.15.7:0/4198796459 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503d284640 1 -- 172.21.15.7:0/4198796459 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605982745 0 0) 0x7f5034001120 con 0x7f50381540e0 2023-11-28T18:46:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503d284640 1 -- 172.21.15.7:0/4198796459 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5024003530 con 0x7f50381540e0 2023-11-28T18:46:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503d284640 1 -- 172.21.15.7:0/4198796459 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697538899 0 0) 0x7f503815d7e0 con 0x7f50380ca4d0 2023-11-28T18:46:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503d284640 1 -- 172.21.15.7:0/4198796459 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5034001120 con 0x7f50380ca4d0 2023-11-28T18:46:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503ea87640 1 -- 172.21.15.7:0/4198796459 >> v1:172.21.15.193:6789/0 conn(0x7f50380ca4d0 legacy=0x7f50380cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503ea87640 1 -- 172.21.15.7:0/4198796459 >> v1:172.21.15.7:6789/0 conn(0x7f50381540e0 legacy=0x7f503815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.433+0000 7f503ea87640 1 -- 172.21.15.7:0/4198796459 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503815ad10 con 0x7f503815a8a0 2023-11-28T18:46:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f503ea87640 1 --2- 172.21.15.7:0/4198796459 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038165770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5030009dc0 tx=0x7f503001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c61620909512b6f2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f503d284640 1 -- 172.21.15.7:0/4198796459 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50300a5920 con 0x7f503815a8a0 2023-11-28T18:46:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f503d284640 1 -- 172.21.15.7:0/4198796459 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50300a5a80 con 0x7f503815a8a0 2023-11-28T18:46:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f503d284640 1 -- 172.21.15.7:0/4198796459 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50300a5e10 con 0x7f503815a8a0 2023-11-28T18:46:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f50404ea640 1 -- 172.21.15.7:0/4198796459 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 msgr2=0x7f5038165770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f50404ea640 1 --2- 172.21.15.7:0/4198796459 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038165770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5030009dc0 tx=0x7f503001c970 comp rx=0 tx=0).stop 2023-11-28T18:46:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f50404ea640 1 -- 172.21.15.7:0/4198796459 shutdown_connections 2023-11-28T18:46:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f50404ea640 1 --2- 172.21.15.7:0/4198796459 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038165770 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f50404ea640 1 -- 172.21.15.7:0/4198796459 >> 172.21.15.7:0/4198796459 conn(0x7f50380662e0 msgr2=0x7f50380666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f50404ea640 1 -- 172.21.15.7:0/4198796459 shutdown_connections 2023-11-28T18:46:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.437+0000 7f50404ea640 1 -- 172.21.15.7:0/4198796459 wait complete. 2023-11-28T18:46:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 Processor -- start 2023-11-28T18:46:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 -- start start 2023-11-28T18:46:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380ca4d0 0x7f50380b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50381540e0 0x7f5038156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50381560d0 con 0x7f50380ca4d0 2023-11-28T18:46:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50381564a0 con 0x7f50381540e0 2023-11-28T18:46:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5038154790 con 0x7f503815a8a0 2023-11-28T18:46:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380ca4d0 0x7f50380b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380ca4d0 0x7f50380b5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47592/0 (socket says 172.21.15.7:47592) 2023-11-28T18:46:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 -- 172.21.15.7:0/310041819 learned_addr learned my addr 172.21.15.7:0/310041819 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503ea87640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503da85640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50381540e0 0x7f5038156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 -- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 msgr2=0x7f5038156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 -- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50381540e0 msgr2=0x7f5038156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50381540e0 0x7f5038156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 -- 172.21.15.7:0/310041819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5030009a70 con 0x7f50380ca4d0 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503ea87640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503da85640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50381540e0 0x7f5038156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f503e286640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380ca4d0 0x7f50380b5010 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f503400c700 tx=0x7f503400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f502effd640 1 -- 172.21.15.7:0/310041819 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034005ac0 con 0x7f50380ca4d0 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f502effd640 1 -- 172.21.15.7:0/310041819 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5034005c20 con 0x7f50380ca4d0 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f502effd640 1 -- 172.21.15.7:0/310041819 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034007070 con 0x7f50380ca4d0 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5038155030 con 0x7f50380ca4d0 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.441+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503815ccc0 con 0x7f50380ca4d0 2023-11-28T18:46:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.445+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503804e890 con 0x7f50380ca4d0 2023-11-28T18:46:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.445+0000 7f502effd640 1 -- 172.21.15.7:0/310041819 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f50340071d0 con 0x7f50380ca4d0 2023-11-28T18:46:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.445+0000 7f502effd640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f501c066900 0x7f501c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.445+0000 7f502effd640 1 -- 172.21.15.7:0/310041819 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f5034083eb0 con 0x7f50380ca4d0 2023-11-28T18:46:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.445+0000 7f503da85640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f501c066900 0x7f501c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:23.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.445+0000 7f503da85640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f501c066900 0x7f501c068dc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5038151650 tx=0x7f5028009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:23.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.449+0000 7f502effd640 1 -- 172.21.15.7:0/310041819 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5034050c70 con 0x7f50380ca4d0 2023-11-28T18:46:23.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.641+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f503815e380 con 0x7f50380ca4d0 2023-11-28T18:46:23.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.641+0000 7f502effd640 1 -- 172.21.15.7:0/310041819 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f503815e380 con 0x7f50380ca4d0 2023-11-28T18:46:23.646 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:23.646 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:23.647 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:23.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f501c066900 msgr2=0x7f501c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:23.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f501c066900 0x7f501c068dc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5038151650 tx=0x7f5028009040 comp rx=0 tx=0).stop 2023-11-28T18:46:23.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380ca4d0 msgr2=0x7f50380b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380ca4d0 0x7f50380b5010 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f503400c700 tx=0x7f503400cbc0 comp rx=0 tx=0).stop 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 shutdown_connections 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f501c066900 0x7f501c068dc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f503815a8a0 0x7f5038156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50381540e0 0x7f5038156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 --2- 172.21.15.7:0/310041819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380ca4d0 0x7f50380b5010 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 >> 172.21.15.7:0/310041819 conn(0x7f50380662e0 msgr2=0x7f50380c5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 shutdown_connections 2023-11-28T18:46:23.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:23.645+0000 7f50404ea640 1 -- 172.21.15.7:0/310041819 wait complete. 2023-11-28T18:46:23.666 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:45:57.508800+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:46:25.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.821+0000 7f52f84e3640 1 Processor -- start 2023-11-28T18:46:25.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.821+0000 7f52f84e3640 1 -- start start 2023-11-28T18:46:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f84e3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f00c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f84e3640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f52f00c3490 0x7f52f00c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f84e3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00c3920 con 0x7f52f00c3bc0 2023-11-28T18:46:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f84e3640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f0157240 con 0x7f52f00c3490 2023-11-28T18:46:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f84e3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00c2ce0 con 0x7f52f01503c0 2023-11-28T18:46:25.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f6a80640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f52f00c3490 0x7f52f00c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:25.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f627f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f00c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f627f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f00c7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:53886/0 (socket says 172.21.15.7:53886) 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f627f640 1 -- 172.21.15.7:0/4250018016 learned_addr learned my addr 172.21.15.7:0/4250018016 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f5a7e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f52f00c3bc0 0x7f52f006d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:53954/0 (socket says 172.21.15.7:53954) 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f6a80640 1 -- 172.21.15.7:0/4250018016 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 msgr2=0x7f52f00c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f6a80640 1 --2- 172.21.15.7:0/4250018016 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f00c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f6a80640 1 -- 172.21.15.7:0/4250018016 >> v1:172.21.15.193:6789/0 conn(0x7f52f00c3bc0 legacy=0x7f52f006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f6a80640 1 -- 172.21.15.7:0/4250018016 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00c2f80 con 0x7f52f00c3490 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f627f640 1 --2- 172.21.15.7:0/4250018016 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f00c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:46:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.825+0000 7f52f527d640 1 -- 172.21.15.7:0/4250018016 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183505425 0 0) 0x7f52e0001120 con 0x7f52f00c3bc0 2023-11-28T18:46:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f6a80640 1 --2- 172.21.15.7:0/4250018016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3490 0x7f52f00c46f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009d90 tx=0x7f52e801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed645fb4fc010ef6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f527d640 1 -- 172.21.15.7:0/4250018016 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e80af600 con 0x7f52f00c3490 2023-11-28T18:46:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f527d640 1 -- 172.21.15.7:0/4250018016 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e80ab050 con 0x7f52f00c3490 2023-11-28T18:46:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f527d640 1 -- 172.21.15.7:0/4250018016 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e80af980 con 0x7f52f00c3490 2023-11-28T18:46:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f84e3640 1 -- 172.21.15.7:0/4250018016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3490 msgr2=0x7f52f00c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f84e3640 1 --2- 172.21.15.7:0/4250018016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3490 0x7f52f00c46f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009d90 tx=0x7f52e801f130 comp rx=0 tx=0).stop 2023-11-28T18:46:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f84e3640 1 -- 172.21.15.7:0/4250018016 shutdown_connections 2023-11-28T18:46:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f84e3640 1 --2- 172.21.15.7:0/4250018016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3490 0x7f52f00c46f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f84e3640 1 --2- 172.21.15.7:0/4250018016 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f00c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f84e3640 1 -- 172.21.15.7:0/4250018016 >> 172.21.15.7:0/4250018016 conn(0x7f52f00662e0 msgr2=0x7f52f00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f84e3640 1 -- 172.21.15.7:0/4250018016 shutdown_connections 2023-11-28T18:46:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.829+0000 7f52f84e3640 1 -- 172.21.15.7:0/4250018016 wait complete. 2023-11-28T18:46:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 Processor -- start 2023-11-28T18:46:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 -- start start 2023-11-28T18:46:25.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f00c3490 0x7f52f01480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3bc0 0x7f52f00cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f0149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00ca430 con 0x7f52f00c3490 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00cb140 con 0x7f52f00c3bc0 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00cada0 con 0x7f52f01503c0 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f00c3490 0x7f52f01480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f00c3490 0x7f52f01480b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53036/0 (socket says 172.21.15.7:53036) 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 -- 172.21.15.7:0/2726302017 learned_addr learned my addr 172.21.15.7:0/2726302017 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f5a7e640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3bc0 0x7f52f00cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f6a80640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f0149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 -- 172.21.15.7:0/2726302017 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 msgr2=0x7f52f0149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:25.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f0149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:25.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 -- 172.21.15.7:0/2726302017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3bc0 msgr2=0x7f52f00cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:25.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3bc0 0x7f52f00cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:25.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 -- 172.21.15.7:0/2726302017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0094d0 con 0x7f52f00c3490 2023-11-28T18:46:25.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f627f640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f00c3490 0x7f52f01480b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00e930 tx=0x7f52ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:25.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52e6ffd640 1 -- 172.21.15.7:0/2726302017 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec00ca00 con 0x7f52f00c3490 2023-11-28T18:46:25.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52e6ffd640 1 -- 172.21.15.7:0/2726302017 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec00cb60 con 0x7f52f00c3490 2023-11-28T18:46:25.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52e6ffd640 1 -- 172.21.15.7:0/2726302017 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec013470 con 0x7f52f00c3490 2023-11-28T18:46:25.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e8009a70 con 0x7f52f00c3490 2023-11-28T18:46:25.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f01f3590 con 0x7f52f00c3490 2023-11-28T18:46:25.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.833+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f00cb3b0 con 0x7f52f00c3490 2023-11-28T18:46:25.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.837+0000 7f52e6ffd640 1 -- 172.21.15.7:0/2726302017 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f52ec0135d0 con 0x7f52f00c3490 2023-11-28T18:46:25.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.837+0000 7f52e6ffd640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52d40668d0 0x7f52d4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:25.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.837+0000 7f52e6ffd640 1 -- 172.21.15.7:0/2726302017 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f52ec084aa0 con 0x7f52f00c3490 2023-11-28T18:46:25.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.837+0000 7f52f5a7e640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52d40668d0 0x7f52d4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:25.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.837+0000 7f52f5a7e640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52d40668d0 0x7f52d4068d90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f52e0007fd0 tx=0x7f52e0007ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:25.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:25.841+0000 7f52e6ffd640 1 -- 172.21.15.7:0/2726302017 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f52ec051910 con 0x7f52f00c3490 2023-11-28T18:46:26.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.029+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52f004e890 con 0x7f52f00c3490 2023-11-28T18:46:26.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.029+0000 7f52e6ffd640 1 -- 172.21.15.7:0/2726302017 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f52ec054f30 con 0x7f52f00c3490 2023-11-28T18:46:26.034 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:26.034 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:26.036 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:26.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52d40668d0 msgr2=0x7f52d4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:26.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52d40668d0 0x7f52d4068d90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f52e0007fd0 tx=0x7f52e0007ba0 comp rx=0 tx=0).stop 2023-11-28T18:46:26.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f00c3490 msgr2=0x7f52f01480b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:26.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f00c3490 0x7f52f01480b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00e930 tx=0x7f52ec00edf0 comp rx=0 tx=0).stop 2023-11-28T18:46:26.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 shutdown_connections 2023-11-28T18:46:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52d40668d0 0x7f52d4068d90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52f01503c0 0x7f52f0149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52f00c3bc0 0x7f52f00cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 --2- 172.21.15.7:0/2726302017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52f00c3490 0x7f52f01480b0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 >> 172.21.15.7:0/2726302017 conn(0x7f52f00662e0 msgr2=0x7f52f00666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:26.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 shutdown_connections 2023-11-28T18:46:26.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:26.033+0000 7f52f84e3640 1 -- 172.21.15.7:0/2726302017 wait complete. 2023-11-28T18:46:26.054 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:27.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~68s 2023-11-28T18:46:28.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:28.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0f92a640 1 Processor -- start 2023-11-28T18:46:28.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0f92a640 1 -- start start 2023-11-28T18:46:28.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0f92a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080ca4d0 0x7f7c0815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0f92a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0815e380 con 0x7f7c080cf6c0 2023-11-28T18:46:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0f92a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0815d7e0 con 0x7f7c081540e0 2023-11-28T18:46:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0f92a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0815ade0 con 0x7f7c080ca4d0 2023-11-28T18:46:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0dec7640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7c080cf6c0 0x7f7c08164760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46158/0 (socket says 172.21.15.7:46158) 2023-11-28T18:46:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0dec7640 1 -- 172.21.15.7:0/3547949632 learned_addr learned my addr 172.21.15.7:0/3547949632 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0cec5640 1 --2- 172.21.15.7:0/3547949632 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080ca4d0 0x7f7c0815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7bfffff640 1 -- 172.21.15.7:0/3547949632 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509304699 0 0) 0x7f7c000014a0 con 0x7f7c080cf6c0 2023-11-28T18:46:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7bfffff640 1 -- 172.21.15.7:0/3547949632 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf4003530 con 0x7f7c080cf6c0 2023-11-28T18:46:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7bfffff640 1 -- 172.21.15.7:0/3547949632 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601494289 0 0) 0x7f7c0815d7e0 con 0x7f7c081540e0 2023-11-28T18:46:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0cec5640 1 -- 172.21.15.7:0/3547949632 >> v1:172.21.15.193:6789/0 conn(0x7f7c081540e0 legacy=0x7f7c0815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0cec5640 1 -- 172.21.15.7:0/3547949632 >> v1:172.21.15.7:6789/0 conn(0x7f7c080cf6c0 legacy=0x7f7c08164760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.213+0000 7f7c0cec5640 1 -- 172.21.15.7:0/3547949632 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c080ca8f0 con 0x7f7c080ca4d0 2023-11-28T18:46:28.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7c0cec5640 1 --2- 172.21.15.7:0/3547949632 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080ca4d0 0x7f7c0815a8a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80097f0 tx=0x7f7bf801c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e2e0dcb4a608687 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7bfffff640 1 -- 172.21.15.7:0/3547949632 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf80ac410 con 0x7f7c080ca4d0 2023-11-28T18:46:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7bfffff640 1 -- 172.21.15.7:0/3547949632 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf80ac570 con 0x7f7c080ca4d0 2023-11-28T18:46:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7bfffff640 1 -- 172.21.15.7:0/3547949632 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf80ac8e0 con 0x7f7c080ca4d0 2023-11-28T18:46:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7c0f92a640 1 -- 172.21.15.7:0/3547949632 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080ca4d0 msgr2=0x7f7c0815a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:28.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7c0f92a640 1 --2- 172.21.15.7:0/3547949632 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080ca4d0 0x7f7c0815a8a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80097f0 tx=0x7f7bf801c4c0 comp rx=0 tx=0).stop 2023-11-28T18:46:28.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7c0f92a640 1 -- 172.21.15.7:0/3547949632 shutdown_connections 2023-11-28T18:46:28.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7c0f92a640 1 --2- 172.21.15.7:0/3547949632 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080ca4d0 0x7f7c0815a8a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:28.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.217+0000 7f7c0f92a640 1 -- 172.21.15.7:0/3547949632 >> 172.21.15.7:0/3547949632 conn(0x7f7c080662e0 msgr2=0x7f7c080666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:28.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 -- 172.21.15.7:0/3547949632 shutdown_connections 2023-11-28T18:46:28.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 -- 172.21.15.7:0/3547949632 wait complete. 2023-11-28T18:46:28.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 Processor -- start 2023-11-28T18:46:28.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 -- start start 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c080ca4d0 0x7f7c080b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080cf6c0 0x7f7c081503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c081540e0 0x7f7c081f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c081f6a30 con 0x7f7c081540e0 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c080b4260 con 0x7f7c080ca4d0 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0f92a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c081f5cb0 con 0x7f7c080cf6c0 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0dec7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c081540e0 0x7f7c081f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0dec7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c081540e0 0x7f7c081f2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53040/0 (socket says 172.21.15.7:53040) 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0dec7640 1 -- 172.21.15.7:0/1186041496 learned_addr learned my addr 172.21.15.7:0/1186041496 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0d6c6640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c080ca4d0 0x7f7c080b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0cec5640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080cf6c0 0x7f7c081503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0dec7640 1 -- 172.21.15.7:0/1186041496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080cf6c0 msgr2=0x7f7c081503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0dec7640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080cf6c0 0x7f7c081503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0dec7640 1 -- 172.21.15.7:0/1186041496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c080ca4d0 msgr2=0x7f7c080b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0dec7640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c080ca4d0 0x7f7c080b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.221+0000 7f7c0dec7640 1 -- 172.21.15.7:0/1186041496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf8009420 con 0x7f7c081540e0 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7c0dec7640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c081540e0 0x7f7c081f2610 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000cb40 tx=0x7f7c0000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7bfdffb640 1 -- 172.21.15.7:0/1186041496 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0000aca0 con 0x7f7c081540e0 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7bfdffb640 1 -- 172.21.15.7:0/1186041496 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0000ae00 con 0x7f7c081540e0 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c080b3f80 con 0x7f7c081540e0 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7bfdffb640 1 -- 172.21.15.7:0/1186041496 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00012620 con 0x7f7c081540e0 2023-11-28T18:46:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c081f6230 con 0x7f7c081540e0 2023-11-28T18:46:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0804e890 con 0x7f7c081540e0 2023-11-28T18:46:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7bfdffb640 1 -- 172.21.15.7:0/1186041496 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7c00007070 con 0x7f7c081540e0 2023-11-28T18:46:28.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7bfdffb640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bec0668d0 0x7f7bec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:28.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7bfdffb640 1 -- 172.21.15.7:0/1186041496 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f7c00084780 con 0x7f7c081540e0 2023-11-28T18:46:28.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.225+0000 7f7c0d6c6640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bec0668d0 0x7f7bec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:28.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.229+0000 7f7c0d6c6640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bec0668d0 0x7f7bec068d90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7c04007fd0 tx=0x7f7c04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.229+0000 7f7bfdffb640 1 -- 172.21.15.7:0/1186041496 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7c00051550 con 0x7f7c081540e0 2023-11-28T18:46:28.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c0815e380 con 0x7f7c081540e0 2023-11-28T18:46:28.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7bfdffb640 1 -- 172.21.15.7:0/1186041496 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f7c0815e380 con 0x7f7c081540e0 2023-11-28T18:46:28.426 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:28.426 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:28.427 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:28.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bec0668d0 msgr2=0x7f7bec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:28.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bec0668d0 0x7f7bec068d90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7c04007fd0 tx=0x7f7c04009040 comp rx=0 tx=0).stop 2023-11-28T18:46:28.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c081540e0 msgr2=0x7f7c081f2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c081540e0 0x7f7c081f2610 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000cb40 tx=0x7f7c0000a060 comp rx=0 tx=0).stop 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 shutdown_connections 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bec0668d0 0x7f7bec068d90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c081540e0 0x7f7c081f2610 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7c080cf6c0 0x7f7c081503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 --2- 172.21.15.7:0/1186041496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c080ca4d0 0x7f7c080b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 >> 172.21.15.7:0/1186041496 conn(0x7f7c080662e0 msgr2=0x7f7c08152200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 shutdown_connections 2023-11-28T18:46:28.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:28.425+0000 7f7c0f92a640 1 -- 172.21.15.7:0/1186041496 wait complete. 2023-11-28T18:46:28.448 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:30.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.597+0000 7fcccc929640 1 Processor -- start 2023-11-28T18:46:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.597+0000 7fcccc929640 1 -- start start 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.597+0000 7fcccc929640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fccc815ca30 0x7fccc8159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fcccc929640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccc815ce10 0x7fccc80ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fcccc929640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc815a620 con 0x7fccc80cafe0 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fcccc929640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc815a900 con 0x7fccc815ce10 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fcccc929640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc815abe0 con 0x7fccc815ca30 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccc815ce10 0x7fccc80ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc6d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fccc815ca30 0x7fccc8159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccc815ce10 0x7fccc80ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53042/0 (socket says 172.21.15.7:53042) 2023-11-28T18:46:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc659c640 1 -- 172.21.15.7:0/2659546026 learned_addr learned my addr 172.21.15.7:0/2659546026 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc659c640 1 -- 172.21.15.7:0/2659546026 >> v2:172.21.15.193:3300/0 conn(0x7fccc815ca30 msgr2=0x7fccc8159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc659c640 1 --2- 172.21.15.7:0/2659546026 >> v2:172.21.15.193:3300/0 conn(0x7fccc815ca30 0x7fccc8159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc659c640 1 -- 172.21.15.7:0/2659546026 >> v1:172.21.15.193:6789/0 conn(0x7fccc80cafe0 legacy=0x7fccc80cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc659c640 1 -- 172.21.15.7:0/2659546026 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80cbb10 con 0x7fccc815ce10 2023-11-28T18:46:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.601+0000 7fccc659c640 1 --2- 172.21.15.7:0/2659546026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc80ca1d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fccc806c660 tx=0x7fccbc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2b04f4e7fca29e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fccc5d9b640 1 -- 172.21.15.7:0/2659546026 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0ac640 con 0x7fccc815ce10 2023-11-28T18:46:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fccc5d9b640 1 -- 172.21.15.7:0/2659546026 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc0ac7a0 con 0x7fccc815ce10 2023-11-28T18:46:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fccc5d9b640 1 -- 172.21.15.7:0/2659546026 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0acb30 con 0x7fccc815ce10 2023-11-28T18:46:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- 172.21.15.7:0/2659546026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 msgr2=0x7fccc80ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 --2- 172.21.15.7:0/2659546026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc80ca1d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fccc806c660 tx=0x7fccbc01c4d0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- 172.21.15.7:0/2659546026 shutdown_connections 2023-11-28T18:46:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 --2- 172.21.15.7:0/2659546026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc80ca1d0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 --2- 172.21.15.7:0/2659546026 >> v2:172.21.15.193:3300/0 conn(0x7fccc815ca30 0x7fccc8159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- 172.21.15.7:0/2659546026 >> 172.21.15.7:0/2659546026 conn(0x7fccc806c060 msgr2=0x7fccc80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- 172.21.15.7:0/2659546026 shutdown_connections 2023-11-28T18:46:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- 172.21.15.7:0/2659546026 wait complete. 2023-11-28T18:46:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 Processor -- start 2023-11-28T18:46:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- start start 2023-11-28T18:46:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccc80cafe0 0x7fccc8156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc815ca30 0x7fccc81f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc81581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc8163910 con 0x7fccc815ce10 2023-11-28T18:46:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc81f33b0 con 0x7fccc815ca30 2023-11-28T18:46:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.605+0000 7fcccc929640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc81556f0 con 0x7fccc80cafe0 2023-11-28T18:46:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc81581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc81581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53046/0 (socket says 172.21.15.7:53046) 2023-11-28T18:46:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 -- 172.21.15.7:0/3342755171 learned_addr learned my addr 172.21.15.7:0/3342755171 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc659c640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc815ca30 0x7fccc81f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc6d9d640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccc80cafe0 0x7fccc8156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 -- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccc80cafe0 msgr2=0x7fccc8156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccc80cafe0 0x7fccc8156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 -- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc815ca30 msgr2=0x7fccc81f2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc815ca30 0x7fccc81f2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 -- 172.21.15.7:0/3342755171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc0094d0 con 0x7fccc815ce10 2023-11-28T18:46:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc659c640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc815ca30 0x7fccc81f2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccc759e640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc81581a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fccc000c760 tx=0x7fccc000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccaf7fe640 1 -- 172.21.15.7:0/3342755171 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0003b80 con 0x7fccc815ce10 2023-11-28T18:46:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccaf7fe640 1 -- 172.21.15.7:0/3342755171 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0003ce0 con 0x7fccc815ce10 2023-11-28T18:46:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccaf7fe640 1 -- 172.21.15.7:0/3342755171 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc000aa30 con 0x7fccc815ce10 2023-11-28T18:46:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc81572b0 con 0x7fccc815ce10 2023-11-28T18:46:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc8155c40 con 0x7fccc815ce10 2023-11-28T18:46:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc804e890 con 0x7fccc815ce10 2023-11-28T18:46:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccaf7fe640 1 -- 172.21.15.7:0/3342755171 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fccc000ab90 con 0x7fccc815ce10 2023-11-28T18:46:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.609+0000 7fccaf7fe640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb4066900 0x7fccb4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.613+0000 7fccaf7fe640 1 -- 172.21.15.7:0/3342755171 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fccc00846c0 con 0x7fccc815ce10 2023-11-28T18:46:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.613+0000 7fccc6d9d640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb4066900 0x7fccb4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.613+0000 7fccc6d9d640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb4066900 0x7fccb4068dc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009a50 tx=0x7fccb8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:30.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.617+0000 7fccaf7fe640 1 -- 172.21.15.7:0/3342755171 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fccc00513e0 con 0x7fccc815ce10 2023-11-28T18:46:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.805+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccc80cad10 con 0x7fccc815ce10 2023-11-28T18:46:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.805+0000 7fccaf7fe640 1 -- 172.21.15.7:0/3342755171 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7fccc0054a00 con 0x7fccc815ce10 2023-11-28T18:46:30.808 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:30.809 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:30.809 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:30.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb4066900 msgr2=0x7fccb4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:30.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb4066900 0x7fccb4068dc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009a50 tx=0x7fccb8008040 comp rx=0 tx=0).stop 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 msgr2=0x7fccc81581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc81581a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fccc000c760 tx=0x7fccc000cc20 comp rx=0 tx=0).stop 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 shutdown_connections 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fccb4066900 0x7fccb4068dc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc815ce10 0x7fccc81581a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc815ca30 0x7fccc81f2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 --2- 172.21.15.7:0/3342755171 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fccc80cafe0 0x7fccc8156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 >> 172.21.15.7:0/3342755171 conn(0x7fccc806c060 msgr2=0x7fccc8153cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 shutdown_connections 2023-11-28T18:46:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:30.809+0000 7fcccc929640 1 -- 172.21.15.7:0/3342755171 wait complete. 2023-11-28T18:46:30.831 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:32.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:32.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01ab32a640 1 Processor -- start 2023-11-28T18:46:32.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01ab32a640 1 -- start start 2023-11-28T18:46:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01ab32a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f01a415a8a0 0x7f01a4165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01ab32a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a415e380 con 0x7f01a41540e0 2023-11-28T18:46:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01ab32a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a415d7e0 con 0x7f01a40ca4d0 2023-11-28T18:46:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01ab32a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a40ca8f0 con 0x7f01a415a8a0 2023-11-28T18:46:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01a90c6640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f01a41540e0 0x7f01a415dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46162/0 (socket says 172.21.15.7:46162) 2023-11-28T18:46:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01a90c6640 1 -- 172.21.15.7:0/2762525604 learned_addr learned my addr 172.21.15.7:0/2762525604 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862804327 0 0) 0x7f01a0001120 con 0x7f01a41540e0 2023-11-28T18:46:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f01a98c7640 1 --2- 172.21.15.7:0/2762525604 >> v2:172.21.15.193:3300/0 conn(0x7f01a415a8a0 0x7f01a4165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.985+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0190003530 con 0x7f01a41540e0 2023-11-28T18:46:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 472123085 0 0) 0x7f0190003530 con 0x7f01a41540e0 2023-11-28T18:46:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 >> v2:172.21.15.193:3300/0 conn(0x7f01a415a8a0 msgr2=0x7f01a4165770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 --2- 172.21.15.7:0/2762525604 >> v2:172.21.15.193:3300/0 conn(0x7f01a415a8a0 0x7f01a4165770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 >> v1:172.21.15.193:6789/0 conn(0x7f01a40ca4d0 legacy=0x7f01a40cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a40cab80 con 0x7f01a41540e0 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f01a0001810 con 0x7f01a41540e0 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 >> v1:172.21.15.7:6789/0 conn(0x7f01a41540e0 legacy=0x7f01a415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01900039d0 0x7f0190003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f0190004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0190005190 0x7f019000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a40ca8f0 con 0x7f01900047d0 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019000c090 con 0x7f01900039d0 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019000c250 con 0x7f0190005190 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a88c5640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f0190004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a90c6640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01900039d0 0x7f0190003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a98c7640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0190005190 0x7f019000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a88c5640 1 -- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0190005190 msgr2=0x7f019000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a88c5640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0190005190 0x7f019000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a88c5640 1 -- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01900039d0 msgr2=0x7f0190003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a88c5640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01900039d0 0x7f0190003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a88c5640 1 -- 172.21.15.7:0/2762525604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190003530 con 0x7f01900047d0 2023-11-28T18:46:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a90c6640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01900039d0 0x7f0190003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:46:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.989+0000 7f01a98c7640 1 -- 172.21.15.7:0/2762525604 reap_dead start 2023-11-28T18:46:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01a88c5640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f0190004c50 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0194001780 tx=0x7f019401c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e06e9f897f078e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01940052d0 con 0x7f01900047d0 2023-11-28T18:46:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01940a7050 con 0x7f01900047d0 2023-11-28T18:46:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f019bfff640 1 -- 172.21.15.7:0/2762525604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194003600 con 0x7f01900047d0 2023-11-28T18:46:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 -- 172.21.15.7:0/2762525604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 msgr2=0x7f0190004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f0190004c50 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0194001780 tx=0x7f019401c390 comp rx=0 tx=0).stop 2023-11-28T18:46:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 -- 172.21.15.7:0/2762525604 shutdown_connections 2023-11-28T18:46:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 --2- 172.21.15.7:0/2762525604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f0190004c50 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 -- 172.21.15.7:0/2762525604 >> 172.21.15.7:0/2762525604 conn(0x7f01a40662e0 msgr2=0x7f01a40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 -- 172.21.15.7:0/2762525604 shutdown_connections 2023-11-28T18:46:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 -- 172.21.15.7:0/2762525604 wait complete. 2023-11-28T18:46:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 Processor -- start 2023-11-28T18:46:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.993+0000 7f01ab32a640 1 -- start start 2023-11-28T18:46:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f01a41594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01a4157230 0x7f01a41e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a41e51b0 0x7f01a4149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a41e4850 con 0x7f01900047d0 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a41e5630 con 0x7f01a41e51b0 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a41f05d0 con 0x7f01a4157230 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f01a41594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f01a41594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53064/0 (socket says 172.21.15.7:53064) 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 -- 172.21.15.7:0/3423289750 learned_addr learned my addr 172.21.15.7:0/3423289750 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a88c5640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01a4157230 0x7f01a41e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 -- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01a4157230 msgr2=0x7f01a41e4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a98c7640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a41e51b0 0x7f01a4149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01a4157230 0x7f01a41e4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:33.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 -- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a41e51b0 msgr2=0x7f01a4149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:33.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a41e51b0 0x7f01a4149e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:33.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 -- 172.21.15.7:0/3423289750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a41f08b0 con 0x7f01900047d0 2023-11-28T18:46:33.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a88c5640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01a4157230 0x7f01a41e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:33.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a98c7640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a41e51b0 0x7f01a4149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:33.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01a90c6640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f01a41594e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f01a0000c10 tx=0x7f01a0012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:33.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f0199ffb640 1 -- 172.21.15.7:0/3423289750 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0014750 con 0x7f01900047d0 2023-11-28T18:46:33.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f0199ffb640 1 -- 172.21.15.7:0/3423289750 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a00148b0 con 0x7f01900047d0 2023-11-28T18:46:33.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a40c6900 con 0x7f01900047d0 2023-11-28T18:46:33.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f0199ffb640 1 -- 172.21.15.7:0/3423289750 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0014c20 con 0x7f01900047d0 2023-11-28T18:46:33.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a40c6ec0 con 0x7f01900047d0 2023-11-28T18:46:33.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a404e890 con 0x7f01900047d0 2023-11-28T18:46:33.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:32.997+0000 7f0199ffb640 1 -- 172.21.15.7:0/3423289750 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f01a00153e0 con 0x7f01900047d0 2023-11-28T18:46:33.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.001+0000 7f0199ffb640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0188066900 0x7f0188068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:33.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.001+0000 7f0199ffb640 1 -- 172.21.15.7:0/3423289750 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f01a00871e0 con 0x7f01900047d0 2023-11-28T18:46:33.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.001+0000 7f01a88c5640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0188066900 0x7f0188068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:33.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.001+0000 7f01a88c5640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0188066900 0x7f0188068dc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0194001750 tx=0x7f01940011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:33.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.005+0000 7f0199ffb640 1 -- 172.21.15.7:0/3423289750 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01a0053f80 con 0x7f01900047d0 2023-11-28T18:46:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01a415e380 con 0x7f01900047d0 2023-11-28T18:46:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f0199ffb640 1 -- 172.21.15.7:0/3423289750 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f01a415e380 con 0x7f01900047d0 2023-11-28T18:46:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:33.195 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0188066900 msgr2=0x7f0188068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0188066900 0x7f0188068dc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0194001750 tx=0x7f01940011a0 comp rx=0 tx=0).stop 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 msgr2=0x7f01a41594e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f01a41594e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f01a0000c10 tx=0x7f01a0012920 comp rx=0 tx=0).stop 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 shutdown_connections 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0188066900 0x7f0188068dc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a41e51b0 0x7f01a4149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01a4157230 0x7f01a41e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 --2- 172.21.15.7:0/3423289750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01900047d0 0x7f01a41594e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 >> 172.21.15.7:0/3423289750 conn(0x7f01a40662e0 msgr2=0x7f01a4071e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.193+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 shutdown_connections 2023-11-28T18:46:33.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:33.197+0000 7f01ab32a640 1 -- 172.21.15.7:0/3423289750 wait complete. 2023-11-28T18:46:33.217 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:33.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~74s 2023-11-28T18:46:35.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.369+0000 7f5389d80640 1 Processor -- start 2023-11-28T18:46:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5389d80640 1 -- start start 2023-11-28T18:46:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5389d80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5384151cd0 0x7f538415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5389d80640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5384157b90 0x7f53840701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5389d80640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384158000 con 0x7f538406e560 2023-11-28T18:46:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5389d80640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5384070730 con 0x7f5384157b90 2023-11-28T18:46:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5389d80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5384071e60 con 0x7f5384151cd0 2023-11-28T18:46:35.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5383fff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f5384157b90 0x7f53840701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:35.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f53837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5384151cd0 0x7f538415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:35.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5382ffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f538406e560 0x7f53840c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:46722/0 (socket says 172.21.15.7:46722) 2023-11-28T18:46:35.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5382ffd640 1 -- 172.21.15.7:0/944384979 learned_addr learned my addr 172.21.15.7:0/944384979 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:35.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f53837fe640 1 -- 172.21.15.7:0/944384979 >> v2:172.21.15.193:3300/0 conn(0x7f5384157b90 msgr2=0x7f53840701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f53837fe640 1 --2- 172.21.15.7:0/944384979 >> v2:172.21.15.193:3300/0 conn(0x7f5384157b90 0x7f53840701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f53837fe640 1 -- 172.21.15.7:0/944384979 >> v1:172.21.15.193:6789/0 conn(0x7f538406e560 legacy=0x7f53840c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f53837fe640 1 -- 172.21.15.7:0/944384979 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384070930 con 0x7f5384151cd0 2023-11-28T18:46:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.373+0000 7f5383fff640 1 --2- 172.21.15.7:0/944384979 >> v2:172.21.15.193:3300/0 conn(0x7f5384157b90 0x7f53840701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:46:35.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f53837fe640 1 --2- 172.21.15.7:0/944384979 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5384151cd0 0x7f538415bd00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f53740097c0 tx=0x7f537401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f36762a2437b285 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:35.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f53827fc640 1 -- 172.21.15.7:0/944384979 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53740ab600 con 0x7f5384151cd0 2023-11-28T18:46:35.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f53827fc640 1 -- 172.21.15.7:0/944384979 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53740a7050 con 0x7f5384151cd0 2023-11-28T18:46:35.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f53827fc640 1 -- 172.21.15.7:0/944384979 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53740ab980 con 0x7f5384151cd0 2023-11-28T18:46:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f5389d80640 1 -- 172.21.15.7:0/944384979 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5384151cd0 msgr2=0x7f538415bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f5389d80640 1 --2- 172.21.15.7:0/944384979 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5384151cd0 0x7f538415bd00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f53740097c0 tx=0x7f537401c3f0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f5389d80640 1 -- 172.21.15.7:0/944384979 shutdown_connections 2023-11-28T18:46:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f5389d80640 1 --2- 172.21.15.7:0/944384979 >> v2:172.21.15.193:3300/0 conn(0x7f5384157b90 0x7f53840701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f5389d80640 1 --2- 172.21.15.7:0/944384979 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5384151cd0 0x7f538415bd00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f5389d80640 1 -- 172.21.15.7:0/944384979 >> 172.21.15.7:0/944384979 conn(0x7f53840662e0 msgr2=0x7f53840666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f5389d80640 1 -- 172.21.15.7:0/944384979 shutdown_connections 2023-11-28T18:46:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.377+0000 7f5389d80640 1 -- 172.21.15.7:0/944384979 wait complete. 2023-11-28T18:46:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 Processor -- start 2023-11-28T18:46:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 -- start start 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f538406e560 0x7f5384150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5384151cd0 0x7f538415a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5384157b90 0x7f538415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53840c7150 con 0x7f5384157b90 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53840c3e70 con 0x7f5384151cd0 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53840c3010 con 0x7f538406e560 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5384157b90 0x7f538415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5384157b90 0x7f538415aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35820/0 (socket says 172.21.15.7:35820) 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 -- 172.21.15.7:0/3292085549 learned_addr learned my addr 172.21.15.7:0/3292085549 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5382ffd640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5384151cd0 0x7f538415a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f53837fe640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f538406e560 0x7f5384150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 -- 172.21.15.7:0/3292085549 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f538406e560 msgr2=0x7f5384150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f538406e560 0x7f5384150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 -- 172.21.15.7:0/3292085549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5384151cd0 msgr2=0x7f538415a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5384151cd0 0x7f538415a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 -- 172.21.15.7:0/3292085549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c009a70 con 0x7f5384157b90 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5383fff640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5384157b90 0x7f538415aba0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f537c00efc0 tx=0x7f537c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f536bfff640 1 -- 172.21.15.7:0/3292085549 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c009d70 con 0x7f5384157b90 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f536bfff640 1 -- 172.21.15.7:0/3292085549 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537c00f040 con 0x7f5384157b90 2023-11-28T18:46:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f536bfff640 1 -- 172.21.15.7:0/3292085549 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c0039a0 con 0x7f5384157b90 2023-11-28T18:46:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5374009420 con 0x7f5384157b90 2023-11-28T18:46:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.381+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53840c35d0 con 0x7f5384157b90 2023-11-28T18:46:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.385+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538404e890 con 0x7f5384157b90 2023-11-28T18:46:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.385+0000 7f536bfff640 1 -- 172.21.15.7:0/3292085549 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f537c0144a0 con 0x7f5384157b90 2023-11-28T18:46:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.385+0000 7f536bfff640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5370066900 0x7f5370068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.385+0000 7f536bfff640 1 -- 172.21.15.7:0/3292085549 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f537c0854c0 con 0x7f5384157b90 2023-11-28T18:46:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.385+0000 7f53837fe640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5370066900 0x7f5370068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.385+0000 7f53837fe640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5370066900 0x7f5370068dc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f53740ad000 tx=0x7f5374008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.389+0000 7f536bfff640 1 -- 172.21.15.7:0/3292085549 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f537c052290 con 0x7f5384157b90 2023-11-28T18:46:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53841555b0 con 0x7f5384157b90 2023-11-28T18:46:35.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f536bfff640 1 -- 172.21.15.7:0/3292085549 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f537c0558b0 con 0x7f5384157b90 2023-11-28T18:46:35.582 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:35.582 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5370066900 msgr2=0x7f5370068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5370066900 0x7f5370068dc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f53740ad000 tx=0x7f5374008040 comp rx=0 tx=0).stop 2023-11-28T18:46:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5384157b90 msgr2=0x7f538415aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5384157b90 0x7f538415aba0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f537c00efc0 tx=0x7f537c009f90 comp rx=0 tx=0).stop 2023-11-28T18:46:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 shutdown_connections 2023-11-28T18:46:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5370066900 0x7f5370068dc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5384157b90 0x7f538415aba0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5384151cd0 0x7f538415a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 --2- 172.21.15.7:0/3292085549 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f538406e560 0x7f5384150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 >> 172.21.15.7:0/3292085549 conn(0x7f53840662e0 msgr2=0x7f53840685e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 shutdown_connections 2023-11-28T18:46:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:35.581+0000 7f5389d80640 1 -- 172.21.15.7:0/3292085549 wait complete. 2023-11-28T18:46:35.604 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:45:57.508800+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:46:37.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f1476c9a640 1 Processor -- start 2023-11-28T18:46:37.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f1476c9a640 1 -- start start 2023-11-28T18:46:37.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f1476c9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 0x7f14700c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:37.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f1476c9a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f14700c3bc0 0x7f14700c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:37.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f1476c9a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700c2ce0 con 0x7f147006d9c0 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f1476c9a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1470157240 con 0x7f14700c3bc0 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f1476c9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14700c2f80 con 0x7f14701503c0 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f146ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f14700c3bc0 0x7f14700c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f146ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f14700c3bc0 0x7f14700c3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35826/0 (socket says 172.21.15.7:35826) 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f146ffff640 1 -- 172.21.15.7:0/296424566 learned_addr learned my addr 172.21.15.7:0/296424566 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.761+0000 7f1474a36640 1 --2- 172.21.15.7:0/296424566 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 0x7f14700c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f146ffff640 1 -- 172.21.15.7:0/296424566 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 msgr2=0x7f14700c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f146ffff640 1 --2- 172.21.15.7:0/296424566 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 0x7f14700c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f146ffff640 1 -- 172.21.15.7:0/296424566 >> v1:172.21.15.193:6789/0 conn(0x7f147006d9c0 legacy=0x7f14700c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f146ffff640 1 -- 172.21.15.7:0/296424566 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470156f60 con 0x7f14700c3bc0 2023-11-28T18:46:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f146ffff640 1 --2- 172.21.15.7:0/296424566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14700c3490 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f145c00aa10 tx=0x7f145c020100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a671215600c55cc7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f146f7fe640 1 -- 172.21.15.7:0/296424566 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c0b0600 con 0x7f14700c3bc0 2023-11-28T18:46:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f146f7fe640 1 -- 172.21.15.7:0/296424566 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c0ac050 con 0x7f14700c3bc0 2023-11-28T18:46:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f146f7fe640 1 -- 172.21.15.7:0/296424566 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c0b0980 con 0x7f14700c3bc0 2023-11-28T18:46:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f1476c9a640 1 -- 172.21.15.7:0/296424566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 msgr2=0x7f14700c3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f1476c9a640 1 --2- 172.21.15.7:0/296424566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14700c3490 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f145c00aa10 tx=0x7f145c020100 comp rx=0 tx=0).stop 2023-11-28T18:46:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f1476c9a640 1 -- 172.21.15.7:0/296424566 shutdown_connections 2023-11-28T18:46:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f1476c9a640 1 --2- 172.21.15.7:0/296424566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14700c3490 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f1476c9a640 1 --2- 172.21.15.7:0/296424566 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 0x7f14700c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f1476c9a640 1 -- 172.21.15.7:0/296424566 >> 172.21.15.7:0/296424566 conn(0x7f14700662e0 msgr2=0x7f14700666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.765+0000 7f1476c9a640 1 -- 172.21.15.7:0/296424566 shutdown_connections 2023-11-28T18:46:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 -- 172.21.15.7:0/296424566 wait complete. 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 Processor -- start 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 -- start start 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147006d9c0 0x7f14700ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14701f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 0x7f14700718c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701ef6e0 con 0x7f14700c3bc0 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14700cc380 con 0x7f147006d9c0 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1476c9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147015a1b0 con 0x7f14701503c0 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14701f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14701f0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35830/0 (socket says 172.21.15.7:35830) 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 -- 172.21.15.7:0/2025823490 learned_addr learned my addr 172.21.15.7:0/2025823490 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1474a36640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147006d9c0 0x7f14700ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f1475237640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 0x7f14700718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 -- 172.21.15.7:0/2025823490 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 msgr2=0x7f14700718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 0x7f14700718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 -- 172.21.15.7:0/2025823490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147006d9c0 msgr2=0x7f14700ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147006d9c0 0x7f14700ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 -- 172.21.15.7:0/2025823490 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c00a670 con 0x7f14700c3bc0 2023-11-28T18:46:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146ffff640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14701f0b50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f145c00a1c0 tx=0x7f145c01e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:37.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.769+0000 7f146d7fa640 1 -- 172.21.15.7:0/2025823490 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c0bf070 con 0x7f14700c3bc0 2023-11-28T18:46:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f146d7fa640 1 -- 172.21.15.7:0/2025823490 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c0b9630 con 0x7f14700c3bc0 2023-11-28T18:46:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f146d7fa640 1 -- 172.21.15.7:0/2025823490 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c0b99c0 con 0x7f14700c3bc0 2023-11-28T18:46:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14701ef8f0 con 0x7f14700c3bc0 2023-11-28T18:46:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701efeb0 con 0x7f14700c3bc0 2023-11-28T18:46:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14701f00a0 con 0x7f14700c3bc0 2023-11-28T18:46:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f146d7fa640 1 -- 172.21.15.7:0/2025823490 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f145c0b7070 con 0x7f14700c3bc0 2023-11-28T18:46:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f146d7fa640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14540668d0 0x7f1454068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f146d7fa640 1 -- 172.21.15.7:0/2025823490 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f145c1227b0 con 0x7f14700c3bc0 2023-11-28T18:46:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f1474a36640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14540668d0 0x7f1454068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.773+0000 7f1474a36640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14540668d0 0x7f1454068d90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f146400b930 tx=0x7f1464008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:37.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.777+0000 7f146d7fa640 1 -- 172.21.15.7:0/2025823490 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f145c0ef560 con 0x7f14700c3bc0 2023-11-28T18:46:37.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f147004e890 con 0x7f14700c3bc0 2023-11-28T18:46:37.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f146d7fa640 1 -- 172.21.15.7:0/2025823490 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5823 (secure 0 0 0) 0x7f145c0f2b80 con 0x7f14700c3bc0 2023-11-28T18:46:37.975 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:37.975 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:45:57.508800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 107 2023-11-28T18:46:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14540668d0 msgr2=0x7f1454068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14540668d0 0x7f1454068d90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f146400b930 tx=0x7f1464008040 comp rx=0 tx=0).stop 2023-11-28T18:46:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 msgr2=0x7f14701f0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14701f0b50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f145c00a1c0 tx=0x7f145c01e940 comp rx=0 tx=0).stop 2023-11-28T18:46:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 shutdown_connections 2023-11-28T18:46:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14540668d0 0x7f1454068d90 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14701503c0 0x7f14700718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14700c3bc0 0x7f14701f0b50 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 --2- 172.21.15.7:0/2025823490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147006d9c0 0x7f14700ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 >> 172.21.15.7:0/2025823490 conn(0x7f14700662e0 msgr2=0x7f147014f150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 shutdown_connections 2023-11-28T18:46:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:37.973+0000 7f1476c9a640 1 -- 172.21.15.7:0/2025823490 wait complete. 2023-11-28T18:46:37.995 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:39.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f195b29a640 1 Processor -- start 2023-11-28T18:46:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f195b29a640 1 -- start start 2023-11-28T18:46:40.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f195b29a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1954150840 0x7f19540c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:40.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f195b29a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f195415a3a0 0x7f19540c1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:40.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f195b29a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195406ccc0 con 0x7f19540c7d60 2023-11-28T18:46:40.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f195b29a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1954074400 con 0x7f1954150840 2023-11-28T18:46:40.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f195b29a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195415a7c0 con 0x7f195415a3a0 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f1959837640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f19540c7d60 0x7f1954073cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54234/0 (socket says 172.21.15.7:54234) 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f1959036640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1954150840 0x7f19540c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f1959837640 1 -- 172.21.15.7:0/840604867 learned_addr learned my addr 172.21.15.7:0/840604867 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.149+0000 7f1958835640 1 --2- 172.21.15.7:0/840604867 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f195415a3a0 0x7f19540c1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f194bfff640 1 -- 172.21.15.7:0/840604867 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426635728 0 0) 0x7f195406ccc0 con 0x7f19540c7d60 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f194bfff640 1 -- 172.21.15.7:0/840604867 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1940003530 con 0x7f19540c7d60 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f1959036640 1 -- 172.21.15.7:0/840604867 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f195415a3a0 msgr2=0x7f19540c1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f1959036640 1 --2- 172.21.15.7:0/840604867 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f195415a3a0 0x7f19540c1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f1959036640 1 -- 172.21.15.7:0/840604867 >> v1:172.21.15.7:6789/0 conn(0x7f19540c7d60 legacy=0x7f1954073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f1959036640 1 -- 172.21.15.7:0/840604867 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195406cf90 con 0x7f1954150840 2023-11-28T18:46:40.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f1959036640 1 --2- 172.21.15.7:0/840604867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1954150840 0x7f19540c7600 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f19500097c0 tx=0x7f195001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedb79dbfad0a526 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:40.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f194bfff640 1 -- 172.21.15.7:0/840604867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500a8970 con 0x7f1954150840 2023-11-28T18:46:40.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f194bfff640 1 -- 172.21.15.7:0/840604867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19500a8ad0 con 0x7f1954150840 2023-11-28T18:46:40.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f194bfff640 1 -- 172.21.15.7:0/840604867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500a8e60 con 0x7f1954150840 2023-11-28T18:46:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f195b29a640 1 -- 172.21.15.7:0/840604867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1954150840 msgr2=0x7f19540c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.153+0000 7f195b29a640 1 --2- 172.21.15.7:0/840604867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1954150840 0x7f19540c7600 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f19500097c0 tx=0x7f195001ec70 comp rx=0 tx=0).stop 2023-11-28T18:46:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 -- 172.21.15.7:0/840604867 shutdown_connections 2023-11-28T18:46:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 --2- 172.21.15.7:0/840604867 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f195415a3a0 0x7f19540c1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 --2- 172.21.15.7:0/840604867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1954150840 0x7f19540c7600 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 -- 172.21.15.7:0/840604867 >> 172.21.15.7:0/840604867 conn(0x7f19540cbf80 msgr2=0x7f19540cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 -- 172.21.15.7:0/840604867 shutdown_connections 2023-11-28T18:46:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 -- 172.21.15.7:0/840604867 wait complete. 2023-11-28T18:46:40.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 Processor -- start 2023-11-28T18:46:40.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 -- start start 2023-11-28T18:46:40.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19540c7d60 0x7f19541f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:40.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1954150840 0x7f1954071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:40.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f195415a3a0 0x7f195415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1954156540 con 0x7f19540c7d60 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19541f3020 con 0x7f195415a3a0 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f195b29a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19541f39f0 con 0x7f1954150840 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19540c7d60 0x7f19541f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19540c7d60 0x7f19541f34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35850/0 (socket says 172.21.15.7:35850) 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 -- 172.21.15.7:0/3987957485 learned_addr learned my addr 172.21.15.7:0/3987957485 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 -- 172.21.15.7:0/3987957485 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1954150840 msgr2=0x7f1954071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1954150840 0x7f1954071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 -- 172.21.15.7:0/3987957485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f195415a3a0 msgr2=0x7f195415ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f195415a3a0 0x7f195415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 -- 172.21.15.7:0/3987957485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1950009420 con 0x7f19540c7d60 2023-11-28T18:46:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.157+0000 7f1959036640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19540c7d60 0x7f19541f34b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f195001c8a0 tx=0x7f195001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f1949ffb640 1 -- 172.21.15.7:0/3987957485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500b0d60 con 0x7f19540c7d60 2023-11-28T18:46:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f1949ffb640 1 -- 172.21.15.7:0/3987957485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19500b9410 con 0x7f19540c7d60 2023-11-28T18:46:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19540c5320 con 0x7f19540c7d60 2023-11-28T18:46:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f1949ffb640 1 -- 172.21.15.7:0/3987957485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500b97c0 con 0x7f19540c7d60 2023-11-28T18:46:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19540c58e0 con 0x7f19540c7d60 2023-11-28T18:46:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19540c5ad0 con 0x7f19540c7d60 2023-11-28T18:46:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f1949ffb640 1 -- 172.21.15.7:0/3987957485 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f19500b7040 con 0x7f19540c7d60 2023-11-28T18:46:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f1949ffb640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19380668d0 0x7f1938068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f1949ffb640 1 -- 172.21.15.7:0/3987957485 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f19501224d0 con 0x7f19540c7d60 2023-11-28T18:46:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.161+0000 7f1958835640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19380668d0 0x7f1938068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:40.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.165+0000 7f1958835640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19380668d0 0x7f1938068d90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1944006aa0 tx=0x7f1944006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.165+0000 7f1949ffb640 1 -- 172.21.15.7:0/3987957485 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f19500ef310 con 0x7f19540c7d60 2023-11-28T18:46:40.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.353+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f195404e890 con 0x7f19540c7d60 2023-11-28T18:46:40.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.353+0000 7f1949ffb640 1 -- 172.21.15.7:0/3987957485 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f19500f7e60 con 0x7f19540c7d60 2023-11-28T18:46:40.356 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:40.356 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:40.357 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:40.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19380668d0 msgr2=0x7f1938068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19380668d0 0x7f1938068d90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1944006aa0 tx=0x7f1944006a30 comp rx=0 tx=0).stop 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19540c7d60 msgr2=0x7f19541f34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19540c7d60 0x7f19541f34b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f195001c8a0 tx=0x7f195001ba10 comp rx=0 tx=0).stop 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 shutdown_connections 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f19380668d0 0x7f1938068d90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f195415a3a0 0x7f195415ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1954150840 0x7f1954071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 --2- 172.21.15.7:0/3987957485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19540c7d60 0x7f19541f34b0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 >> 172.21.15.7:0/3987957485 conn(0x7f19540cbf80 msgr2=0x7f19540713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 shutdown_connections 2023-11-28T18:46:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:40.357+0000 7f195b29a640 1 -- 172.21.15.7:0/3987957485 wait complete. 2023-11-28T18:46:40.378 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:40.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~81s 2023-11-28T18:46:41.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:46:41.529 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:46:42.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:42.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.533+0000 7ff41a0c9640 1 Processor -- start 2023-11-28T18:46:42.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.533+0000 7ff41a0c9640 1 -- start start 2023-11-28T18:46:42.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff41a0c9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff414151cd0 0x7ff41415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:42.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff41a0c9640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff41406e560 0x7ff414157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:42.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff41a0c9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff4140701f0 0x7ff4140c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff41a0c9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff414071e60 con 0x7ff414151cd0 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff41a0c9640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff414070930 con 0x7ff41406e560 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff41a0c9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff414070bf0 con 0x7ff4140701f0 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff414151cd0 0x7ff41415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff414151cd0 0x7ff41415bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35856/0 (socket says 172.21.15.7:35856) 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 -- 172.21.15.7:0/582847239 learned_addr learned my addr 172.21.15.7:0/582847239 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 -- 172.21.15.7:0/582847239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff4140701f0 msgr2=0x7ff4140c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 --2- 172.21.15.7:0/582847239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff4140701f0 0x7ff4140c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 -- 172.21.15.7:0/582847239 >> v2:172.21.15.193:3300/0 conn(0x7ff41406e560 msgr2=0x7ff414157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 --2- 172.21.15.7:0/582847239 >> v2:172.21.15.193:3300/0 conn(0x7ff41406e560 0x7ff414157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 -- 172.21.15.7:0/582847239 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff414070ed0 con 0x7ff414151cd0 2023-11-28T18:46:42.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4137fe640 1 --2- 172.21.15.7:0/582847239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff41415bd00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff4140671b0 tx=0x7ff40401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78434783310cba1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:42.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4127fc640 1 -- 172.21.15.7:0/582847239 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040ab600 con 0x7ff414151cd0 2023-11-28T18:46:42.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4127fc640 1 -- 172.21.15.7:0/582847239 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4040a7050 con 0x7ff414151cd0 2023-11-28T18:46:42.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.537+0000 7ff4127fc640 1 -- 172.21.15.7:0/582847239 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040ab980 con 0x7ff414151cd0 2023-11-28T18:46:42.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- 172.21.15.7:0/582847239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 msgr2=0x7ff41415bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/582847239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff41415bd00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff4140671b0 tx=0x7ff40401c470 comp rx=0 tx=0).stop 2023-11-28T18:46:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- 172.21.15.7:0/582847239 shutdown_connections 2023-11-28T18:46:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/582847239 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff4140701f0 0x7ff4140c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/582847239 >> v2:172.21.15.193:3300/0 conn(0x7ff41406e560 0x7ff414157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/582847239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff41415bd00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- 172.21.15.7:0/582847239 >> 172.21.15.7:0/582847239 conn(0x7ff4140662e0 msgr2=0x7ff4140666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- 172.21.15.7:0/582847239 shutdown_connections 2023-11-28T18:46:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- 172.21.15.7:0/582847239 wait complete. 2023-11-28T18:46:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 Processor -- start 2023-11-28T18:46:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- start start 2023-11-28T18:46:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff41406e560 0x7ff4141f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4140701f0 0x7ff4140c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff4140ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4141f38c0 con 0x7ff414151cd0 2023-11-28T18:46:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff414145ed0 con 0x7ff4140701f0 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff41a0c9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4141f32d0 con 0x7ff41406e560 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff413fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff4140ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff413fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff4140ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35864/0 (socket says 172.21.15.7:35864) 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff413fff640 1 -- 172.21.15.7:0/127722058 learned_addr learned my addr 172.21.15.7:0/127722058 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff4137fe640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff41406e560 0x7ff4141f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.541+0000 7ff412ffd640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4140701f0 0x7ff4140c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff413fff640 1 -- 172.21.15.7:0/127722058 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff41406e560 msgr2=0x7ff4141f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff413fff640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff41406e560 0x7ff4141f3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff413fff640 1 -- 172.21.15.7:0/127722058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4140701f0 msgr2=0x7ff4140c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff413fff640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4140701f0 0x7ff4140c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff413fff640 1 -- 172.21.15.7:0/127722058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040094d0 con 0x7ff414151cd0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff413fff640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff4140ca270 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00b970 tx=0x7ff40c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff3fbfff640 1 -- 172.21.15.7:0/127722058 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c00cb40 con 0x7ff414151cd0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff3fbfff640 1 -- 172.21.15.7:0/127722058 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c00ccc0 con 0x7ff414151cd0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff3fbfff640 1 -- 172.21.15.7:0/127722058 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c014680 con 0x7ff414151cd0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41413e070 con 0x7ff414151cd0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4141f7720 con 0x7ff414151cd0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4141f3a00 con 0x7ff414151cd0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff3fbfff640 1 -- 172.21.15.7:0/127722058 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff40c01d3b0 con 0x7ff414151cd0 2023-11-28T18:46:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff3fbfff640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff400066900 0x7ff400068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:42.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff3fbfff640 1 -- 172.21.15.7:0/127722058 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff40c0846a0 con 0x7ff414151cd0 2023-11-28T18:46:42.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.545+0000 7ff4137fe640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff400066900 0x7ff400068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:42.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.549+0000 7ff4137fe640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff400066900 0x7ff400068dc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff4040ad000 tx=0x7ff4040a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:42.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.549+0000 7ff3fbfff640 1 -- 172.21.15.7:0/127722058 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff4141f3a00 con 0x7ff414151cd0 2023-11-28T18:46:42.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.741+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff41413e070 con 0x7ff414151cd0 2023-11-28T18:46:42.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.741+0000 7ff3fbfff640 1 -- 172.21.15.7:0/127722058 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7ff41413e070 con 0x7ff414151cd0 2023-11-28T18:46:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:42.744 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff400066900 msgr2=0x7ff400068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff400066900 0x7ff400068dc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff4040ad000 tx=0x7ff4040a6000 comp rx=0 tx=0).stop 2023-11-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 msgr2=0x7ff4140ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff4140ca270 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00b970 tx=0x7ff40c00be30 comp rx=0 tx=0).stop 2023-11-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 shutdown_connections 2023-11-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff400066900 0x7ff400068dc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff414151cd0 0x7ff4140ca270 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4140701f0 0x7ff4140c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 --2- 172.21.15.7:0/127722058 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff41406e560 0x7ff4141f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:42.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 >> 172.21.15.7:0/127722058 conn(0x7ff4140662e0 msgr2=0x7ff414072580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:42.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 shutdown_connections 2023-11-28T18:46:42.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:42.745+0000 7ff41a0c9640 1 -- 172.21.15.7:0/127722058 wait complete. 2023-11-28T18:46:42.765 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:44.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:44.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44addf640 1 Processor -- start 2023-11-28T18:46:44.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44addf640 1 -- start start 2023-11-28T18:46:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44addf640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe444072950 0x7fe444074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44addf640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444073070 0x7fe4440c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44addf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440c8490 con 0x7fe4440c1ef0 2023-11-28T18:46:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44addf640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4440c81f0 con 0x7fe444072950 2023-11-28T18:46:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44addf640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44415ab00 con 0x7fe444073070 2023-11-28T18:46:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe448b7b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe444072950 0x7fe444074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44937c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe4440c1ef0 0x7fe44406c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52052/0 (socket says 172.21.15.7:52052) 2023-11-28T18:46:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.921+0000 7fe44937c640 1 -- 172.21.15.7:0/1399802831 learned_addr learned my addr 172.21.15.7:0/1399802831 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe443fff640 1 --2- 172.21.15.7:0/1399802831 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444073070 0x7fe4440c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe448b7b640 1 -- 172.21.15.7:0/1399802831 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444073070 msgr2=0x7fe4440c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe448b7b640 1 --2- 172.21.15.7:0/1399802831 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444073070 0x7fe4440c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe4437fe640 1 -- 172.21.15.7:0/1399802831 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164827445 0 0) 0x7fe4440c8490 con 0x7fe4440c1ef0 2023-11-28T18:46:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe448b7b640 1 -- 172.21.15.7:0/1399802831 >> v1:172.21.15.7:6789/0 conn(0x7fe4440c1ef0 legacy=0x7fe44406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe448b7b640 1 -- 172.21.15.7:0/1399802831 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44415ade0 con 0x7fe444072950 2023-11-28T18:46:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe448b7b640 1 --2- 172.21.15.7:0/1399802831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe444072950 0x7fe444074030 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe4380097c0 tx=0x7fe43801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17703847333d34dd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe4437fe640 1 -- 172.21.15.7:0/1399802831 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380ab600 con 0x7fe444072950 2023-11-28T18:46:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe4437fe640 1 -- 172.21.15.7:0/1399802831 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4380a7050 con 0x7fe444072950 2023-11-28T18:46:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe4437fe640 1 -- 172.21.15.7:0/1399802831 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380ab980 con 0x7fe444072950 2023-11-28T18:46:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe44addf640 1 -- 172.21.15.7:0/1399802831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe444072950 msgr2=0x7fe444074030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.925+0000 7fe44addf640 1 --2- 172.21.15.7:0/1399802831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe444072950 0x7fe444074030 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe4380097c0 tx=0x7fe43801c360 comp rx=0 tx=0).stop 2023-11-28T18:46:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 -- 172.21.15.7:0/1399802831 shutdown_connections 2023-11-28T18:46:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 --2- 172.21.15.7:0/1399802831 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444073070 0x7fe4440c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 --2- 172.21.15.7:0/1399802831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe444072950 0x7fe444074030 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 -- 172.21.15.7:0/1399802831 >> 172.21.15.7:0/1399802831 conn(0x7fe4440cbf80 msgr2=0x7fe4440cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 -- 172.21.15.7:0/1399802831 shutdown_connections 2023-11-28T18:46:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 -- 172.21.15.7:0/1399802831 wait complete. 2023-11-28T18:46:44.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 Processor -- start 2023-11-28T18:46:44.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 -- start start 2023-11-28T18:46:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444072950 0x7fe444070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe444073070 0x7fe4440714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4440c1ef0 0x7fe44415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44415bc20 con 0x7fe4440c1ef0 2023-11-28T18:46:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4441466d0 con 0x7fe444073070 2023-11-28T18:46:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44addf640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440c7ab0 con 0x7fe444072950 2023-11-28T18:46:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4440c1ef0 0x7fe44415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4440c1ef0 0x7fe44415c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53228/0 (socket says 172.21.15.7:53228) 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 -- 172.21.15.7:0/1799671601 learned_addr learned my addr 172.21.15.7:0/1799671601 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe443fff640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe444073070 0x7fe4440714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe448b7b640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444072950 0x7fe444070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 -- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444072950 msgr2=0x7fe444070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444072950 0x7fe444070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 -- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe444073070 msgr2=0x7fe4440714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe444073070 0x7fe4440714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 -- 172.21.15.7:0/1799671601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe438009420 con 0x7fe4440c1ef0 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe443fff640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe444073070 0x7fe4440714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe448b7b640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444072950 0x7fe444070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.929+0000 7fe44937c640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4440c1ef0 0x7fe44415c7e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00cac0 tx=0x7fe43c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:44.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe4417fa640 1 -- 172.21.15.7:0/1799671601 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00abd0 con 0x7fe4440c1ef0 2023-11-28T18:46:44.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe4417fa640 1 -- 172.21.15.7:0/1799671601 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43c00ad30 con 0x7fe4440c1ef0 2023-11-28T18:46:44.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe4417fa640 1 -- 172.21.15.7:0/1799671601 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00aea0 con 0x7fe4440c1ef0 2023-11-28T18:46:44.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe444150240 con 0x7fe4440c1ef0 2023-11-28T18:46:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44414fc80 con 0x7fe4440c1ef0 2023-11-28T18:46:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44404e890 con 0x7fe4440c1ef0 2023-11-28T18:46:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe4417fa640 1 -- 172.21.15.7:0/1799671601 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe43c007070 con 0x7fe4440c1ef0 2023-11-28T18:46:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe4417fa640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4280668d0 0x7fe428068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe4417fa640 1 -- 172.21.15.7:0/1799671601 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fe43c0848f0 con 0x7fe4440c1ef0 2023-11-28T18:46:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.933+0000 7fe448b7b640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4280668d0 0x7fe428068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:44.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.937+0000 7fe448b7b640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4280668d0 0x7fe428068d90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe4380ad000 tx=0x7fe438008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:44.937+0000 7fe4417fa640 1 -- 172.21.15.7:0/1799671601 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe43c0515a0 con 0x7fe4440c1ef0 2023-11-28T18:46:45.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.129+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe444150020 con 0x7fe4440c1ef0 2023-11-28T18:46:45.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.129+0000 7fe4417fa640 1 -- 172.21.15.7:0/1799671601 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fe43c054bc0 con 0x7fe4440c1ef0 2023-11-28T18:46:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:45.133 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:45.134 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:45.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4280668d0 msgr2=0x7fe428068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:45.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4280668d0 0x7fe428068d90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe4380ad000 tx=0x7fe438008040 comp rx=0 tx=0).stop 2023-11-28T18:46:45.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4440c1ef0 msgr2=0x7fe44415c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:45.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4440c1ef0 0x7fe44415c7e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00cac0 tx=0x7fe43c00cf80 comp rx=0 tx=0).stop 2023-11-28T18:46:45.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 shutdown_connections 2023-11-28T18:46:45.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4280668d0 0x7fe428068d90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:45.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4440c1ef0 0x7fe44415c7e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:45.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe444073070 0x7fe4440714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:45.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 --2- 172.21.15.7:0/1799671601 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe444072950 0x7fe444070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:45.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 >> 172.21.15.7:0/1799671601 conn(0x7fe4440cbf80 msgr2=0x7fe4440cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:45.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 shutdown_connections 2023-11-28T18:46:45.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:45.133+0000 7fe44addf640 1 -- 172.21.15.7:0/1799671601 wait complete. 2023-11-28T18:46:45.153 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:47.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:47.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~88s 2023-11-28T18:46:47.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb94769a640 1 Processor -- start 2023-11-28T18:46:47.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb94769a640 1 -- start start 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb94769a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb9401542e0 0x7fb94015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb94769a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb940072450 0x7fb9400728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb94769a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9400ca920 con 0x7fb9400cbb70 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb94769a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb940071d50 con 0x7fb9401542e0 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb94769a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb940072030 con 0x7fb940072450 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945436640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb9401542e0 0x7fb94015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945436640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb9401542e0 0x7fb94015e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53236/0 (socket says 172.21.15.7:53236) 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945436640 1 -- 172.21.15.7:0/2023705750 learned_addr learned my addr 172.21.15.7:0/2023705750 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945c37640 1 --2- 172.21.15.7:0/2023705750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb940072450 0x7fb9400728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945436640 1 -- 172.21.15.7:0/2023705750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb940072450 msgr2=0x7fb9400728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945436640 1 --2- 172.21.15.7:0/2023705750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb940072450 0x7fb9400728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945436640 1 -- 172.21.15.7:0/2023705750 >> v1:172.21.15.193:6789/0 conn(0x7fb9400cbb70 legacy=0x7fb9400cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:47.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945436640 1 -- 172.21.15.7:0/2023705750 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94015e550 con 0x7fb9401542e0 2023-11-28T18:46:47.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.309+0000 7fb945c37640 1 --2- 172.21.15.7:0/2023705750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb940072450 0x7fb9400728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:47.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb945436640 1 --2- 172.21.15.7:0/2023705750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9401542e0 0x7fb94015e010 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0098a0 tx=0x7fb93c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d86b683b8951f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb937fff640 1 -- 172.21.15.7:0/2023705750 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0a7070 con 0x7fb9401542e0 2023-11-28T18:46:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb937fff640 1 -- 172.21.15.7:0/2023705750 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c0ab430 con 0x7fb9401542e0 2023-11-28T18:46:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb937fff640 1 -- 172.21.15.7:0/2023705750 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0ab7c0 con 0x7fb9401542e0 2023-11-28T18:46:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb94769a640 1 -- 172.21.15.7:0/2023705750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9401542e0 msgr2=0x7fb94015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb94769a640 1 --2- 172.21.15.7:0/2023705750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9401542e0 0x7fb94015e010 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0098a0 tx=0x7fb93c01c520 comp rx=0 tx=0).stop 2023-11-28T18:46:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb94769a640 1 -- 172.21.15.7:0/2023705750 shutdown_connections 2023-11-28T18:46:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb94769a640 1 --2- 172.21.15.7:0/2023705750 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb940072450 0x7fb9400728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb94769a640 1 --2- 172.21.15.7:0/2023705750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9401542e0 0x7fb94015e010 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb94769a640 1 -- 172.21.15.7:0/2023705750 >> 172.21.15.7:0/2023705750 conn(0x7fb9400717e0 msgr2=0x7fb9400662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb94769a640 1 -- 172.21.15.7:0/2023705750 shutdown_connections 2023-11-28T18:46:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.313+0000 7fb94769a640 1 -- 172.21.15.7:0/2023705750 wait complete. 2023-11-28T18:46:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 Processor -- start 2023-11-28T18:46:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 -- start start 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb940072450 0x7fb940159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9400cbb70 0x7fb9400c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9401542e0 0x7fb9400ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94015a270 con 0x7fb940072450 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400c79f0 con 0x7fb9400cbb70 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400c9160 con 0x7fb9401542e0 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb940072450 0x7fb940159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb940072450 0x7fb940159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53242/0 (socket says 172.21.15.7:53242) 2023-11-28T18:46:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 -- 172.21.15.7:0/1504608962 learned_addr learned my addr 172.21.15.7:0/1504608962 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb944c35640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9400cbb70 0x7fb9400c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945c37640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9401542e0 0x7fb9400ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 -- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9401542e0 msgr2=0x7fb9400ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9401542e0 0x7fb9400ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 -- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9400cbb70 msgr2=0x7fb9400c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9400cbb70 0x7fb9400c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 -- 172.21.15.7:0/1504608962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c0094d0 con 0x7fb940072450 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945c37640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9401542e0 0x7fb9400ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb944c35640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9400cbb70 0x7fb9400c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb945436640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb940072450 0x7fb940159d30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0ad040 tx=0x7fb93c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb935ffb640 1 -- 172.21.15.7:0/1504608962 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0bc070 con 0x7fb940072450 2023-11-28T18:46:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb935ffb640 1 -- 172.21.15.7:0/1504608962 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c0abe50 con 0x7fb940072450 2023-11-28T18:46:47.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb935ffb640 1 -- 172.21.15.7:0/1504608962 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0b4870 con 0x7fb940072450 2023-11-28T18:46:47.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9400c73a0 con 0x7fb940072450 2023-11-28T18:46:47.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.317+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94015bf40 con 0x7fb940072450 2023-11-28T18:46:47.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.321+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94006c670 con 0x7fb940072450 2023-11-28T18:46:47.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.321+0000 7fb935ffb640 1 -- 172.21.15.7:0/1504608962 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb93c0b49d0 con 0x7fb940072450 2023-11-28T18:46:47.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.321+0000 7fb935ffb640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb924066900 0x7fb924068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:47.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.321+0000 7fb935ffb640 1 -- 172.21.15.7:0/1504608962 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fb93c11e0b0 con 0x7fb940072450 2023-11-28T18:46:47.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.321+0000 7fb944c35640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb924066900 0x7fb924068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:47.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.321+0000 7fb944c35640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb924066900 0x7fb924068dc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb940157e10 tx=0x7fb930009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:47.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.325+0000 7fb935ffb640 1 -- 172.21.15.7:0/1504608962 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb93c0eaf80 con 0x7fb940072450 2023-11-28T18:46:47.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb94004e890 con 0x7fb940072450 2023-11-28T18:46:47.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb935ffb640 1 -- 172.21.15.7:0/1504608962 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fb93c0ee5a0 con 0x7fb940072450 2023-11-28T18:46:47.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:47.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:47.519 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb924066900 msgr2=0x7fb924068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb924066900 0x7fb924068dc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb940157e10 tx=0x7fb930009040 comp rx=0 tx=0).stop 2023-11-28T18:46:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb940072450 msgr2=0x7fb940159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb940072450 0x7fb940159d30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0ad040 tx=0x7fb93c0192c0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 shutdown_connections 2023-11-28T18:46:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb924066900 0x7fb924068dc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb9401542e0 0x7fb9400ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9400cbb70 0x7fb9400c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 --2- 172.21.15.7:0/1504608962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb940072450 0x7fb940159d30 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 >> 172.21.15.7:0/1504608962 conn(0x7fb9400717e0 msgr2=0x7fb94015dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 shutdown_connections 2023-11-28T18:46:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:47.517+0000 7fb94769a640 1 -- 172.21.15.7:0/1504608962 wait complete. 2023-11-28T18:46:47.542 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:46:49.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:49.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.697+0000 7f8e61901640 1 Processor -- start 2023-11-28T18:46:49.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.697+0000 7f8e61901640 1 -- start start 2023-11-28T18:46:49.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.697+0000 7f8e61901640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8e5c150b30 0x7f8e5c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:49.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.697+0000 7f8e61901640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e5c1575c0 0x7f8e5c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:49.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.697+0000 7f8e61901640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c0c6140 con 0x7f8e5c0c4020 2023-11-28T18:46:49.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.697+0000 7f8e61901640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c0c72b0 con 0x7f8e5c1575c0 2023-11-28T18:46:49.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.697+0000 7f8e61901640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c070c00 con 0x7f8e5c150b30 2023-11-28T18:46:49.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e53fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e5c1575c0 0x7f8e5c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:49.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e5b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8e5c0c4020 0x7f8e5c0c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52068/0 (socket says 172.21.15.7:52068) 2023-11-28T18:46:49.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e5b7fe640 1 -- 172.21.15.7:0/4265879179 learned_addr learned my addr 172.21.15.7:0/4265879179 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:49.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e5affd640 1 --2- 172.21.15.7:0/4265879179 >> v2:172.21.15.193:3300/0 conn(0x7f8e5c150b30 0x7f8e5c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:49.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e53fff640 1 -- 172.21.15.7:0/4265879179 >> v2:172.21.15.193:3300/0 conn(0x7f8e5c150b30 msgr2=0x7f8e5c0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:49.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e53fff640 1 --2- 172.21.15.7:0/4265879179 >> v2:172.21.15.193:3300/0 conn(0x7f8e5c150b30 0x7f8e5c0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e53fff640 1 -- 172.21.15.7:0/4265879179 >> v1:172.21.15.7:6789/0 conn(0x7f8e5c0c4020 legacy=0x7f8e5c0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:49.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e53fff640 1 -- 172.21.15.7:0/4265879179 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c0c6370 con 0x7f8e5c1575c0 2023-11-28T18:46:49.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e5affd640 1 --2- 172.21.15.7:0/4265879179 >> v2:172.21.15.193:3300/0 conn(0x7f8e5c150b30 0x7f8e5c0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:49.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e5a7fc640 1 -- 172.21.15.7:0/4265879179 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624223569 0 0) 0x7f8e5c0c6140 con 0x7f8e5c0c4020 2023-11-28T18:46:49.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e53fff640 1 --2- 172.21.15.7:0/4265879179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c0c3ae0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c06ec80 tx=0x7f8e4401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc69416192e6024 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:49.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e5a7fc640 1 -- 172.21.15.7:0/4265879179 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e440a8950 con 0x7f8e5c1575c0 2023-11-28T18:46:49.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e5a7fc640 1 -- 172.21.15.7:0/4265879179 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e440a8ab0 con 0x7f8e5c1575c0 2023-11-28T18:46:49.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.701+0000 7f8e5a7fc640 1 -- 172.21.15.7:0/4265879179 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e440a8e40 con 0x7f8e5c1575c0 2023-11-28T18:46:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- 172.21.15.7:0/4265879179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 msgr2=0x7f8e5c0c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 --2- 172.21.15.7:0/4265879179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c0c3ae0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c06ec80 tx=0x7f8e4401ec50 comp rx=0 tx=0).stop 2023-11-28T18:46:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- 172.21.15.7:0/4265879179 shutdown_connections 2023-11-28T18:46:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 --2- 172.21.15.7:0/4265879179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c0c3ae0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 --2- 172.21.15.7:0/4265879179 >> v2:172.21.15.193:3300/0 conn(0x7f8e5c150b30 0x7f8e5c0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- 172.21.15.7:0/4265879179 >> 172.21.15.7:0/4265879179 conn(0x7f8e5c0662e0 msgr2=0x7f8e5c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- 172.21.15.7:0/4265879179 shutdown_connections 2023-11-28T18:46:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- 172.21.15.7:0/4265879179 wait complete. 2023-11-28T18:46:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 Processor -- start 2023-11-28T18:46:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- start start 2023-11-28T18:46:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e5c0c4020 0x7f8e5c152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5c150b30 0x7f8e5c152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c1f0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c152590 con 0x7f8e5c1575c0 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c14ae80 con 0x7f8e5c150b30 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e61901640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c14b160 con 0x7f8e5c0c4020 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c1f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c1f0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53256/0 (socket says 172.21.15.7:53256) 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5b7fe640 1 -- 172.21.15.7:0/2634023678 learned_addr learned my addr 172.21.15.7:0/2634023678 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5affd640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e5c0c4020 0x7f8e5c152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e53fff640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5c150b30 0x7f8e5c152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5b7fe640 1 -- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e5c0c4020 msgr2=0x7f8e5c152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5b7fe640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e5c0c4020 0x7f8e5c152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5b7fe640 1 -- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5c150b30 msgr2=0x7f8e5c152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5b7fe640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5c150b30 0x7f8e5c152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5b7fe640 1 -- 172.21.15.7:0/2634023678 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e44009420 con 0x7f8e5c1575c0 2023-11-28T18:46:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.705+0000 7f8e5affd640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e5c0c4020 0x7f8e5c152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e5b7fe640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c1f0770 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8e54001570 tx=0x7f8e5400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e58ff9640 1 -- 172.21.15.7:0/2634023678 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5400a640 con 0x7f8e5c1575c0 2023-11-28T18:46:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e58ff9640 1 -- 172.21.15.7:0/2634023678 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5400a7c0 con 0x7f8e5c1575c0 2023-11-28T18:46:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e58ff9640 1 -- 172.21.15.7:0/2634023678 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5400ab70 con 0x7f8e5c1575c0 2023-11-28T18:46:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c14b440 con 0x7f8e5c1575c0 2023-11-28T18:46:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c1f0200 con 0x7f8e5c1575c0 2023-11-28T18:46:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c152700 con 0x7f8e5c1575c0 2023-11-28T18:46:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e58ff9640 1 -- 172.21.15.7:0/2634023678 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8e5400acd0 con 0x7f8e5c1575c0 2023-11-28T18:46:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e58ff9640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e3c0668d0 0x7f8e3c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:49.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e58ff9640 1 -- 172.21.15.7:0/2634023678 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f8e54083fc0 con 0x7f8e5c1575c0 2023-11-28T18:46:49.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.709+0000 7f8e5affd640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e3c0668d0 0x7f8e3c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:49.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.713+0000 7f8e5affd640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e3c0668d0 0x7f8e3c068d90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c008930 tx=0x7f8e4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.713+0000 7f8e58ff9640 1 -- 172.21.15.7:0/2634023678 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8e54050c50 con 0x7f8e5c1575c0 2023-11-28T18:46:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.905+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e5c04e890 con 0x7f8e5c1575c0 2023-11-28T18:46:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.905+0000 7f8e58ff9640 1 -- 172.21.15.7:0/2634023678 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f8e54018030 con 0x7f8e5c1575c0 2023-11-28T18:46:49.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:49.908 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e3c0668d0 msgr2=0x7f8e3c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e3c0668d0 0x7f8e3c068d90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c008930 tx=0x7f8e4c008040 comp rx=0 tx=0).stop 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 msgr2=0x7f8e5c1f0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c1f0770 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8e54001570 tx=0x7f8e5400cca0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 shutdown_connections 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8e3c0668d0 0x7f8e3c068d90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c1575c0 0x7f8e5c1f0770 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5c150b30 0x7f8e5c152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 --2- 172.21.15.7:0/2634023678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8e5c0c4020 0x7f8e5c152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 >> 172.21.15.7:0/2634023678 conn(0x7f8e5c0662e0 msgr2=0x7f8e5c0666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 shutdown_connections 2023-11-28T18:46:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:49.909+0000 7f8e61901640 1 -- 172.21.15.7:0/2634023678 wait complete. 2023-11-28T18:46:49.929 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:51.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d6501640 1 Processor -- start 2023-11-28T18:46:52.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d6501640 1 -- start start 2023-11-28T18:46:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d6501640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f06d0150b30 0x7f06d00c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d6501640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d00c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d6501640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d00c6140 con 0x7f06d01575c0 2023-11-28T18:46:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d6501640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d00c72b0 con 0x7f06d0150b30 2023-11-28T18:46:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d6501640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0070c00 con 0x7f06d00c7980 2023-11-28T18:46:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06cf7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f06d01575c0 0x7f06d00c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52072/0 (socket says 172.21.15.7:52072) 2023-11-28T18:46:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06cf7fe640 1 -- 172.21.15.7:0/1130653032 learned_addr learned my addr 172.21.15.7:0/1130653032 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d4a9e640 1 --2- 172.21.15.7:0/1130653032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d00c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:52.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06cffff640 1 --2- 172.21.15.7:0/1130653032 >> v2:172.21.15.193:3300/0 conn(0x7f06d0150b30 0x7f06d00c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:52.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06ceffd640 1 -- 172.21.15.7:0/1130653032 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476437167 0 0) 0x7f06d00c6140 con 0x7f06d01575c0 2023-11-28T18:46:52.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06ceffd640 1 -- 172.21.15.7:0/1130653032 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b8005740 con 0x7f06d01575c0 2023-11-28T18:46:52.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d4a9e640 1 -- 172.21.15.7:0/1130653032 >> v2:172.21.15.193:3300/0 conn(0x7f06d0150b30 msgr2=0x7f06d00c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:52.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d4a9e640 1 --2- 172.21.15.7:0/1130653032 >> v2:172.21.15.193:3300/0 conn(0x7f06d0150b30 0x7f06d00c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d4a9e640 1 -- 172.21.15.7:0/1130653032 >> v1:172.21.15.7:6789/0 conn(0x7f06d01575c0 legacy=0x7f06d00c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:52.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.085+0000 7f06d4a9e640 1 -- 172.21.15.7:0/1130653032 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d00c8350 con 0x7f06d00c7980 2023-11-28T18:46:52.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d4a9e640 1 --2- 172.21.15.7:0/1130653032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d00c7e10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009d90 tx=0x7f06c801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78358377a1c02c3d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:52.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06ceffd640 1 -- 172.21.15.7:0/1130653032 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c80af600 con 0x7f06d00c7980 2023-11-28T18:46:52.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06ceffd640 1 -- 172.21.15.7:0/1130653032 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c80ab050 con 0x7f06d00c7980 2023-11-28T18:46:52.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06ceffd640 1 -- 172.21.15.7:0/1130653032 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c80af980 con 0x7f06d00c7980 2023-11-28T18:46:52.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d6501640 1 -- 172.21.15.7:0/1130653032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 msgr2=0x7f06d00c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:52.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d6501640 1 --2- 172.21.15.7:0/1130653032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d00c7e10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009d90 tx=0x7f06c801f130 comp rx=0 tx=0).stop 2023-11-28T18:46:52.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d6501640 1 -- 172.21.15.7:0/1130653032 shutdown_connections 2023-11-28T18:46:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d6501640 1 --2- 172.21.15.7:0/1130653032 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d00c7e10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d6501640 1 --2- 172.21.15.7:0/1130653032 >> v2:172.21.15.193:3300/0 conn(0x7f06d0150b30 0x7f06d00c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d6501640 1 -- 172.21.15.7:0/1130653032 >> 172.21.15.7:0/1130653032 conn(0x7f06d00662e0 msgr2=0x7f06d00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d6501640 1 -- 172.21.15.7:0/1130653032 shutdown_connections 2023-11-28T18:46:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.089+0000 7f06d6501640 1 -- 172.21.15.7:0/1130653032 wait complete. 2023-11-28T18:46:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 Processor -- start 2023-11-28T18:46:52.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 -- start start 2023-11-28T18:46:52.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d0152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:52.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06d0150b30 0x7f06d0156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d01575c0 0x7f06d00713e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0152bd0 con 0x7f06d01575c0 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0153780 con 0x7f06d0150b30 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0151430 con 0x7f06d00c7980 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d01575c0 0x7f06d00713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d01575c0 0x7f06d00713e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53258/0 (socket says 172.21.15.7:53258) 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 -- 172.21.15.7:0/1577763353 learned_addr learned my addr 172.21.15.7:0/1577763353 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06cf7fe640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06d0150b30 0x7f06d0156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06cffff640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d0152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 -- 172.21.15.7:0/1577763353 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 msgr2=0x7f06d0152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d0152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 -- 172.21.15.7:0/1577763353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06d0150b30 msgr2=0x7f06d0156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06d0150b30 0x7f06d0156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 -- 172.21.15.7:0/1577763353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c8009a70 con 0x7f06d01575c0 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d4a9e640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d01575c0 0x7f06d00713e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f06c80b2040 tx=0x7f06c801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06ccff9640 1 -- 172.21.15.7:0/1577763353 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8009450 con 0x7f06d01575c0 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06ccff9640 1 -- 172.21.15.7:0/1577763353 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c80b9410 con 0x7f06d01575c0 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d0151cd0 con 0x7f06d01575c0 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06ccff9640 1 -- 172.21.15.7:0/1577763353 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c80b97c0 con 0x7f06d01575c0 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.093+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d0156a10 con 0x7f06d01575c0 2023-11-28T18:46:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.097+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d0151e60 con 0x7f06d01575c0 2023-11-28T18:46:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.097+0000 7f06ccff9640 1 -- 172.21.15.7:0/1577763353 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f06c80b6050 con 0x7f06d01575c0 2023-11-28T18:46:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.097+0000 7f06ccff9640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f06bc066900 0x7f06bc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.097+0000 7f06ccff9640 1 -- 172.21.15.7:0/1577763353 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f06c8122b80 con 0x7f06d01575c0 2023-11-28T18:46:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.097+0000 7f06cffff640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f06bc066900 0x7f06bc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.097+0000 7f06cffff640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f06bc066900 0x7f06bc068dc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009b60 tx=0x7f06c0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:52.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.101+0000 7f06ccff9640 1 -- 172.21.15.7:0/1577763353 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f06c80efb00 con 0x7f06d01575c0 2023-11-28T18:46:52.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.293+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06d004e890 con 0x7f06d01575c0 2023-11-28T18:46:52.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06ccff9640 1 -- 172.21.15.7:0/1577763353 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f06c8122e60 con 0x7f06d01575c0 2023-11-28T18:46:52.297 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:52.297 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:52.297 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:52.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f06bc066900 msgr2=0x7f06bc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:52.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f06bc066900 0x7f06bc068dc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009b60 tx=0x7f06c0009190 comp rx=0 tx=0).stop 2023-11-28T18:46:52.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d01575c0 msgr2=0x7f06d00713e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d01575c0 0x7f06d00713e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f06c80b2040 tx=0x7f06c801bed0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 shutdown_connections 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f06bc066900 0x7f06bc068dc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d01575c0 0x7f06d00713e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06d0150b30 0x7f06d0156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 --2- 172.21.15.7:0/1577763353 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f06d00c7980 0x7f06d0152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 >> 172.21.15.7:0/1577763353 conn(0x7f06d00662e0 msgr2=0x7f06d014fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 shutdown_connections 2023-11-28T18:46:52.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:52.297+0000 7f06d6501640 1 -- 172.21.15.7:0/1577763353 wait complete. 2023-11-28T18:46:52.318 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:53.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~94s 2023-11-28T18:46:54.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:54.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fc55b8640 1 Processor -- start 2023-11-28T18:46:54.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fc55b8640 1 -- start start 2023-11-28T18:46:54.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fc55b8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1fc00c3bc0 0x7f1fc00c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fc55b8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc00c38e0 con 0x7f1fc00c3490 2023-11-28T18:46:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fc55b8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc00c2ce0 con 0x7f1fc01503c0 2023-11-28T18:46:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fc55b8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc0157240 con 0x7f1fc00c3bc0 2023-11-28T18:46:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fbf7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1fc00c3490 0x7f1fc00c46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42210/0 (socket says 172.21.15.7:42210) 2023-11-28T18:46:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fbf7fe640 1 -- 172.21.15.7:0/3084374757 learned_addr learned my addr 172.21.15.7:0/3084374757 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:46:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.473+0000 7f1fb7fff640 1 --2- 172.21.15.7:0/3084374757 >> v2:172.21.15.193:3300/0 conn(0x7f1fc00c3bc0 0x7f1fc00c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fbe7fc640 1 -- 172.21.15.7:0/3084374757 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954551051 0 0) 0x7f1fc00c38e0 con 0x7f1fc00c3490 2023-11-28T18:46:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fbe7fc640 1 -- 172.21.15.7:0/3084374757 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fac003450 con 0x7f1fc00c3490 2023-11-28T18:46:54.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fb7fff640 1 -- 172.21.15.7:0/3084374757 >> v1:172.21.15.193:6789/0 conn(0x7f1fc01503c0 legacy=0x7f1fc006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:54.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fbe7fc640 1 -- 172.21.15.7:0/3084374757 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602250772 0 0) 0x7f1fc00c2ce0 con 0x7f1fc01503c0 2023-11-28T18:46:54.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fb7fff640 1 -- 172.21.15.7:0/3084374757 >> v1:172.21.15.7:6789/0 conn(0x7f1fc00c3490 legacy=0x7f1fc00c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:54.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fb7fff640 1 -- 172.21.15.7:0/3084374757 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc00c2f80 con 0x7f1fc00c3bc0 2023-11-28T18:46:54.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fb7fff640 1 --2- 172.21.15.7:0/3084374757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc00c3bc0 0x7f1fc00c7c60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80097c0 tx=0x7f1fa801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69f16f552bc4a43 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:54.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fbe7fc640 1 -- 172.21.15.7:0/3084374757 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80a8970 con 0x7f1fc00c3bc0 2023-11-28T18:46:54.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fbe7fc640 1 -- 172.21.15.7:0/3084374757 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa80a8ad0 con 0x7f1fc00c3bc0 2023-11-28T18:46:54.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.477+0000 7f1fbe7fc640 1 -- 172.21.15.7:0/3084374757 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80a8e60 con 0x7f1fc00c3bc0 2023-11-28T18:46:54.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- 172.21.15.7:0/3084374757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc00c3bc0 msgr2=0x7f1fc00c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:54.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 --2- 172.21.15.7:0/3084374757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc00c3bc0 0x7f1fc00c7c60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80097c0 tx=0x7f1fa801ec70 comp rx=0 tx=0).stop 2023-11-28T18:46:54.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- 172.21.15.7:0/3084374757 shutdown_connections 2023-11-28T18:46:54.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 --2- 172.21.15.7:0/3084374757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc00c3bc0 0x7f1fc00c7c60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:54.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- 172.21.15.7:0/3084374757 >> 172.21.15.7:0/3084374757 conn(0x7f1fc00662e0 msgr2=0x7f1fc00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:54.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- 172.21.15.7:0/3084374757 shutdown_connections 2023-11-28T18:46:54.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- 172.21.15.7:0/3084374757 wait complete. 2023-11-28T18:46:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 Processor -- start 2023-11-28T18:46:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- start start 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc00c3490 0x7f1fc01eba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1fc00c3bc0 0x7f1fc014c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc01503c0 0x7f1fc01eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc014cd00 con 0x7f1fc00c3490 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc01ef2e0 con 0x7f1fc01503c0 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fc55b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc01eb580 con 0x7f1fc00c3bc0 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbeffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc00c3490 0x7f1fc01eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbeffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc00c3490 0x7f1fc01eba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60664/0 (socket says 172.21.15.7:60664) 2023-11-28T18:46:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbeffd640 1 -- 172.21.15.7:0/1597674293 learned_addr learned my addr 172.21.15.7:0/1597674293 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbf7fe640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc01503c0 0x7f1fc01eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fb7fff640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1fc00c3bc0 0x7f1fc014c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbeffd640 1 -- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1fc00c3bc0 msgr2=0x7f1fc014c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbeffd640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1fc00c3bc0 0x7f1fc014c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbeffd640 1 -- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc01503c0 msgr2=0x7f1fc01eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbeffd640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc01503c0 0x7f1fc01eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbeffd640 1 -- 172.21.15.7:0/1597674293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa8009420 con 0x7f1fc00c3490 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fb7fff640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1fc00c3bc0 0x7f1fc014c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.481+0000 7f1fbf7fe640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc01503c0 0x7f1fc01eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fbeffd640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc00c3490 0x7f1fc01eba10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000c440 tx=0x7f1fb000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fb77fe640 1 -- 172.21.15.7:0/1597674293 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb00054e0 con 0x7f1fc00c3490 2023-11-28T18:46:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fb77fe640 1 -- 172.21.15.7:0/1597674293 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb0003600 con 0x7f1fc00c3490 2023-11-28T18:46:54.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc00c7200 con 0x7f1fc00c3490 2023-11-28T18:46:54.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fb77fe640 1 -- 172.21.15.7:0/1597674293 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0003990 con 0x7f1fc00c3490 2023-11-28T18:46:54.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc00c7780 con 0x7f1fc00c3490 2023-11-28T18:46:54.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc004e890 con 0x7f1fc00c3490 2023-11-28T18:46:54.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fb77fe640 1 -- 172.21.15.7:0/1597674293 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1fb0003af0 con 0x7f1fc00c3490 2023-11-28T18:46:54.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fb77fe640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1fa0066900 0x7f1fa0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:54.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fb77fe640 1 -- 172.21.15.7:0/1597674293 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f1fb0083e10 con 0x7f1fc00c3490 2023-11-28T18:46:54.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.485+0000 7f1fb7fff640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1fa0066900 0x7f1fa0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:54.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.489+0000 7f1fb7fff640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1fa0066900 0x7f1fa0068dc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80b2000 tx=0x7f1fa80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:54.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.489+0000 7f1fb77fe640 1 -- 172.21.15.7:0/1597674293 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1fb0087030 con 0x7f1fc00c3490 2023-11-28T18:46:54.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fc0154000 con 0x7f1fc00c3490 2023-11-28T18:46:54.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fb77fe640 1 -- 172.21.15.7:0/1597674293 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f1fb0017020 con 0x7f1fc00c3490 2023-11-28T18:46:54.687 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:54.687 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:54.687 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:54.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1fa0066900 msgr2=0x7f1fa0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:54.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1fa0066900 0x7f1fa0068dc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80b2000 tx=0x7f1fa80aa070 comp rx=0 tx=0).stop 2023-11-28T18:46:54.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc00c3490 msgr2=0x7f1fc01eba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:54.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc00c3490 0x7f1fc01eba10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000c440 tx=0x7f1fb000c900 comp rx=0 tx=0).stop 2023-11-28T18:46:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 shutdown_connections 2023-11-28T18:46:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1fa0066900 0x7f1fa0068dc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fc01503c0 0x7f1fc01eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1fc00c3bc0 0x7f1fc014c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 --2- 172.21.15.7:0/1597674293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc00c3490 0x7f1fc01eba10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 >> 172.21.15.7:0/1597674293 conn(0x7f1fc00662e0 msgr2=0x7f1fc014a850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.685+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 shutdown_connections 2023-11-28T18:46:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:54.689+0000 7f1fc55b8640 1 -- 172.21.15.7:0/1597674293 wait complete. 2023-11-28T18:46:54.709 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:56.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017fe32640 1 Processor -- start 2023-11-28T18:46:56.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017fe32640 1 -- start start 2023-11-28T18:46:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017fe32640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0178151cd0 0x7f017815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017fe32640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f017806e560 0x7f0178157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017fe32640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0178071e60 con 0x7f01780701f0 2023-11-28T18:46:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017fe32640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0178070930 con 0x7f017806e560 2023-11-28T18:46:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017fe32640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0178070bf0 con 0x7f0178151cd0 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017d3cd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f017806e560 0x7f0178157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017d3cd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f017806e560 0x7f0178157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60668/0 (socket says 172.21.15.7:60668) 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017d3cd640 1 -- 172.21.15.7:0/2157571753 learned_addr learned my addr 172.21.15.7:0/2157571753 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.865+0000 7f017dbce640 1 --2- 172.21.15.7:0/2157571753 >> v2:172.21.15.193:3300/0 conn(0x7f0178151cd0 0x7f017815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017d3cd640 1 -- 172.21.15.7:0/2157571753 >> v2:172.21.15.193:3300/0 conn(0x7f0178151cd0 msgr2=0x7f017815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017d3cd640 1 --2- 172.21.15.7:0/2157571753 >> v2:172.21.15.193:3300/0 conn(0x7f0178151cd0 0x7f017815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017d3cd640 1 -- 172.21.15.7:0/2157571753 >> v1:172.21.15.193:6789/0 conn(0x7f01780701f0 legacy=0x7f01780c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017d3cd640 1 -- 172.21.15.7:0/2157571753 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0178070ed0 con 0x7f017806e560 2023-11-28T18:46:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017dbce640 1 --2- 172.21.15.7:0/2157571753 >> v2:172.21.15.193:3300/0 conn(0x7f0178151cd0 0x7f017815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:46:56.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017d3cd640 1 --2- 172.21.15.7:0/2157571753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f017806e560 0x7f0178157b90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f017806fee0 tx=0x7f016801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7bf338bb4a8f843 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:56.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017cbcc640 1 -- 172.21.15.7:0/2157571753 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01680ad660 con 0x7f017806e560 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017cbcc640 1 -- 172.21.15.7:0/2157571753 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01680ad7c0 con 0x7f017806e560 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017cbcc640 1 -- 172.21.15.7:0/2157571753 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01680adb50 con 0x7f017806e560 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017fe32640 1 -- 172.21.15.7:0/2157571753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f017806e560 msgr2=0x7f0178157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.869+0000 7f017fe32640 1 --2- 172.21.15.7:0/2157571753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f017806e560 0x7f0178157b90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f017806fee0 tx=0x7f016801c470 comp rx=0 tx=0).stop 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 -- 172.21.15.7:0/2157571753 shutdown_connections 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 --2- 172.21.15.7:0/2157571753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f017806e560 0x7f0178157b90 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 --2- 172.21.15.7:0/2157571753 >> v2:172.21.15.193:3300/0 conn(0x7f0178151cd0 0x7f017815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 -- 172.21.15.7:0/2157571753 >> 172.21.15.7:0/2157571753 conn(0x7f01780662e0 msgr2=0x7f01780666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 -- 172.21.15.7:0/2157571753 shutdown_connections 2023-11-28T18:46:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 -- 172.21.15.7:0/2157571753 wait complete. 2023-11-28T18:46:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 Processor -- start 2023-11-28T18:46:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 -- start start 2023-11-28T18:46:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f017806e560 0x7f01781f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01780701f0 0x7f0178158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0178151cd0 0x7f0178145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0178159e20 con 0x7f0178151cd0 2023-11-28T18:46:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01781616d0 con 0x7f017806e560 2023-11-28T18:46:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017fe32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0178161d00 con 0x7f01780701f0 2023-11-28T18:46:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0178151cd0 0x7f0178145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017dbce640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f017806e560 0x7f01781f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0178151cd0 0x7f0178145d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60670/0 (socket says 172.21.15.7:60670) 2023-11-28T18:46:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017dbce640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f017806e560 0x7f01781f34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:36716/0 (socket says 172.21.15.7:36716) 2023-11-28T18:46:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 -- 172.21.15.7:0/537703747 learned_addr learned my addr 172.21.15.7:0/537703747 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017d3cd640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01780701f0 0x7f0178158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:56.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 -- 172.21.15.7:0/537703747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01780701f0 msgr2=0x7f0178158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:56.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01780701f0 0x7f0178158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:56.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 -- 172.21.15.7:0/537703747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f017806e560 msgr2=0x7f01781f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:56.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f017806e560 0x7f01781f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:56.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 -- 172.21.15.7:0/537703747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01680094d0 con 0x7f0178151cd0 2023-11-28T18:46:56.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.873+0000 7f017e3cf640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0178151cd0 0x7f0178145d00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f017000c770 tx=0x7f017000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f016e7fc640 1 -- 172.21.15.7:0/537703747 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0170003b80 con 0x7f0178151cd0 2023-11-28T18:46:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f016e7fc640 1 -- 172.21.15.7:0/537703747 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0170003ce0 con 0x7f0178151cd0 2023-11-28T18:46:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f016e7fc640 1 -- 172.21.15.7:0/537703747 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017000aa90 con 0x7f0178151cd0 2023-11-28T18:46:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017815ae40 con 0x7f0178151cd0 2023-11-28T18:46:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01781f2ce0 con 0x7f0178151cd0 2023-11-28T18:46:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017804e890 con 0x7f0178151cd0 2023-11-28T18:46:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f016e7fc640 1 -- 172.21.15.7:0/537703747 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f017000abf0 con 0x7f0178151cd0 2023-11-28T18:46:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f016e7fc640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f015c066900 0x7f015c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f016e7fc640 1 -- 172.21.15.7:0/537703747 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f01700846a0 con 0x7f0178151cd0 2023-11-28T18:46:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f017dbce640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f015c066900 0x7f015c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.877+0000 7f017dbce640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f015c066900 0x7f015c068dc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f01740045f0 tx=0x7f0174008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:56.881+0000 7f016e7fc640 1 -- 172.21.15.7:0/537703747 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01700513e0 con 0x7f0178151cd0 2023-11-28T18:46:57.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.073+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0178150ec0 con 0x7f0178151cd0 2023-11-28T18:46:57.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f016e7fc640 1 -- 172.21.15.7:0/537703747 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f0170054a00 con 0x7f0178151cd0 2023-11-28T18:46:57.077 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:57.077 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:57.078 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:57.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f015c066900 msgr2=0x7f015c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:57.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f015c066900 0x7f015c068dc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f01740045f0 tx=0x7f0174008040 comp rx=0 tx=0).stop 2023-11-28T18:46:57.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0178151cd0 msgr2=0x7f0178145d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0178151cd0 0x7f0178145d00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f017000c770 tx=0x7f017000cc30 comp rx=0 tx=0).stop 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 shutdown_connections 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f015c066900 0x7f015c068dc0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0178151cd0 0x7f0178145d00 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01780701f0 0x7f0178158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 --2- 172.21.15.7:0/537703747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f017806e560 0x7f01781f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 >> 172.21.15.7:0/537703747 conn(0x7f01780662e0 msgr2=0x7f017814f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 shutdown_connections 2023-11-28T18:46:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:57.077+0000 7f017fe32640 1 -- 172.21.15.7:0/537703747 wait complete. 2023-11-28T18:46:57.099 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:46:59.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:46:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.253+0000 7effc705f640 1 Processor -- start 2023-11-28T18:46:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.253+0000 7effc705f640 1 -- start start 2023-11-28T18:46:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc705f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc01503c0 0x7effc00c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc705f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7effc00c3bc0 0x7effc00c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc705f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7effc006d9c0 0x7effc006de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc705f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc0157240 con 0x7effc00c3bc0 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc705f640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc00c2ce0 con 0x7effc006d9c0 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc705f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00c2f80 con 0x7effc01503c0 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effbffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7effc00c3bc0 0x7effc00c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effbffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7effc00c3bc0 0x7effc00c3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60680/0 (socket says 172.21.15.7:60680) 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effbffff640 1 -- 172.21.15.7:0/898615788 learned_addr learned my addr 172.21.15.7:0/898615788 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc4dfb640 1 --2- 172.21.15.7:0/898615788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc01503c0 0x7effc00c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc55fc640 1 --2- 172.21.15.7:0/898615788 >> v2:172.21.15.193:3300/0 conn(0x7effc006d9c0 0x7effc006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effbffff640 1 -- 172.21.15.7:0/898615788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc01503c0 msgr2=0x7effc00c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effbffff640 1 --2- 172.21.15.7:0/898615788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc01503c0 0x7effc00c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effbffff640 1 -- 172.21.15.7:0/898615788 >> v2:172.21.15.193:3300/0 conn(0x7effc006d9c0 msgr2=0x7effc006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:46:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effbffff640 1 --2- 172.21.15.7:0/898615788 >> v2:172.21.15.193:3300/0 conn(0x7effc006d9c0 0x7effc006de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effbffff640 1 -- 172.21.15.7:0/898615788 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc0156f60 con 0x7effc00c3bc0 2023-11-28T18:46:59.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.257+0000 7effc55fc640 1 --2- 172.21.15.7:0/898615788 >> v2:172.21.15.193:3300/0 conn(0x7effc006d9c0 0x7effc006de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:46:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effbffff640 1 --2- 172.21.15.7:0/898615788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc00c3bc0 0x7effc00c3490 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7effc0067290 tx=0x7effac0200c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fb3c037658df80 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effbf7fe640 1 -- 172.21.15.7:0/898615788 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac0a9e90 con 0x7effc00c3bc0 2023-11-28T18:46:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effbf7fe640 1 -- 172.21.15.7:0/898615788 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac0b1440 con 0x7effc00c3bc0 2023-11-28T18:46:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effbf7fe640 1 -- 172.21.15.7:0/898615788 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac0b17d0 con 0x7effc00c3bc0 2023-11-28T18:46:59.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 -- 172.21.15.7:0/898615788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc00c3bc0 msgr2=0x7effc00c3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:59.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 --2- 172.21.15.7:0/898615788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc00c3bc0 0x7effc00c3490 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7effc0067290 tx=0x7effac0200c0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 -- 172.21.15.7:0/898615788 shutdown_connections 2023-11-28T18:46:59.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 --2- 172.21.15.7:0/898615788 >> v2:172.21.15.193:3300/0 conn(0x7effc006d9c0 0x7effc006de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 --2- 172.21.15.7:0/898615788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc00c3bc0 0x7effc00c3490 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 --2- 172.21.15.7:0/898615788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc01503c0 0x7effc00c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 -- 172.21.15.7:0/898615788 >> 172.21.15.7:0/898615788 conn(0x7effc00662e0 msgr2=0x7effc00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:59.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 -- 172.21.15.7:0/898615788 shutdown_connections 2023-11-28T18:46:59.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 -- 172.21.15.7:0/898615788 wait complete. 2023-11-28T18:46:59.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 Processor -- start 2023-11-28T18:46:59.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.261+0000 7effc705f640 1 -- start start 2023-11-28T18:46:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc006d9c0 0x7effc0158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effc00c3bc0 0x7effc01efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc01503c0 0x7effc0071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0159640 con 0x7effc01503c0 2023-11-28T18:46:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00c1720 con 0x7effc00c3bc0 2023-11-28T18:46:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc0159360 con 0x7effc006d9c0 2023-11-28T18:46:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc01503c0 0x7effc0071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc01503c0 0x7effc0071390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60694/0 (socket says 172.21.15.7:60694) 2023-11-28T18:46:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 -- 172.21.15.7:0/437836144 learned_addr learned my addr 172.21.15.7:0/437836144 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:46:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effbffff640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effc00c3bc0 0x7effc01efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc4dfb640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc006d9c0 0x7effc0158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:59.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 -- 172.21.15.7:0/437836144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc006d9c0 msgr2=0x7effc0158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:59.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc006d9c0 0x7effc0158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 -- 172.21.15.7:0/437836144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effc00c3bc0 msgr2=0x7effc01efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:59.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effc00c3bc0 0x7effc01efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 -- 172.21.15.7:0/437836144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac00a670 con 0x7effc01503c0 2023-11-28T18:46:59.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc55fc640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc01503c0 0x7effc0071390 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7effb800cef0 tx=0x7effb800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effbd7fa640 1 -- 172.21.15.7:0/437836144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb800d840 con 0x7effc01503c0 2023-11-28T18:46:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effbd7fa640 1 -- 172.21.15.7:0/437836144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb800d9c0 con 0x7effc01503c0 2023-11-28T18:46:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effbd7fa640 1 -- 172.21.15.7:0/437836144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb800dd70 con 0x7effc01503c0 2023-11-28T18:46:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc015a010 con 0x7effc01503c0 2023-11-28T18:46:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc01ef880 con 0x7effc01503c0 2023-11-28T18:46:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc01efa70 con 0x7effc01503c0 2023-11-28T18:46:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.265+0000 7effbd7fa640 1 -- 172.21.15.7:0/437836144 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7effb801e3b0 con 0x7effc01503c0 2023-11-28T18:46:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.269+0000 7effbd7fa640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7effa4066920 0x7effa4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:46:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.269+0000 7effbd7fa640 1 -- 172.21.15.7:0/437836144 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7effb8085350 con 0x7effc01503c0 2023-11-28T18:46:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.269+0000 7effc4dfb640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7effa4066920 0x7effa4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:46:59.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.269+0000 7effc4dfb640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7effa4066920 0x7effa4068de0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7effb4007940 tx=0x7effb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:46:59.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.273+0000 7effbd7fa640 1 -- 172.21.15.7:0/437836144 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7effb8052260 con 0x7effc01503c0 2023-11-28T18:46:59.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.465+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effc004e890 con 0x7effc01503c0 2023-11-28T18:46:59.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.465+0000 7effbd7fa640 1 -- 172.21.15.7:0/437836144 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7effb8055880 con 0x7effc01503c0 2023-11-28T18:46:59.468 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:46:59.469 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:46:59.469 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:46:59.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7effa4066920 msgr2=0x7effa4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:59.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7effa4066920 0x7effa4068de0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7effb4007940 tx=0x7effb4008040 comp rx=0 tx=0).stop 2023-11-28T18:46:59.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc01503c0 msgr2=0x7effc0071390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:46:59.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc01503c0 0x7effc0071390 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7effb800cef0 tx=0x7effb800a710 comp rx=0 tx=0).stop 2023-11-28T18:46:59.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 shutdown_connections 2023-11-28T18:46:59.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7effa4066920 0x7effa4068de0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc01503c0 0x7effc0071390 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effc00c3bc0 0x7effc01efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 --2- 172.21.15.7:0/437836144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7effc006d9c0 0x7effc0158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:46:59.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 >> 172.21.15.7:0/437836144 conn(0x7effc00662e0 msgr2=0x7effc00727f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:46:59.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 shutdown_connections 2023-11-28T18:46:59.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:46:59.469+0000 7effc705f640 1 -- 172.21.15.7:0/437836144 wait complete. 2023-11-28T18:46:59.491 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:47:00.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~101s 2023-11-28T18:47:01.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:01.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.949+0000 7fd645279640 1 Processor -- start 2023-11-28T18:47:01.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.949+0000 7fd645279640 1 -- start start 2023-11-28T18:47:01.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd645279640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fd6401542e0 0x7fd64015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:01.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd645279640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6400cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:01.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd645279640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6400ca920 con 0x7fd6400cc810 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd645279640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd640071d50 con 0x7fd6401542e0 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd645279640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd640072030 con 0x7fd6400cbb70 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd6400cc810 0x7fd640072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42218/0 (socket says 172.21.15.7:42218) 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63f7fe640 1 -- 172.21.15.7:0/401237168 learned_addr learned my addr 172.21.15.7:0/401237168 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63e7fc640 1 --2- 172.21.15.7:0/401237168 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6400cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63effd640 1 --2- 172.21.15.7:0/401237168 >> v2:172.21.15.193:3300/0 conn(0x7fd6401542e0 0x7fd64015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63dffb640 1 -- 172.21.15.7:0/401237168 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364823532 0 0) 0x7fd6400ca920 con 0x7fd6400cc810 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63dffb640 1 -- 172.21.15.7:0/401237168 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd628003530 con 0x7fd6400cc810 2023-11-28T18:47:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63e7fc640 1 -- 172.21.15.7:0/401237168 >> v2:172.21.15.193:3300/0 conn(0x7fd6401542e0 msgr2=0x7fd64015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:01.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63e7fc640 1 --2- 172.21.15.7:0/401237168 >> v2:172.21.15.193:3300/0 conn(0x7fd6401542e0 0x7fd64015e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:01.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63e7fc640 1 -- 172.21.15.7:0/401237168 >> v1:172.21.15.7:6789/0 conn(0x7fd6400cc810 legacy=0x7fd640072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:01.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.953+0000 7fd63e7fc640 1 -- 172.21.15.7:0/401237168 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64015e550 con 0x7fd6400cbb70 2023-11-28T18:47:01.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd63e7fc640 1 --2- 172.21.15.7:0/401237168 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6400cc2d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd6340097c0 tx=0x7fd63401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cf143585334206c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:01.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd63dffb640 1 -- 172.21.15.7:0/401237168 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6340ab680 con 0x7fd6400cbb70 2023-11-28T18:47:01.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd63dffb640 1 -- 172.21.15.7:0/401237168 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6340ab7e0 con 0x7fd6400cbb70 2023-11-28T18:47:01.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd63dffb640 1 -- 172.21.15.7:0/401237168 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6340abb70 con 0x7fd6400cbb70 2023-11-28T18:47:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 -- 172.21.15.7:0/401237168 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 msgr2=0x7fd6400cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 --2- 172.21.15.7:0/401237168 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6400cc2d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd6340097c0 tx=0x7fd63401c480 comp rx=0 tx=0).stop 2023-11-28T18:47:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 -- 172.21.15.7:0/401237168 shutdown_connections 2023-11-28T18:47:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 --2- 172.21.15.7:0/401237168 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6400cc2d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 --2- 172.21.15.7:0/401237168 >> v2:172.21.15.193:3300/0 conn(0x7fd6401542e0 0x7fd64015e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 -- 172.21.15.7:0/401237168 >> 172.21.15.7:0/401237168 conn(0x7fd6400717e0 msgr2=0x7fd6400662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 -- 172.21.15.7:0/401237168 shutdown_connections 2023-11-28T18:47:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 -- 172.21.15.7:0/401237168 wait complete. 2023-11-28T18:47:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 Processor -- start 2023-11-28T18:47:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.957+0000 7fd645279640 1 -- start start 2023-11-28T18:47:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6401432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6400cc810 0x7fd6401f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6401542e0 0x7fd6401f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6401f2380 con 0x7fd6401542e0 2023-11-28T18:47:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64013eed0 con 0x7fd6400cc810 2023-11-28T18:47:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd640140800 con 0x7fd6400cbb70 2023-11-28T18:47:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6401542e0 0x7fd6401f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6401542e0 0x7fd6401f2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60706/0 (socket says 172.21.15.7:60706) 2023-11-28T18:47:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 -- 172.21.15.7:0/149530883 learned_addr learned my addr 172.21.15.7:0/149530883 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63e7fc640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6400cc810 0x7fd6401f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63effd640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6401432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 -- 172.21.15.7:0/149530883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 msgr2=0x7fd6401432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6401432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 -- 172.21.15.7:0/149530883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6400cc810 msgr2=0x7fd6401f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6400cc810 0x7fd6401f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 -- 172.21.15.7:0/149530883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634009420 con 0x7fd6401542e0 2023-11-28T18:47:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd63f7fe640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6401542e0 0x7fd6401f2cf0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd63800cb40 tx=0x7fd63800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd6277fe640 1 -- 172.21.15.7:0/149530883 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63800aca0 con 0x7fd6401542e0 2023-11-28T18:47:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd6277fe640 1 -- 172.21.15.7:0/149530883 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63800ae00 con 0x7fd6401542e0 2023-11-28T18:47:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd6277fe640 1 -- 172.21.15.7:0/149530883 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638012620 con 0x7fd6401542e0 2023-11-28T18:47:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd640140aa0 con 0x7fd6401542e0 2023-11-28T18:47:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6400c7090 con 0x7fd6401542e0 2023-11-28T18:47:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.961+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64006c670 con 0x7fd6401542e0 2023-11-28T18:47:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.965+0000 7fd6277fe640 1 -- 172.21.15.7:0/149530883 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd638007070 con 0x7fd6401542e0 2023-11-28T18:47:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.965+0000 7fd6277fe640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd62c0668d0 0x7fd62c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.965+0000 7fd6277fe640 1 -- 172.21.15.7:0/149530883 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fd638084980 con 0x7fd6401542e0 2023-11-28T18:47:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.965+0000 7fd63effd640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd62c0668d0 0x7fd62c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.965+0000 7fd63effd640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd62c0668d0 0x7fd62c068d90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd630009b60 tx=0x7fd630009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:01.969+0000 7fd6277fe640 1 -- 172.21.15.7:0/149530883 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd6380516c0 con 0x7fd6401542e0 2023-11-28T18:47:02.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.157+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6401f24d0 con 0x7fd6401542e0 2023-11-28T18:47:02.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.157+0000 7fd6277fe640 1 -- 172.21.15.7:0/149530883 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fd6401f24d0 con 0x7fd6401542e0 2023-11-28T18:47:02.160 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:02.160 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:02.160 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:02.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd62c0668d0 msgr2=0x7fd62c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:02.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd62c0668d0 0x7fd62c068d90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd630009b60 tx=0x7fd630009190 comp rx=0 tx=0).stop 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6401542e0 msgr2=0x7fd6401f2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6401542e0 0x7fd6401f2cf0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd63800cb40 tx=0x7fd63800a060 comp rx=0 tx=0).stop 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 shutdown_connections 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd62c0668d0 0x7fd62c068d90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6401542e0 0x7fd6401f2cf0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6400cc810 0x7fd6401f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 --2- 172.21.15.7:0/149530883 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd6400cbb70 0x7fd6401432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 >> 172.21.15.7:0/149530883 conn(0x7fd6400717e0 msgr2=0x7fd6400b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 shutdown_connections 2023-11-28T18:47:02.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:02.161+0000 7fd645279640 1 -- 172.21.15.7:0/149530883 wait complete. 2023-11-28T18:47:02.181 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:04.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa172474640 1 Processor -- start 2023-11-28T18:47:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa172474640 1 -- start start 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa172474640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa16c15ce10 0x7fa16c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa172474640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa16c0cafe0 0x7fa16c0ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa172474640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c15a3a0 con 0x7fa16c15ca30 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa172474640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16c15a620 con 0x7fa16c0cafe0 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa172474640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16c15a900 con 0x7fa16c15ce10 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa171472640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa16c15ca30 0x7fa16c159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34588/0 (socket says 172.21.15.7:34588) 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa171472640 1 -- 172.21.15.7:0/2757279812 learned_addr learned my addr 172.21.15.7:0/2757279812 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa171c73640 1 --2- 172.21.15.7:0/2757279812 >> v2:172.21.15.7:3300/0 conn(0x7fa16c0cafe0 0x7fa16c0ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa170c71640 1 --2- 172.21.15.7:0/2757279812 >> v2:172.21.15.193:3300/0 conn(0x7fa16c15ce10 0x7fa16c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:04.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa163fff640 1 -- 172.21.15.7:0/2757279812 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018856352 0 0) 0x7fa16c15a3a0 con 0x7fa16c15ca30 2023-11-28T18:47:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa163fff640 1 -- 172.21.15.7:0/2757279812 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa158003530 con 0x7fa16c15ca30 2023-11-28T18:47:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa171c73640 1 -- 172.21.15.7:0/2757279812 >> v2:172.21.15.193:3300/0 conn(0x7fa16c15ce10 msgr2=0x7fa16c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa171c73640 1 --2- 172.21.15.7:0/2757279812 >> v2:172.21.15.193:3300/0 conn(0x7fa16c15ce10 0x7fa16c0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa171c73640 1 -- 172.21.15.7:0/2757279812 >> v1:172.21.15.7:6789/0 conn(0x7fa16c15ca30 legacy=0x7fa16c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.337+0000 7fa171c73640 1 -- 172.21.15.7:0/2757279812 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c15abe0 con 0x7fa16c0cafe0 2023-11-28T18:47:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa171c73640 1 --2- 172.21.15.7:0/2757279812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0ca710 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0d0200 tx=0x7fa16401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abf04b61aa12dc1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa163fff640 1 -- 172.21.15.7:0/2757279812 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1640a59a0 con 0x7fa16c0cafe0 2023-11-28T18:47:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa163fff640 1 -- 172.21.15.7:0/2757279812 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1640a5b00 con 0x7fa16c0cafe0 2023-11-28T18:47:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa163fff640 1 -- 172.21.15.7:0/2757279812 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1640a5e90 con 0x7fa16c0cafe0 2023-11-28T18:47:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa172474640 1 -- 172.21.15.7:0/2757279812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 msgr2=0x7fa16c0ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa172474640 1 --2- 172.21.15.7:0/2757279812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0ca710 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0d0200 tx=0x7fa16401c8e0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa172474640 1 -- 172.21.15.7:0/2757279812 shutdown_connections 2023-11-28T18:47:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa172474640 1 --2- 172.21.15.7:0/2757279812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0ca710 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa172474640 1 --2- 172.21.15.7:0/2757279812 >> v2:172.21.15.193:3300/0 conn(0x7fa16c15ce10 0x7fa16c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa172474640 1 -- 172.21.15.7:0/2757279812 >> 172.21.15.7:0/2757279812 conn(0x7fa16c06c060 msgr2=0x7fa16c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa172474640 1 -- 172.21.15.7:0/2757279812 shutdown_connections 2023-11-28T18:47:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.341+0000 7fa172474640 1 -- 172.21.15.7:0/2757279812 wait complete. 2023-11-28T18:47:04.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 Processor -- start 2023-11-28T18:47:04.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 -- start start 2023-11-28T18:47:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa16c15ca30 0x7fa16c1f5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa16c15ce10 0x7fa16c1f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c156220 con 0x7fa16c0cafe0 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c150b80 con 0x7fa16c15ca30 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c150e60 con 0x7fa16c15ce10 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58564/0 (socket says 172.21.15.7:58564) 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 -- 172.21.15.7:0/990644253 learned_addr learned my addr 172.21.15.7:0/990644253 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa170c71640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa16c15ca30 0x7fa16c1f5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171c73640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa16c15ce10 0x7fa16c1f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 -- 172.21.15.7:0/990644253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa16c15ce10 msgr2=0x7fa16c1f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa16c15ce10 0x7fa16c1f6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 -- 172.21.15.7:0/990644253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa16c15ca30 msgr2=0x7fa16c1f5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa16c15ca30 0x7fa16c1f5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 -- 172.21.15.7:0/990644253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa164009a70 con 0x7fa16c0cafe0 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa171472640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0c3f80 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c4d0 tx=0x7fa16800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa161ffb640 1 -- 172.21.15.7:0/990644253 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168005cf0 con 0x7fa16c0cafe0 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa161ffb640 1 -- 172.21.15.7:0/990644253 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168005e50 con 0x7fa16c0cafe0 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa161ffb640 1 -- 172.21.15.7:0/990644253 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168007270 con 0x7fa16c0cafe0 2023-11-28T18:47:04.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16c151140 con 0x7fa16c0cafe0 2023-11-28T18:47:04.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16c155a20 con 0x7fa16c0cafe0 2023-11-28T18:47:04.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.345+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa16c155be0 con 0x7fa16c0cafe0 2023-11-28T18:47:04.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.349+0000 7fa161ffb640 1 -- 172.21.15.7:0/990644253 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa1680073d0 con 0x7fa16c0cafe0 2023-11-28T18:47:04.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.349+0000 7fa161ffb640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1500668d0 0x7fa150068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:04.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.349+0000 7fa161ffb640 1 -- 172.21.15.7:0/990644253 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fa168083d20 con 0x7fa16c0cafe0 2023-11-28T18:47:04.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.349+0000 7fa170c71640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1500668d0 0x7fa150068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:04.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.349+0000 7fa170c71640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1500668d0 0x7fa150068d90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa16c14e300 tx=0x7fa15c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:04.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.353+0000 7fa161ffb640 1 -- 172.21.15.7:0/990644253 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa168050c20 con 0x7fa16c0cafe0 2023-11-28T18:47:04.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.541+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa16c04e890 con 0x7fa16c0cafe0 2023-11-28T18:47:04.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.541+0000 7fa161ffb640 1 -- 172.21.15.7:0/990644253 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fa168017020 con 0x7fa16c0cafe0 2023-11-28T18:47:04.544 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:04.544 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:04.545 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1500668d0 msgr2=0x7fa150068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1500668d0 0x7fa150068d90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa16c14e300 tx=0x7fa15c008040 comp rx=0 tx=0).stop 2023-11-28T18:47:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 msgr2=0x7fa16c0c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0c3f80 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c4d0 tx=0x7fa16800c990 comp rx=0 tx=0).stop 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 shutdown_connections 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa1500668d0 0x7fa150068d90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa16c15ce10 0x7fa16c1f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa16c15ca30 0x7fa16c1f5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 --2- 172.21.15.7:0/990644253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa16c0cafe0 0x7fa16c0c3f80 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 >> 172.21.15.7:0/990644253 conn(0x7fa16c06c060 msgr2=0x7fa16c0ceec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 shutdown_connections 2023-11-28T18:47:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:04.545+0000 7fa172474640 1 -- 172.21.15.7:0/990644253 wait complete. 2023-11-28T18:47:04.566 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:06.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:06.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052a2fe640 1 Processor -- start 2023-11-28T18:47:06.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052a2fe640 1 -- start start 2023-11-28T18:47:06.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052a2fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05240cbb70 0x7f05240cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052a2fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f05240cc810 0x7f0524072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052a2fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05240ca920 con 0x7f05241542e0 2023-11-28T18:47:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052a2fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0524071d50 con 0x7f05240cc810 2023-11-28T18:47:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052a2fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0524072030 con 0x7f05240cbb70 2023-11-28T18:47:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052889b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f05240cc810 0x7f0524072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f051bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05240cbb70 0x7f05240cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052889b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f05240cc810 0x7f0524072450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58574/0 (socket says 172.21.15.7:58574) 2023-11-28T18:47:06.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052889b640 1 -- 172.21.15.7:0/2024138091 learned_addr learned my addr 172.21.15.7:0/2024138091 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:06.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052889b640 1 -- 172.21.15.7:0/2024138091 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05240cbb70 msgr2=0x7f05240cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:06.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052889b640 1 --2- 172.21.15.7:0/2024138091 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05240cbb70 0x7f05240cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052889b640 1 -- 172.21.15.7:0/2024138091 >> v1:172.21.15.193:6789/0 conn(0x7f05241542e0 legacy=0x7f052415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:06.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f052889b640 1 -- 172.21.15.7:0/2024138091 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05240c4cd0 con 0x7f05240cc810 2023-11-28T18:47:06.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.721+0000 7f05237fe640 1 -- 172.21.15.7:0/2024138091 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507268646 0 0) 0x7f05240ca920 con 0x7f05241542e0 2023-11-28T18:47:06.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052889b640 1 --2- 172.21.15.7:0/2024138091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f0524072450 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f051c009e50 tx=0x7f051c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2d5d1bf9c50f62 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:06.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f05237fe640 1 -- 172.21.15.7:0/2024138091 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c0a57f0 con 0x7f05240cc810 2023-11-28T18:47:06.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f05237fe640 1 -- 172.21.15.7:0/2024138091 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051c0a5950 con 0x7f05240cc810 2023-11-28T18:47:06.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f05237fe640 1 -- 172.21.15.7:0/2024138091 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c0a5ce0 con 0x7f05240cc810 2023-11-28T18:47:06.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052a2fe640 1 -- 172.21.15.7:0/2024138091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 msgr2=0x7f0524072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:06.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2024138091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f0524072450 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f051c009e50 tx=0x7f051c01ca10 comp rx=0 tx=0).stop 2023-11-28T18:47:06.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052a2fe640 1 -- 172.21.15.7:0/2024138091 shutdown_connections 2023-11-28T18:47:06.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2024138091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f0524072450 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2024138091 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05240cbb70 0x7f05240cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052a2fe640 1 -- 172.21.15.7:0/2024138091 >> 172.21.15.7:0/2024138091 conn(0x7f05240717e0 msgr2=0x7f05240662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052a2fe640 1 -- 172.21.15.7:0/2024138091 shutdown_connections 2023-11-28T18:47:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.725+0000 7f052a2fe640 1 -- 172.21.15.7:0/2024138091 wait complete. 2023-11-28T18:47:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 Processor -- start 2023-11-28T18:47:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 -- start start 2023-11-28T18:47:06.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05240cbb70 0x7f05241f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f05241488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05241542e0 0x7f052414c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05241f29d0 con 0x7f05240cc810 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05240ce300 con 0x7f05240cbb70 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05240ce5e0 con 0x7f05241542e0 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f05241488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f05241488b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58588/0 (socket says 172.21.15.7:58588) 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 -- 172.21.15.7:0/2422484467 learned_addr learned my addr 172.21.15.7:0/2422484467 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 -- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05241542e0 msgr2=0x7f052414c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052889b640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05241542e0 0x7f052414c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f0523fff640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05240cbb70 0x7f05241f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05241542e0 0x7f052414c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 -- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05240cbb70 msgr2=0x7f05241f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05240cbb70 0x7f05241f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 -- 172.21.15.7:0/2422484467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c009b30 con 0x7f05240cc810 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052889b640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05241542e0 0x7f052414c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f0523fff640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05240cbb70 0x7f05241f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f051bfff640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f05241488b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f050c00e930 tx=0x7f050c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f05217fa640 1 -- 172.21.15.7:0/2422484467 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c009690 con 0x7f05240cc810 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f05217fa640 1 -- 172.21.15.7:0/2422484467 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c00ca00 con 0x7f05240cc810 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f05217fa640 1 -- 172.21.15.7:0/2422484467 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c00cd70 con 0x7f05240cc810 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05240ce8c0 con 0x7f05240cc810 2023-11-28T18:47:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.729+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05241f2f70 con 0x7f05240cc810 2023-11-28T18:47:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.733+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05241f3130 con 0x7f05240cc810 2023-11-28T18:47:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.733+0000 7f05217fa640 1 -- 172.21.15.7:0/2422484467 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f050c013470 con 0x7f05240cc810 2023-11-28T18:47:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.733+0000 7f05217fa640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f05040668d0 0x7f0504068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:06.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.733+0000 7f05217fa640 1 -- 172.21.15.7:0/2422484467 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f050c084bb0 con 0x7f05240cc810 2023-11-28T18:47:06.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.733+0000 7f0523fff640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f05040668d0 0x7f0504068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:06.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.733+0000 7f0523fff640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f05040668d0 0x7f0504068d90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0514000f70 tx=0x7f0514009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.737+0000 7f05217fa640 1 -- 172.21.15.7:0/2422484467 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f050c051990 con 0x7f05240cc810 2023-11-28T18:47:06.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.925+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f052406c670 con 0x7f05240cc810 2023-11-28T18:47:06.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.925+0000 7f05217fa640 1 -- 172.21.15.7:0/2422484467 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f050c054fb0 con 0x7f05240cc810 2023-11-28T18:47:06.928 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:06.928 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:06.929 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:06.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f05040668d0 msgr2=0x7f0504068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:06.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f05040668d0 0x7f0504068d90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0514000f70 tx=0x7f0514009040 comp rx=0 tx=0).stop 2023-11-28T18:47:06.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 msgr2=0x7f05241488b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:06.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f05241488b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f050c00e930 tx=0x7f050c00edf0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 shutdown_connections 2023-11-28T18:47:06.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f05040668d0 0x7f0504068d90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f05241542e0 0x7f052414c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05240cc810 0x7f05241488b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 --2- 172.21.15.7:0/2422484467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05240cbb70 0x7f05241f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:06.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 >> 172.21.15.7:0/2422484467 conn(0x7f05240717e0 msgr2=0x7f0524157ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:06.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 shutdown_connections 2023-11-28T18:47:06.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:06.929+0000 7f052a2fe640 1 -- 172.21.15.7:0/2422484467 wait complete. 2023-11-28T18:47:06.950 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:07.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~108s 2023-11-28T18:47:08.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:09.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.105+0000 7fbb7dfd1640 1 Processor -- start 2023-11-28T18:47:09.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.105+0000 7fbb7dfd1640 1 -- start start 2023-11-28T18:47:09.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.105+0000 7fbb7dfd1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb78151cd0 0x7fbb7815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:09.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.105+0000 7fbb7dfd1640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fbb7806e560 0x7fbb78157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:09.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.105+0000 7fbb7dfd1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78071e60 con 0x7fbb780701f0 2023-11-28T18:47:09.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.105+0000 7fbb7dfd1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb78070930 con 0x7fbb78151cd0 2023-11-28T18:47:09.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.105+0000 7fbb7dfd1640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb78070bf0 con 0x7fbb7806e560 2023-11-28T18:47:09.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb78151cd0 0x7fbb7815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:09.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb78151cd0 0x7fbb7815bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58598/0 (socket says 172.21.15.7:58598) 2023-11-28T18:47:09.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb777fe640 1 -- 172.21.15.7:0/1415589318 learned_addr learned my addr 172.21.15.7:0/1415589318 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:09.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb76ffd640 1 --2- 172.21.15.7:0/1415589318 >> v2:172.21.15.193:3300/0 conn(0x7fbb7806e560 0x7fbb78157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:09.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb777fe640 1 -- 172.21.15.7:0/1415589318 >> v2:172.21.15.193:3300/0 conn(0x7fbb7806e560 msgr2=0x7fbb78157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:09.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb777fe640 1 --2- 172.21.15.7:0/1415589318 >> v2:172.21.15.193:3300/0 conn(0x7fbb7806e560 0x7fbb78157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb777fe640 1 -- 172.21.15.7:0/1415589318 >> v1:172.21.15.193:6789/0 conn(0x7fbb780701f0 legacy=0x7fbb780c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:09.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb777fe640 1 -- 172.21.15.7:0/1415589318 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb78070ed0 con 0x7fbb78151cd0 2023-11-28T18:47:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb777fe640 1 --2- 172.21.15.7:0/1415589318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb78151cd0 0x7fbb7815bd00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbb780671b0 tx=0x7fbb6801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefbe856bcbb203 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.109+0000 7fbb767fc640 1 -- 172.21.15.7:0/1415589318 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb680ad660 con 0x7fbb78151cd0 2023-11-28T18:47:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb767fc640 1 -- 172.21.15.7:0/1415589318 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb680ad7c0 con 0x7fbb78151cd0 2023-11-28T18:47:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb767fc640 1 -- 172.21.15.7:0/1415589318 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb680adb50 con 0x7fbb78151cd0 2023-11-28T18:47:09.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1415589318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb78151cd0 msgr2=0x7fbb7815bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:09.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1415589318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb78151cd0 0x7fbb7815bd00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbb780671b0 tx=0x7fbb6801c470 comp rx=0 tx=0).stop 2023-11-28T18:47:09.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1415589318 shutdown_connections 2023-11-28T18:47:09.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1415589318 >> v2:172.21.15.193:3300/0 conn(0x7fbb7806e560 0x7fbb78157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1415589318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb78151cd0 0x7fbb7815bd00 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1415589318 >> 172.21.15.7:0/1415589318 conn(0x7fbb780662e0 msgr2=0x7fbb780666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:09.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1415589318 shutdown_connections 2023-11-28T18:47:09.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1415589318 wait complete. 2023-11-28T18:47:09.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 Processor -- start 2023-11-28T18:47:09.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- start start 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb7806e560 0x7fbb78156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb780701f0 0x7fbb78155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb78151cd0 0x7fbb780c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78157080 con 0x7fbb780701f0 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7815a0e0 con 0x7fbb7806e560 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb7dfd1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb780c6fa0 con 0x7fbb78151cd0 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb76ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb780701f0 0x7fbb78155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb76ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb780701f0 0x7fbb78155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58606/0 (socket says 172.21.15.7:58606) 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb76ffd640 1 -- 172.21.15.7:0/1722916233 learned_addr learned my addr 172.21.15.7:0/1722916233 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb777fe640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb7806e560 0x7fbb78156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:09.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.113+0000 7fbb77fff640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb78151cd0 0x7fbb780c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb76ffd640 1 -- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb78151cd0 msgr2=0x7fbb780c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb76ffd640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb78151cd0 0x7fbb780c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb76ffd640 1 -- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb7806e560 msgr2=0x7fbb78156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb76ffd640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb7806e560 0x7fbb78156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb76ffd640 1 -- 172.21.15.7:0/1722916233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb680094d0 con 0x7fbb780701f0 2023-11-28T18:47:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb77fff640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb78151cd0 0x7fbb780c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:09.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb76ffd640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb780701f0 0x7fbb78155a30 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00d830 tx=0x7fbb6c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:09.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb5ffff640 1 -- 172.21.15.7:0/1722916233 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00b6c0 con 0x7fbb780701f0 2023-11-28T18:47:09.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb5ffff640 1 -- 172.21.15.7:0/1722916233 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c00b820 con 0x7fbb780701f0 2023-11-28T18:47:09.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb5ffff640 1 -- 172.21.15.7:0/1722916233 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00bb90 con 0x7fbb780701f0 2023-11-28T18:47:09.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7815a330 con 0x7fbb780701f0 2023-11-28T18:47:09.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7815a8f0 con 0x7fbb780701f0 2023-11-28T18:47:09.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7804e890 con 0x7fbb780701f0 2023-11-28T18:47:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb5ffff640 1 -- 172.21.15.7:0/1722916233 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbb6c01e460 con 0x7fbb780701f0 2023-11-28T18:47:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb5ffff640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb640668d0 0x7fbb64068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb5ffff640 1 -- 172.21.15.7:0/1722916233 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fbb6c0850c0 con 0x7fbb780701f0 2023-11-28T18:47:09.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.117+0000 7fbb777fe640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb640668d0 0x7fbb64068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:09.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.121+0000 7fbb777fe640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb640668d0 0x7fbb64068d90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbb680af000 tx=0x7fbb68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:09.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.121+0000 7fbb5ffff640 1 -- 172.21.15.7:0/1722916233 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbb6c051de0 con 0x7fbb780701f0 2023-11-28T18:47:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.313+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb781555b0 con 0x7fbb780701f0 2023-11-28T18:47:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.313+0000 7fbb5ffff640 1 -- 172.21.15.7:0/1722916233 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fbb6c055400 con 0x7fbb780701f0 2023-11-28T18:47:09.316 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:09.316 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb640668d0 msgr2=0x7fbb64068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb640668d0 0x7fbb64068d90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbb680af000 tx=0x7fbb68008040 comp rx=0 tx=0).stop 2023-11-28T18:47:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb780701f0 msgr2=0x7fbb78155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb780701f0 0x7fbb78155a30 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00d830 tx=0x7fbb6c00dcf0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 shutdown_connections 2023-11-28T18:47:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbb640668d0 0x7fbb64068d90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb78151cd0 0x7fbb780c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb780701f0 0x7fbb78155a30 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 --2- 172.21.15.7:0/1722916233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb7806e560 0x7fbb78156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 >> 172.21.15.7:0/1722916233 conn(0x7fbb780662e0 msgr2=0x7fbb780666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 shutdown_connections 2023-11-28T18:47:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:09.317+0000 7fbb7dfd1640 1 -- 172.21.15.7:0/1722916233 wait complete. 2023-11-28T18:47:09.338 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:11.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:11.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7c0c17640 1 Processor -- start 2023-11-28T18:47:11.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7c0c17640 1 -- start start 2023-11-28T18:47:11.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7c0c17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc150b30 0x7ff7bc0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:11.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7c0c17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff7bc0c7980 0x7ff7bc0c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:11.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7c0c17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc0c6140 con 0x7ff7bc1575c0 2023-11-28T18:47:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7c0c17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7bc0c72b0 con 0x7ff7bc0c7980 2023-11-28T18:47:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7c0c17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc070c00 con 0x7ff7bc150b30 2023-11-28T18:47:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7b3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff7bc1575c0 0x7ff7bc0c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34592/0 (socket says 172.21.15.7:34592) 2023-11-28T18:47:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7b3fff640 1 -- 172.21.15.7:0/2902457144 learned_addr learned my addr 172.21.15.7:0/2902457144 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7ba59c640 1 --2- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc150b30 0x7ff7bc0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.493+0000 7ff7bad9d640 1 --2- 172.21.15.7:0/2902457144 >> v2:172.21.15.193:3300/0 conn(0x7ff7bc0c7980 0x7ff7bc0c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7b9d9b640 1 -- 172.21.15.7:0/2902457144 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221131080 0 0) 0x7ff7bc0c6140 con 0x7ff7bc1575c0 2023-11-28T18:47:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7b9d9b640 1 -- 172.21.15.7:0/2902457144 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a8003530 con 0x7ff7bc1575c0 2023-11-28T18:47:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7bad9d640 1 -- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc150b30 msgr2=0x7ff7bc0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7bad9d640 1 --2- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc150b30 0x7ff7bc0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7bad9d640 1 -- 172.21.15.7:0/2902457144 >> v1:172.21.15.7:6789/0 conn(0x7ff7bc1575c0 legacy=0x7ff7bc0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7bad9d640 1 -- 172.21.15.7:0/2902457144 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc0c8350 con 0x7ff7bc0c7980 2023-11-28T18:47:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7ba59c640 1 --2- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc150b30 0x7ff7bc0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:47:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7bad9d640 1 --2- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc0c7980 0x7ff7bc0c7e10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009d90 tx=0x7ff7b401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=259752794cf26bd5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7b9d9b640 1 -- 172.21.15.7:0/2902457144 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b40af600 con 0x7ff7bc0c7980 2023-11-28T18:47:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7b9d9b640 1 -- 172.21.15.7:0/2902457144 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b40ab050 con 0x7ff7bc0c7980 2023-11-28T18:47:11.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.497+0000 7ff7b9d9b640 1 -- 172.21.15.7:0/2902457144 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b40af980 con 0x7ff7bc0c7980 2023-11-28T18:47:11.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc0c7980 msgr2=0x7ff7bc0c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:11.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc0c7980 0x7ff7bc0c7e10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009d90 tx=0x7ff7b401f130 comp rx=0 tx=0).stop 2023-11-28T18:47:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- 172.21.15.7:0/2902457144 shutdown_connections 2023-11-28T18:47:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc0c7980 0x7ff7bc0c7e10 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/2902457144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc150b30 0x7ff7bc0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- 172.21.15.7:0/2902457144 >> 172.21.15.7:0/2902457144 conn(0x7ff7bc0662e0 msgr2=0x7ff7bc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- 172.21.15.7:0/2902457144 shutdown_connections 2023-11-28T18:47:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- 172.21.15.7:0/2902457144 wait complete. 2023-11-28T18:47:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 Processor -- start 2023-11-28T18:47:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- start start 2023-11-28T18:47:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc0c7980 0x7ff7bc156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc150b30 0x7ff7bc1f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc1575c0 0x7ff7bc1e1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc155b30 con 0x7ff7bc150b30 2023-11-28T18:47:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc1560f0 con 0x7ff7bc1575c0 2023-11-28T18:47:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7c0c17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc0c4720 con 0x7ff7bc0c7980 2023-11-28T18:47:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc150b30 0x7ff7bc1f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc150b30 0x7ff7bc1f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58612/0 (socket says 172.21.15.7:58612) 2023-11-28T18:47:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7b3fff640 1 -- 172.21.15.7:0/4017773209 learned_addr learned my addr 172.21.15.7:0/4017773209 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7ba59c640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc0c7980 0x7ff7bc156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7bad9d640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc1575c0 0x7ff7bc1e1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.501+0000 7ff7b3fff640 1 -- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc0c7980 msgr2=0x7ff7bc156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b3fff640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc0c7980 0x7ff7bc156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b3fff640 1 -- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc1575c0 msgr2=0x7ff7bc1e1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b3fff640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc1575c0 0x7ff7bc1e1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b3fff640 1 -- 172.21.15.7:0/4017773209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac009420 con 0x7ff7bc150b30 2023-11-28T18:47:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7bad9d640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc1575c0 0x7ff7bc1e1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:47:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b3fff640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc150b30 0x7ff7bc1f2c10 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400c450 tx=0x7ff7a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b2ffd640 1 -- 172.21.15.7:0/4017773209 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4005cf0 con 0x7ff7bc150b30 2023-11-28T18:47:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b2ffd640 1 -- 172.21.15.7:0/4017773209 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a4005e50 con 0x7ff7bc150b30 2023-11-28T18:47:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b4009a70 con 0x7ff7bc150b30 2023-11-28T18:47:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b2ffd640 1 -- 172.21.15.7:0/4017773209 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4007270 con 0x7ff7bc150b30 2023-11-28T18:47:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc14a520 con 0x7ff7bc150b30 2023-11-28T18:47:11.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7bc04e890 con 0x7ff7bc150b30 2023-11-28T18:47:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b2ffd640 1 -- 172.21.15.7:0/4017773209 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff7a40073d0 con 0x7ff7bc150b30 2023-11-28T18:47:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b2ffd640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff79c0668d0 0x7ff79c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7b2ffd640 1 -- 172.21.15.7:0/4017773209 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff7a4083d00 con 0x7ff7bc150b30 2023-11-28T18:47:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.505+0000 7ff7ba59c640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff79c0668d0 0x7ff79c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:11.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.509+0000 7ff7ba59c640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff79c0668d0 0x7ff79c068d90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009c80 tx=0x7ff7ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:11.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.509+0000 7ff7b2ffd640 1 -- 172.21.15.7:0/4017773209 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff7a4050b60 con 0x7ff7bc150b30 2023-11-28T18:47:11.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:47:11.629 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:47:11.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.701+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7bc0c2d80 con 0x7ff7bc150b30 2023-11-28T18:47:11.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.701+0000 7ff7b2ffd640 1 -- 172.21.15.7:0/4017773209 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7ff7a4017020 con 0x7ff7bc150b30 2023-11-28T18:47:11.704 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:11.704 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:11.706 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:11.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff79c0668d0 msgr2=0x7ff79c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:11.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff79c0668d0 0x7ff79c068d90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009c80 tx=0x7ff7ac008040 comp rx=0 tx=0).stop 2023-11-28T18:47:11.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc150b30 msgr2=0x7ff7bc1f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:11.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc150b30 0x7ff7bc1f2c10 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400c450 tx=0x7ff7a400c910 comp rx=0 tx=0).stop 2023-11-28T18:47:11.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 shutdown_connections 2023-11-28T18:47:11.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff79c0668d0 0x7ff79c068d90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7bc1575c0 0x7ff7bc1e1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc150b30 0x7ff7bc1f2c10 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 --2- 172.21.15.7:0/4017773209 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7bc0c7980 0x7ff7bc156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:11.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 >> 172.21.15.7:0/4017773209 conn(0x7ff7bc0662e0 msgr2=0x7ff7bc14fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:11.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 shutdown_connections 2023-11-28T18:47:11.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:11.705+0000 7ff7c0c17640 1 -- 172.21.15.7:0/4017773209 wait complete. 2023-11-28T18:47:11.726 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:47:13.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:13.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51b51b8640 1 Processor -- start 2023-11-28T18:47:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51b51b8640 1 -- start start 2023-11-28T18:47:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51b51b8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f51b00c68c0 0x7f51b00c2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51b51b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b0157e90 0x7f51b0158300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51b51b8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b015f2c0 con 0x7f51b00c6ca0 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51b51b8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51b0158840 con 0x7f51b00c68c0 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51b51b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b015e740 con 0x7f51b0157e90 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51ae59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f51b00c6ca0 0x7f51b0161f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60234/0 (socket says 172.21.15.7:60234) 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51ae59c640 1 -- 172.21.15.7:0/1641652965 learned_addr learned my addr 172.21.15.7:0/1641652965 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51aed9d640 1 --2- 172.21.15.7:0/1641652965 >> v2:172.21.15.193:3300/0 conn(0x7f51b00c68c0 0x7f51b00c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.881+0000 7f51af59e640 1 --2- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b0157e90 0x7f51b0158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51add9b640 1 -- 172.21.15.7:0/1641652965 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927562240 0 0) 0x7f51b015f2c0 con 0x7f51b00c6ca0 2023-11-28T18:47:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51add9b640 1 -- 172.21.15.7:0/1641652965 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5198003530 con 0x7f51b00c6ca0 2023-11-28T18:47:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51aed9d640 1 -- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b0157e90 msgr2=0x7f51b0158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51aed9d640 1 --2- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b0157e90 0x7f51b0158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51aed9d640 1 -- 172.21.15.7:0/1641652965 >> v1:172.21.15.7:6789/0 conn(0x7f51b00c6ca0 legacy=0x7f51b0161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51aed9d640 1 -- 172.21.15.7:0/1641652965 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b015ea20 con 0x7f51b00c68c0 2023-11-28T18:47:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51af59e640 1 --2- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b0157e90 0x7f51b0158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:47:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51aed9d640 1 --2- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c68c0 0x7f51b00c2260 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f51a00097c0 tx=0x7f51a001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4382987ed7ca1be9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51add9b640 1 -- 172.21.15.7:0/1641652965 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a00ab650 con 0x7f51b00c68c0 2023-11-28T18:47:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51add9b640 1 -- 172.21.15.7:0/1641652965 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a00ab7b0 con 0x7f51b00c68c0 2023-11-28T18:47:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.885+0000 7f51add9b640 1 -- 172.21.15.7:0/1641652965 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a00abb40 con 0x7f51b00c68c0 2023-11-28T18:47:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c68c0 msgr2=0x7f51b00c2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 --2- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c68c0 0x7f51b00c2260 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f51a00097c0 tx=0x7f51a001c450 comp rx=0 tx=0).stop 2023-11-28T18:47:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- 172.21.15.7:0/1641652965 shutdown_connections 2023-11-28T18:47:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 --2- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b0157e90 0x7f51b0158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 --2- 172.21.15.7:0/1641652965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c68c0 0x7f51b00c2260 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:13.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- 172.21.15.7:0/1641652965 >> 172.21.15.7:0/1641652965 conn(0x7f51b00662e0 msgr2=0x7f51b00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:13.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- 172.21.15.7:0/1641652965 shutdown_connections 2023-11-28T18:47:13.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- 172.21.15.7:0/1641652965 wait complete. 2023-11-28T18:47:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 Processor -- start 2023-11-28T18:47:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- start start 2023-11-28T18:47:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b00c68c0 0x7f51b0147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c6ca0 0x7f51b01564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51b0157e90 0x7f51b01f9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b0146d80 con 0x7f51b0157e90 2023-11-28T18:47:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b00c9770 con 0x7f51b00c6ca0 2023-11-28T18:47:13.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51b51b8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b00ca2b0 con 0x7f51b00c68c0 2023-11-28T18:47:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51af59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51b0157e90 0x7f51b01f9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51af59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51b0157e90 0x7f51b01f9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39668/0 (socket says 172.21.15.7:39668) 2023-11-28T18:47:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.889+0000 7f51af59e640 1 -- 172.21.15.7:0/3807982419 learned_addr learned my addr 172.21.15.7:0/3807982419 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51aed9d640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b00c68c0 0x7f51b0147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51ae59c640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c6ca0 0x7f51b01564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51af59e640 1 -- 172.21.15.7:0/3807982419 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b00c68c0 msgr2=0x7f51b0147210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51af59e640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b00c68c0 0x7f51b0147210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51af59e640 1 -- 172.21.15.7:0/3807982419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c6ca0 msgr2=0x7f51b01564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51af59e640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c6ca0 0x7f51b01564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51af59e640 1 -- 172.21.15.7:0/3807982419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a8009a70 con 0x7f51b0157e90 2023-11-28T18:47:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51af59e640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51b0157e90 0x7f51b01f9a90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f51a800efc0 tx=0x7f51a8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51977fe640 1 -- 172.21.15.7:0/3807982419 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a8009d70 con 0x7f51b0157e90 2023-11-28T18:47:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51977fe640 1 -- 172.21.15.7:0/3807982419 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a800f040 con 0x7f51b0157e90 2023-11-28T18:47:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a0009420 con 0x7f51b0157e90 2023-11-28T18:47:13.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51977fe640 1 -- 172.21.15.7:0/3807982419 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a80039a0 con 0x7f51b0157e90 2023-11-28T18:47:13.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51b00c9cf0 con 0x7f51b0157e90 2023-11-28T18:47:13.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51b0146ef0 con 0x7f51b0157e90 2023-11-28T18:47:13.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51977fe640 1 -- 172.21.15.7:0/3807982419 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f51a80144a0 con 0x7f51b0157e90 2023-11-28T18:47:13.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.893+0000 7f51977fe640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f519c066900 0x7f519c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:13.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.897+0000 7f51977fe640 1 -- 172.21.15.7:0/3807982419 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f51a80853c0 con 0x7f51b0157e90 2023-11-28T18:47:13.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.897+0000 7f51aed9d640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f519c066900 0x7f519c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:13.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.897+0000 7f51aed9d640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f519c066900 0x7f519c068dc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f51a00af000 tx=0x7f51a00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:13.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:13.901+0000 7f51977fe640 1 -- 172.21.15.7:0/3807982419 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f51a8052230 con 0x7f51b0157e90 2023-11-28T18:47:14.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.089+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51b004e890 con 0x7f51b0157e90 2023-11-28T18:47:14.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.089+0000 7f51977fe640 1 -- 172.21.15.7:0/3807982419 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f51a80596d0 con 0x7f51b0157e90 2023-11-28T18:47:14.091 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:14.091 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:14.091 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.089+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f519c066900 msgr2=0x7f519c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.089+0000 7f51b51b8640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f519c066900 0x7f519c068dc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f51a00af000 tx=0x7f51a00a7000 comp rx=0 tx=0).stop 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.089+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51b0157e90 msgr2=0x7f51b01f9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.089+0000 7f51b51b8640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51b0157e90 0x7f51b01f9a90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f51a800efc0 tx=0x7f51a8009f90 comp rx=0 tx=0).stop 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.093+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 shutdown_connections 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.093+0000 7f51b51b8640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f519c066900 0x7f519c068dc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.093+0000 7f51b51b8640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51b0157e90 0x7f51b01f9a90 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.093+0000 7f51b51b8640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51b00c6ca0 0x7f51b01564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.093+0000 7f51b51b8640 1 --2- 172.21.15.7:0/3807982419 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f51b00c68c0 0x7f51b0147210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.093+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 >> 172.21.15.7:0/3807982419 conn(0x7f51b00662e0 msgr2=0x7f51b006d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.093+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 shutdown_connections 2023-11-28T18:47:14.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:14.093+0000 7f51b51b8640 1 -- 172.21.15.7:0/3807982419 wait complete. 2023-11-28T18:47:14.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~115s 2023-11-28T18:47:14.114 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:16.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:16.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f117d3f8640 1 Processor -- start 2023-11-28T18:47:16.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f117d3f8640 1 -- start start 2023-11-28T18:47:16.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f117d3f8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f117815bd00 0x7f1178157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:16.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f117d3f8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117815c110 con 0x7f1178151cd0 2023-11-28T18:47:16.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f117d3f8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1178071e60 con 0x7f117806e560 2023-11-28T18:47:16.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f117d3f8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1178070930 con 0x7f117815bd00 2023-11-28T18:47:16.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f1176ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1178151cd0 0x7f11780c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60240/0 (socket says 172.21.15.7:60240) 2023-11-28T18:47:16.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f1176ffd640 1 -- 172.21.15.7:0/3570358836 learned_addr learned my addr 172.21.15.7:0/3570358836 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:16.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f11777fe640 1 --2- 172.21.15.7:0/3570358836 >> v2:172.21.15.193:3300/0 conn(0x7f117815bd00 0x7f1178157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:16.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f11767fc640 1 -- 172.21.15.7:0/3570358836 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756825775 0 0) 0x7f117815c110 con 0x7f1178151cd0 2023-11-28T18:47:16.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f11767fc640 1 -- 172.21.15.7:0/3570358836 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1164003530 con 0x7f1178151cd0 2023-11-28T18:47:16.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f11777fe640 1 -- 172.21.15.7:0/3570358836 >> v1:172.21.15.193:6789/0 conn(0x7f117806e560 legacy=0x7f1178160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:16.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f11777fe640 1 -- 172.21.15.7:0/3570358836 >> v1:172.21.15.7:6789/0 conn(0x7f1178151cd0 legacy=0x7f11780c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:16.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.273+0000 7f11777fe640 1 -- 172.21.15.7:0/3570358836 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1178070bf0 con 0x7f117815bd00 2023-11-28T18:47:16.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f11777fe640 1 --2- 172.21.15.7:0/3570358836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 0x7f1178157b90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1170009d90 tx=0x7f117001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea288ec870cda1c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:16.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f11767fc640 1 -- 172.21.15.7:0/3570358836 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700a5900 con 0x7f117815bd00 2023-11-28T18:47:16.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f11767fc640 1 -- 172.21.15.7:0/3570358836 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11700a5a60 con 0x7f117815bd00 2023-11-28T18:47:16.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f11767fc640 1 -- 172.21.15.7:0/3570358836 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700a5df0 con 0x7f117815bd00 2023-11-28T18:47:16.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f117d3f8640 1 -- 172.21.15.7:0/3570358836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 msgr2=0x7f1178157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:16.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f117d3f8640 1 --2- 172.21.15.7:0/3570358836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 0x7f1178157b90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1170009d90 tx=0x7f117001c950 comp rx=0 tx=0).stop 2023-11-28T18:47:16.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f117d3f8640 1 -- 172.21.15.7:0/3570358836 shutdown_connections 2023-11-28T18:47:16.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f117d3f8640 1 --2- 172.21.15.7:0/3570358836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 0x7f1178157b90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:16.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f117d3f8640 1 -- 172.21.15.7:0/3570358836 >> 172.21.15.7:0/3570358836 conn(0x7f11780662e0 msgr2=0x7f11780666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:16.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f117d3f8640 1 -- 172.21.15.7:0/3570358836 shutdown_connections 2023-11-28T18:47:16.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.277+0000 7f117d3f8640 1 -- 172.21.15.7:0/3570358836 wait complete. 2023-11-28T18:47:16.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 Processor -- start 2023-11-28T18:47:16.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 -- start start 2023-11-28T18:47:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f117806e560 0x7f1178153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178151cd0 0x7f11781f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 0x7f11781f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11781f1850 con 0x7f1178151cd0 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1178152b80 con 0x7f117815bd00 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11781550f0 con 0x7f117806e560 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178151cd0 0x7f11781f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178151cd0 0x7f11781f1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39682/0 (socket says 172.21.15.7:39682) 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 -- 172.21.15.7:0/3302784000 learned_addr learned my addr 172.21.15.7:0/3302784000 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f1176ffd640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f117806e560 0x7f1178153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f11777fe640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 0x7f11781f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 -- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f117806e560 msgr2=0x7f1178153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f117806e560 0x7f1178153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 -- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 msgr2=0x7f11781f1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 0x7f11781f1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 -- 172.21.15.7:0/3302784000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1170009a70 con 0x7f1178151cd0 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f11777fe640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 0x7f11781f1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f1176ffd640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f117806e560 0x7f1178153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116ffff640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178151cd0 0x7f11781f1d00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f116000c500 tx=0x7f116000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116f7fe640 1 -- 172.21.15.7:0/3302784000 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160005220 con 0x7f1178151cd0 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116f7fe640 1 -- 172.21.15.7:0/3302784000 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1160005380 con 0x7f1178151cd0 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1178152df0 con 0x7f1178151cd0 2023-11-28T18:47:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f116f7fe640 1 -- 172.21.15.7:0/3302784000 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160007270 con 0x7f1178151cd0 2023-11-28T18:47:16.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.281+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11781533b0 con 0x7f1178151cd0 2023-11-28T18:47:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.285+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1178150540 con 0x7f1178151cd0 2023-11-28T18:47:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.285+0000 7f116f7fe640 1 -- 172.21.15.7:0/3302784000 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f11600073d0 con 0x7f1178151cd0 2023-11-28T18:47:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.285+0000 7f116f7fe640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1158066900 0x7f1158068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.285+0000 7f116f7fe640 1 -- 172.21.15.7:0/3302784000 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f11600841e0 con 0x7f1178151cd0 2023-11-28T18:47:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.285+0000 7f1176ffd640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1158066900 0x7f1158068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.285+0000 7f1176ffd640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1158066900 0x7f1158068dc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f11680011f0 tx=0x7f1168009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:16.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.289+0000 7f116f7fe640 1 -- 172.21.15.7:0/3302784000 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1160050f10 con 0x7f1178151cd0 2023-11-28T18:47:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.477+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f117804e890 con 0x7f1178151cd0 2023-11-28T18:47:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.477+0000 7f116f7fe640 1 -- 172.21.15.7:0/3302784000 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f1160054530 con 0x7f1178151cd0 2023-11-28T18:47:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:16.479 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1158066900 msgr2=0x7f1158068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1158066900 0x7f1158068dc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f11680011f0 tx=0x7f1168009040 comp rx=0 tx=0).stop 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178151cd0 msgr2=0x7f11781f1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178151cd0 0x7f11781f1d00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f116000c500 tx=0x7f116000c9c0 comp rx=0 tx=0).stop 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 shutdown_connections 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1158066900 0x7f1158068dc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f117815bd00 0x7f11781f1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178151cd0 0x7f11781f1d00 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 --2- 172.21.15.7:0/3302784000 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f117806e560 0x7f1178153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 >> 172.21.15.7:0/3302784000 conn(0x7f11780662e0 msgr2=0x7f11780cad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 shutdown_connections 2023-11-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:16.481+0000 7f117d3f8640 1 -- 172.21.15.7:0/3302784000 wait complete. 2023-11-28T18:47:16.501 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:18.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:18.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7faa0051d640 1 Processor -- start 2023-11-28T18:47:18.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7faa0051d640 1 -- start start 2023-11-28T18:47:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7faa0051d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f8150b30 0x7fa9f80c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7faa0051d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa9f81575c0 0x7fa9f80c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7faa0051d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f80c6140 con 0x7fa9f80c4020 2023-11-28T18:47:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7faa0051d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f80c72b0 con 0x7fa9f81575c0 2023-11-28T18:47:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7faa0051d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8070c00 con 0x7fa9f8150b30 2023-11-28T18:47:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7fa9feaba640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa9f80c4020 0x7fa9f80c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60254/0 (socket says 172.21.15.7:60254) 2023-11-28T18:47:18.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7fa9feaba640 1 -- 172.21.15.7:0/1929921777 learned_addr learned my addr 172.21.15.7:0/1929921777 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:18.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.657+0000 7fa9fdab8640 1 --2- 172.21.15.7:0/1929921777 >> v2:172.21.15.7:3300/0 conn(0x7fa9f81575c0 0x7fa9f80c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:18.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fe2b9640 1 --2- 172.21.15.7:0/1929921777 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f8150b30 0x7fa9f80c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fdab8640 1 -- 172.21.15.7:0/1929921777 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f8150b30 msgr2=0x7fa9f80c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fdab8640 1 --2- 172.21.15.7:0/1929921777 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f8150b30 0x7fa9f80c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fdab8640 1 -- 172.21.15.7:0/1929921777 >> v1:172.21.15.7:6789/0 conn(0x7fa9f80c4020 legacy=0x7fa9f80c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fdab8640 1 -- 172.21.15.7:0/1929921777 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80c6370 con 0x7fa9f81575c0 2023-11-28T18:47:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fd2b7640 1 -- 172.21.15.7:0/1929921777 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079729432 0 0) 0x7fa9f80c6140 con 0x7fa9f80c4020 2023-11-28T18:47:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fdab8640 1 --2- 172.21.15.7:0/1929921777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f81575c0 0x7fa9f80c3ae0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80097c0 tx=0x7fa9e801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f808195f575704a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fd2b7640 1 -- 172.21.15.7:0/1929921777 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80a8970 con 0x7fa9f81575c0 2023-11-28T18:47:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fd2b7640 1 -- 172.21.15.7:0/1929921777 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e80a8ad0 con 0x7fa9f81575c0 2023-11-28T18:47:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7fa9fd2b7640 1 -- 172.21.15.7:0/1929921777 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80a8e60 con 0x7fa9f81575c0 2023-11-28T18:47:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7faa0051d640 1 -- 172.21.15.7:0/1929921777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f81575c0 msgr2=0x7fa9f80c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.661+0000 7faa0051d640 1 --2- 172.21.15.7:0/1929921777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f81575c0 0x7fa9f80c3ae0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80097c0 tx=0x7fa9e801ec70 comp rx=0 tx=0).stop 2023-11-28T18:47:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 -- 172.21.15.7:0/1929921777 shutdown_connections 2023-11-28T18:47:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 --2- 172.21.15.7:0/1929921777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f81575c0 0x7fa9f80c3ae0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 --2- 172.21.15.7:0/1929921777 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f8150b30 0x7fa9f80c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 -- 172.21.15.7:0/1929921777 >> 172.21.15.7:0/1929921777 conn(0x7fa9f80662e0 msgr2=0x7fa9f80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 -- 172.21.15.7:0/1929921777 shutdown_connections 2023-11-28T18:47:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 -- 172.21.15.7:0/1929921777 wait complete. 2023-11-28T18:47:18.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 Processor -- start 2023-11-28T18:47:18.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 -- start start 2023-11-28T18:47:18.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f80c4020 0x7fa9f8154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:18.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8150b30 0x7fa9f814c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:18.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9f81575c0 0x7fa9f8153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:18.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f81dcc20 con 0x7fa9f8150b30 2023-11-28T18:47:18.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f814b150 con 0x7fa9f81575c0 2023-11-28T18:47:18.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7faa0051d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f81ee250 con 0x7fa9f80c4020 2023-11-28T18:47:18.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8150b30 0x7fa9f814c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8150b30 0x7fa9f814c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39694/0 (socket says 172.21.15.7:39694) 2023-11-28T18:47:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 -- 172.21.15.7:0/1310148472 learned_addr learned my addr 172.21.15.7:0/1310148472 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9feaba640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9f81575c0 0x7fa9f8153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fe2b9640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f80c4020 0x7fa9f8154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 -- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f80c4020 msgr2=0x7fa9f8154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f80c4020 0x7fa9f8154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 -- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9f81575c0 msgr2=0x7fa9f8153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9f81575c0 0x7fa9f8153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 -- 172.21.15.7:0/1310148472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8009420 con 0x7fa9f8150b30 2023-11-28T18:47:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9feaba640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9f81575c0 0x7fa9f8153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:47:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9fdab8640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8150b30 0x7fa9f814c400 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa9e801c8a0 tx=0x7fa9e801d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9eeffd640 1 -- 172.21.15.7:0/1310148472 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80a8970 con 0x7fa9f8150b30 2023-11-28T18:47:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9eeffd640 1 -- 172.21.15.7:0/1310148472 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e8006e20 con 0x7fa9f8150b30 2023-11-28T18:47:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.665+0000 7fa9eeffd640 1 -- 172.21.15.7:0/1310148472 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8004230 con 0x7fa9f8150b30 2023-11-28T18:47:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.669+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f81dce70 con 0x7fa9f8150b30 2023-11-28T18:47:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.669+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f814ade0 con 0x7fa9f8150b30 2023-11-28T18:47:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.669+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f804e890 con 0x7fa9f8150b30 2023-11-28T18:47:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.669+0000 7fa9eeffd640 1 -- 172.21.15.7:0/1310148472 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa9e8004390 con 0x7fa9f8150b30 2023-11-28T18:47:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.669+0000 7fa9eeffd640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9dc066900 0x7fa9dc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.669+0000 7fa9eeffd640 1 -- 172.21.15.7:0/1310148472 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fa9e81220d0 con 0x7fa9f8150b30 2023-11-28T18:47:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.669+0000 7fa9fe2b9640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9dc066900 0x7fa9dc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.669+0000 7fa9fe2b9640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9dc066900 0x7fa9dc068dc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400b9d0 tx=0x7fa9f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.673+0000 7fa9eeffd640 1 -- 172.21.15.7:0/1310148472 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa9e80eee80 con 0x7fa9f8150b30 2023-11-28T18:47:18.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.865+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9f80c2d80 con 0x7fa9f8150b30 2023-11-28T18:47:18.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.865+0000 7fa9eeffd640 1 -- 172.21.15.7:0/1310148472 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fa9e80f24a0 con 0x7fa9f8150b30 2023-11-28T18:47:18.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:18.868 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:18.868 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9dc066900 msgr2=0x7fa9dc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9dc066900 0x7fa9dc068dc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400b9d0 tx=0x7fa9f4008040 comp rx=0 tx=0).stop 2023-11-28T18:47:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8150b30 msgr2=0x7fa9f814c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8150b30 0x7fa9f814c400 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa9e801c8a0 tx=0x7fa9e801d4b0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 shutdown_connections 2023-11-28T18:47:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa9dc066900 0x7fa9dc068dc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9f81575c0 0x7fa9f8153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8150b30 0x7fa9f814c400 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 --2- 172.21.15.7:0/1310148472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa9f80c4020 0x7fa9f8154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 >> 172.21.15.7:0/1310148472 conn(0x7fa9f80662e0 msgr2=0x7fa9f806d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 shutdown_connections 2023-11-28T18:47:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:18.869+0000 7faa0051d640 1 -- 172.21.15.7:0/1310148472 wait complete. 2023-11-28T18:47:18.890 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:20.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~121s 2023-11-28T18:47:20.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:21.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3cb2a640 1 Processor -- start 2023-11-28T18:47:21.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3cb2a640 1 -- start start 2023-11-28T18:47:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3cb2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb38151cd0 0x7feb3815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3cb2a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb3806e560 0x7feb38157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3cb2a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7feb380701f0 0x7feb380c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3cb2a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb38071e60 con 0x7feb3806e560 2023-11-28T18:47:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3cb2a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb38070930 con 0x7feb380701f0 2023-11-28T18:47:21.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3cb2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb38070660 con 0x7feb38151cd0 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb2ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb3806e560 0x7feb38157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb2ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb3806e560 0x7feb38157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39708/0 (socket says 172.21.15.7:39708) 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb2ffff640 1 -- 172.21.15.7:0/3677508161 learned_addr learned my addr 172.21.15.7:0/3677508161 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb36d9d640 1 --2- 172.21.15.7:0/3677508161 >> v2:172.21.15.193:3300/0 conn(0x7feb380701f0 0x7feb380c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb3659c640 1 --2- 172.21.15.7:0/3677508161 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb38151cd0 0x7feb3815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb2ffff640 1 -- 172.21.15.7:0/3677508161 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb38151cd0 msgr2=0x7feb3815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb2ffff640 1 --2- 172.21.15.7:0/3677508161 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb38151cd0 0x7feb3815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb2ffff640 1 -- 172.21.15.7:0/3677508161 >> v2:172.21.15.193:3300/0 conn(0x7feb380701f0 msgr2=0x7feb380c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb2ffff640 1 --2- 172.21.15.7:0/3677508161 >> v2:172.21.15.193:3300/0 conn(0x7feb380701f0 0x7feb380c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb2ffff640 1 -- 172.21.15.7:0/3677508161 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38070bf0 con 0x7feb3806e560 2023-11-28T18:47:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.045+0000 7feb36d9d640 1 --2- 172.21.15.7:0/3677508161 >> v2:172.21.15.193:3300/0 conn(0x7feb380701f0 0x7feb380c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:21.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb2ffff640 1 --2- 172.21.15.7:0/3677508161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3806e560 0x7feb38157b90 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7feb380c95f0 tx=0x7feb2001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a2257a0e101369 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:21.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb35d9b640 1 -- 172.21.15.7:0/3677508161 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb200ab600 con 0x7feb3806e560 2023-11-28T18:47:21.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb35d9b640 1 -- 172.21.15.7:0/3677508161 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb200a7050 con 0x7feb3806e560 2023-11-28T18:47:21.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb35d9b640 1 -- 172.21.15.7:0/3677508161 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb200ab980 con 0x7feb3806e560 2023-11-28T18:47:21.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3677508161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3806e560 msgr2=0x7feb38157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:21.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3677508161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3806e560 0x7feb38157b90 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7feb380c95f0 tx=0x7feb2001c470 comp rx=0 tx=0).stop 2023-11-28T18:47:21.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3677508161 shutdown_connections 2023-11-28T18:47:21.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3677508161 >> v2:172.21.15.193:3300/0 conn(0x7feb380701f0 0x7feb380c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3677508161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3806e560 0x7feb38157b90 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3677508161 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb38151cd0 0x7feb3815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3677508161 >> 172.21.15.7:0/3677508161 conn(0x7feb380662e0 msgr2=0x7feb380666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:21.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3677508161 shutdown_connections 2023-11-28T18:47:21.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.049+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3677508161 wait complete. 2023-11-28T18:47:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 Processor -- start 2023-11-28T18:47:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 -- start start 2023-11-28T18:47:21.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb3806e560 0x7feb380c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:21.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb380701f0 0x7feb38150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb38151cd0 0x7feb38149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb381f3530 con 0x7feb38151cd0 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb381f2c20 con 0x7feb3806e560 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb38150600 con 0x7feb380701f0 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb38151cd0 0x7feb38149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb38151cd0 0x7feb38149020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39718/0 (socket says 172.21.15.7:39718) 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 -- 172.21.15.7:0/3865481875 learned_addr learned my addr 172.21.15.7:0/3865481875 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb2ffff640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb380701f0 0x7feb38150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3659c640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb3806e560 0x7feb380c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:21.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 -- 172.21.15.7:0/3865481875 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb380701f0 msgr2=0x7feb38150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb380701f0 0x7feb38150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 -- 172.21.15.7:0/3865481875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb3806e560 msgr2=0x7feb380c3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb3806e560 0x7feb380c3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 -- 172.21.15.7:0/3865481875 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200094d0 con 0x7feb38151cd0 2023-11-28T18:47:21.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb36d9d640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb38151cd0 0x7feb38149020 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7feb30007e10 tx=0x7feb3000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:21.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb2effd640 1 -- 172.21.15.7:0/3865481875 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30003ba0 con 0x7feb38151cd0 2023-11-28T18:47:21.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb2effd640 1 -- 172.21.15.7:0/3865481875 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3000f040 con 0x7feb38151cd0 2023-11-28T18:47:21.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb2effd640 1 -- 172.21.15.7:0/3865481875 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3000de80 con 0x7feb38151cd0 2023-11-28T18:47:21.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.053+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb381508e0 con 0x7feb38151cd0 2023-11-28T18:47:21.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.057+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb381f31c0 con 0x7feb38151cd0 2023-11-28T18:47:21.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.057+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb381f36a0 con 0x7feb38151cd0 2023-11-28T18:47:21.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.057+0000 7feb2effd640 1 -- 172.21.15.7:0/3865481875 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7feb30004030 con 0x7feb38151cd0 2023-11-28T18:47:21.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.057+0000 7feb2effd640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb18066900 0x7feb18068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:21.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.057+0000 7feb2effd640 1 -- 172.21.15.7:0/3865481875 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7feb30084ae0 con 0x7feb38151cd0 2023-11-28T18:47:21.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.057+0000 7feb3659c640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb18066900 0x7feb18068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:21.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.057+0000 7feb3659c640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb18066900 0x7feb18068dc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7feb2800b930 tx=0x7feb28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.061+0000 7feb2effd640 1 -- 172.21.15.7:0/3865481875 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feb30051940 con 0x7feb38151cd0 2023-11-28T18:47:21.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.253+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb3804e890 con 0x7feb38151cd0 2023-11-28T18:47:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.253+0000 7feb2effd640 1 -- 172.21.15.7:0/3865481875 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7feb30054f60 con 0x7feb38151cd0 2023-11-28T18:47:21.256 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:21.256 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:21.257 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:21.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb18066900 msgr2=0x7feb18068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:21.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb18066900 0x7feb18068dc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7feb2800b930 tx=0x7feb28008040 comp rx=0 tx=0).stop 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb38151cd0 msgr2=0x7feb38149020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb38151cd0 0x7feb38149020 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7feb30007e10 tx=0x7feb3000d4a0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 shutdown_connections 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7feb18066900 0x7feb18068dc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb38151cd0 0x7feb38149020 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7feb380701f0 0x7feb38150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 --2- 172.21.15.7:0/3865481875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb3806e560 0x7feb380c3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 >> 172.21.15.7:0/3865481875 conn(0x7feb380662e0 msgr2=0x7feb380c5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 shutdown_connections 2023-11-28T18:47:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:21.257+0000 7feb3cb2a640 1 -- 172.21.15.7:0/3865481875 wait complete. 2023-11-28T18:47:21.278 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:23.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0fa15f5640 1 Processor -- start 2023-11-28T18:47:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0fa15f5640 1 -- start start 2023-11-28T18:47:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0fa15f5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f9c072950 0x7f0f9c074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0fa15f5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0f9c06c920 0x7f0f9c06cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0fa15f5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c06d310 con 0x7f0f9c073070 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0fa15f5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c0c8490 con 0x7f0f9c072950 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0fa15f5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c0c81f0 con 0x7f0f9c06c920 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0f9affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f9c072950 0x7f0f9c074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0f9affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f9c072950 0x7f0f9c074030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39730/0 (socket says 172.21.15.7:39730) 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0f9affd640 1 -- 172.21.15.7:0/1745592890 learned_addr learned my addr 172.21.15.7:0/1745592890 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0f9b7fe640 1 --2- 172.21.15.7:0/1745592890 >> v2:172.21.15.193:3300/0 conn(0x7f0f9c06c920 0x7f0f9c06cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0f9affd640 1 -- 172.21.15.7:0/1745592890 >> v2:172.21.15.193:3300/0 conn(0x7f0f9c06c920 msgr2=0x7f0f9c06cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0f9affd640 1 --2- 172.21.15.7:0/1745592890 >> v2:172.21.15.193:3300/0 conn(0x7f0f9c06c920 0x7f0f9c06cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0f9affd640 1 -- 172.21.15.7:0/1745592890 >> v1:172.21.15.193:6789/0 conn(0x7f0f9c073070 legacy=0x7f0f9c0c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.433+0000 7f0f9affd640 1 -- 172.21.15.7:0/1745592890 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c15ab00 con 0x7f0f9c072950 2023-11-28T18:47:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0f9affd640 1 --2- 172.21.15.7:0/1745592890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c072950 0x7f0f9c074030 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c15e480 tx=0x7f0f8c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2d94fe0e6c5f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0f9a7fc640 1 -- 172.21.15.7:0/1745592890 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0ab600 con 0x7f0f9c072950 2023-11-28T18:47:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0f9a7fc640 1 -- 172.21.15.7:0/1745592890 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8c0a7050 con 0x7f0f9c072950 2023-11-28T18:47:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0f9a7fc640 1 -- 172.21.15.7:0/1745592890 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0ab980 con 0x7f0f9c072950 2023-11-28T18:47:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1745592890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c072950 msgr2=0x7f0f9c074030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1745592890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c072950 0x7f0f9c074030 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c15e480 tx=0x7f0f8c01c3a0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1745592890 shutdown_connections 2023-11-28T18:47:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1745592890 >> v2:172.21.15.193:3300/0 conn(0x7f0f9c06c920 0x7f0f9c06cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1745592890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c072950 0x7f0f9c074030 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1745592890 >> 172.21.15.7:0/1745592890 conn(0x7f0f9c0cbf80 msgr2=0x7f0f9c0cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1745592890 shutdown_connections 2023-11-28T18:47:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.437+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1745592890 wait complete. 2023-11-28T18:47:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 Processor -- start 2023-11-28T18:47:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 -- start start 2023-11-28T18:47:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0f9c06c920 0x7f0f9c149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f9c072950 0x7f0f9c1f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c073070 0x7f0f9c1f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c141c70 con 0x7f0f9c073070 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c141f50 con 0x7f0f9c072950 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c142230 con 0x7f0f9c06c920 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c073070 0x7f0f9c1f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c073070 0x7f0f9c1f6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39746/0 (socket says 172.21.15.7:39746) 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 -- 172.21.15.7:0/1676265478 learned_addr learned my addr 172.21.15.7:0/1676265478 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9affd640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0f9c06c920 0x7f0f9c149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f93fff640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f9c072950 0x7f0f9c1f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 -- 172.21.15.7:0/1676265478 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0f9c06c920 msgr2=0x7f0f9c149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0f9c06c920 0x7f0f9c149900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 -- 172.21.15.7:0/1676265478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f9c072950 msgr2=0x7f0f9c1f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f9c072950 0x7f0f9c1f6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 -- 172.21.15.7:0/1676265478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c0094d0 con 0x7f0f9c073070 2023-11-28T18:47:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f9b7fe640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c073070 0x7f0f9c1f6bb0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f0f94007e40 tx=0x7f0f9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f937fe640 1 -- 172.21.15.7:0/1676265478 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94014620 con 0x7f0f9c073070 2023-11-28T18:47:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f937fe640 1 -- 172.21.15.7:0/1676265478 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9400ee90 con 0x7f0f9c073070 2023-11-28T18:47:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0f937fe640 1 -- 172.21.15.7:0/1676265478 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f940149e0 con 0x7f0f9c073070 2023-11-28T18:47:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c142510 con 0x7f0f9c073070 2023-11-28T18:47:23.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.441+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c1427f0 con 0x7f0f9c073070 2023-11-28T18:47:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.445+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c1f6390 con 0x7f0f9c073070 2023-11-28T18:47:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.445+0000 7f0f937fe640 1 -- 172.21.15.7:0/1676265478 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0f94014b40 con 0x7f0f9c073070 2023-11-28T18:47:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.445+0000 7f0f937fe640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0f7c0668d0 0x7f0f7c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.445+0000 7f0f937fe640 1 -- 172.21.15.7:0/1676265478 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f0f940851a0 con 0x7f0f9c073070 2023-11-28T18:47:23.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.445+0000 7f0f9affd640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0f7c0668d0 0x7f0f7c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:23.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.445+0000 7f0f9affd640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0f7c0668d0 0x7f0f7c068d90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0ad000 tx=0x7f0f8c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:23.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.449+0000 7f0f937fe640 1 -- 172.21.15.7:0/1676265478 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0f9c1f6390 con 0x7f0f9c073070 2023-11-28T18:47:23.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f9c1f6390 con 0x7f0f9c073070 2023-11-28T18:47:23.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0f937fe640 1 -- 172.21.15.7:0/1676265478 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f0f9c1f6390 con 0x7f0f9c073070 2023-11-28T18:47:23.642 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:23.642 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:23.642 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0f7c0668d0 msgr2=0x7f0f7c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0f7c0668d0 0x7f0f7c068d90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0ad000 tx=0x7f0f8c0a6000 comp rx=0 tx=0).stop 2023-11-28T18:47:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c073070 msgr2=0x7f0f9c1f6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c073070 0x7f0f9c1f6bb0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f0f94007e40 tx=0x7f0f9400e4b0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 shutdown_connections 2023-11-28T18:47:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0f7c0668d0 0x7f0f7c068d90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f9c073070 0x7f0f9c1f6bb0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f9c072950 0x7f0f9c1f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 --2- 172.21.15.7:0/1676265478 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0f9c06c920 0x7f0f9c149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:23.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 >> 172.21.15.7:0/1676265478 conn(0x7f0f9c0cbf80 msgr2=0x7f0f9c0c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:23.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 shutdown_connections 2023-11-28T18:47:23.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:23.641+0000 7f0fa15f5640 1 -- 172.21.15.7:0/1676265478 wait complete. 2023-11-28T18:47:23.665 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:47:25.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:25.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff633009640 1 Processor -- start 2023-11-28T18:47:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff633009640 1 -- start start 2023-11-28T18:47:25.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff633009640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff62c150b30 0x7ff62c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:25.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff633009640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c1575c0 0x7ff62c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:25.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff633009640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62c0c6140 con 0x7ff62c0c4020 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff633009640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62c0c72b0 con 0x7ff62c150b30 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff633009640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c070c00 con 0x7ff62c1575c0 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff630da5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff62c150b30 0x7ff62c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff630da5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff62c150b30 0x7ff62c0c24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51886/0 (socket says 172.21.15.7:51886) 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff630da5640 1 -- 172.21.15.7:0/2237659305 learned_addr learned my addr 172.21.15.7:0/2237659305 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff62bfff640 1 --2- 172.21.15.7:0/2237659305 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c1575c0 0x7ff62c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff630da5640 1 -- 172.21.15.7:0/2237659305 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c1575c0 msgr2=0x7ff62c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff630da5640 1 --2- 172.21.15.7:0/2237659305 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c1575c0 0x7ff62c0c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff630da5640 1 -- 172.21.15.7:0/2237659305 >> v1:172.21.15.193:6789/0 conn(0x7ff62c0c4020 legacy=0x7ff62c0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:25.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.821+0000 7ff630da5640 1 -- 172.21.15.7:0/2237659305 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c0c6370 con 0x7ff62c150b30 2023-11-28T18:47:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff630da5640 1 --2- 172.21.15.7:0/2237659305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c150b30 0x7ff62c0c24f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff620009870 tx=0x7ff62001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62c7b212be1285d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:25.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff62b7fe640 1 -- 172.21.15.7:0/2237659305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200a89b0 con 0x7ff62c150b30 2023-11-28T18:47:25.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff62b7fe640 1 -- 172.21.15.7:0/2237659305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6200a8b10 con 0x7ff62c150b30 2023-11-28T18:47:25.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff62b7fe640 1 -- 172.21.15.7:0/2237659305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200a8ea0 con 0x7ff62c150b30 2023-11-28T18:47:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff633009640 1 -- 172.21.15.7:0/2237659305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c150b30 msgr2=0x7ff62c0c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff633009640 1 --2- 172.21.15.7:0/2237659305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c150b30 0x7ff62c0c24f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff620009870 tx=0x7ff62001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:47:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff633009640 1 -- 172.21.15.7:0/2237659305 shutdown_connections 2023-11-28T18:47:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff633009640 1 --2- 172.21.15.7:0/2237659305 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c1575c0 0x7ff62c0c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff633009640 1 --2- 172.21.15.7:0/2237659305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c150b30 0x7ff62c0c24f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff633009640 1 -- 172.21.15.7:0/2237659305 >> 172.21.15.7:0/2237659305 conn(0x7ff62c0662e0 msgr2=0x7ff62c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff633009640 1 -- 172.21.15.7:0/2237659305 shutdown_connections 2023-11-28T18:47:25.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.825+0000 7ff633009640 1 -- 172.21.15.7:0/2237659305 wait complete. 2023-11-28T18:47:25.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 Processor -- start 2023-11-28T18:47:25.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 -- start start 2023-11-28T18:47:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff62c0c4020 0x7ff62c1ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c150b30 0x7ff62c1eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c1575c0 0x7ff62c1530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:25.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c1ee670 con 0x7ff62c1575c0 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c153950 con 0x7ff62c0c4020 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c153c30 con 0x7ff62c150b30 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c1575c0 0x7ff62c1530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c1575c0 0x7ff62c1530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51894/0 (socket says 172.21.15.7:51894) 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 -- 172.21.15.7:0/1205137558 learned_addr learned my addr 172.21.15.7:0/1205137558 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff630da5640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff62c0c4020 0x7ff62c1ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff62bfff640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c150b30 0x7ff62c1eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 -- 172.21.15.7:0/1205137558 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c150b30 msgr2=0x7ff62c1eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c150b30 0x7ff62c1eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 -- 172.21.15.7:0/1205137558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff62c0c4020 msgr2=0x7ff62c1ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff62c0c4020 0x7ff62c1ee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 -- 172.21.15.7:0/1205137558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6200094d0 con 0x7ff62c1575c0 2023-11-28T18:47:25.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6315a6640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c1575c0 0x7ff62c1530e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff62400c780 tx=0x7ff62400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:25.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6297fa640 1 -- 172.21.15.7:0/1205137558 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624004b70 con 0x7ff62c1575c0 2023-11-28T18:47:25.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6297fa640 1 -- 172.21.15.7:0/1205137558 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62400ce90 con 0x7ff62c1575c0 2023-11-28T18:47:25.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff6297fa640 1 -- 172.21.15.7:0/1205137558 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62400a850 con 0x7ff62c1575c0 2023-11-28T18:47:25.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62c1514d0 con 0x7ff62c1575c0 2023-11-28T18:47:25.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.829+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62c151a90 con 0x7ff62c1575c0 2023-11-28T18:47:25.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.833+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62c04e890 con 0x7ff62c1575c0 2023-11-28T18:47:25.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.833+0000 7ff6297fa640 1 -- 172.21.15.7:0/1205137558 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff62400a9b0 con 0x7ff62c1575c0 2023-11-28T18:47:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.833+0000 7ff6297fa640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6100668d0 0x7ff610068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.833+0000 7ff6297fa640 1 -- 172.21.15.7:0/1205137558 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff624084a00 con 0x7ff62c1575c0 2023-11-28T18:47:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.833+0000 7ff630da5640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6100668d0 0x7ff610068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:25.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.833+0000 7ff630da5640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6100668d0 0x7ff610068d90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff6200b2000 tx=0x7ff620009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:25.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:25.837+0000 7ff6297fa640 1 -- 172.21.15.7:0/1205137558 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff6240517d0 con 0x7ff62c1575c0 2023-11-28T18:47:26.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.025+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff62c0c2d80 con 0x7ff62c1575c0 2023-11-28T18:47:26.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff6297fa640 1 -- 172.21.15.7:0/1205137558 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7ff624054df0 con 0x7ff62c1575c0 2023-11-28T18:47:26.029 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:26.029 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6100668d0 msgr2=0x7ff610068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6100668d0 0x7ff610068d90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff6200b2000 tx=0x7ff620009040 comp rx=0 tx=0).stop 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c1575c0 msgr2=0x7ff62c1530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c1575c0 0x7ff62c1530e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff62400c780 tx=0x7ff62400cc40 comp rx=0 tx=0).stop 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 shutdown_connections 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6100668d0 0x7ff610068d90 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff62c1575c0 0x7ff62c1530e0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff62c150b30 0x7ff62c1eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 --2- 172.21.15.7:0/1205137558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff62c0c4020 0x7ff62c1ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 >> 172.21.15.7:0/1205137558 conn(0x7ff62c0662e0 msgr2=0x7ff62c0c99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:26.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 shutdown_connections 2023-11-28T18:47:26.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:26.029+0000 7ff633009640 1 -- 172.21.15.7:0/1205137558 wait complete. 2023-11-28T18:47:26.051 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:27.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~128s 2023-11-28T18:47:28.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f01602fe640 1 Processor -- start 2023-11-28T18:47:28.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f01602fe640 1 -- start start 2023-11-28T18:47:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f01602fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f01580c24f0 0x7f01581575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f01602fe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01580c6140 con 0x7f01580c7980 2023-11-28T18:47:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f01602fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0158157b00 con 0x7f0158150b30 2023-11-28T18:47:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f01602fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01580c72b0 con 0x7f01580c24f0 2023-11-28T18:47:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f015e89b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f01580c7980 0x7f01580c7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59084/0 (socket says 172.21.15.7:59084) 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f015e89b640 1 -- 172.21.15.7:0/3043605441 learned_addr learned my addr 172.21.15.7:0/3043605441 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f015d899640 1 --2- 172.21.15.7:0/3043605441 >> v2:172.21.15.193:3300/0 conn(0x7f01580c24f0 0x7f01581575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f015d098640 1 -- 172.21.15.7:0/3043605441 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581519117 0 0) 0x7f01580c6140 con 0x7f01580c7980 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.205+0000 7f015d098640 1 -- 172.21.15.7:0/3043605441 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0144003530 con 0x7f01580c7980 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d098640 1 -- 172.21.15.7:0/3043605441 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279370152 0 0) 0x7f0158157b00 con 0x7f0158150b30 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d098640 1 -- 172.21.15.7:0/3043605441 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01580c6140 con 0x7f0158150b30 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d899640 1 -- 172.21.15.7:0/3043605441 >> v1:172.21.15.193:6789/0 conn(0x7f0158150b30 legacy=0x7f01580c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d098640 1 -- 172.21.15.7:0/3043605441 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 867625187 0 0) 0x7f0144003530 con 0x7f01580c7980 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d899640 1 -- 172.21.15.7:0/3043605441 >> v1:172.21.15.7:6789/0 conn(0x7f01580c7980 legacy=0x7f01580c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:28.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d899640 1 -- 172.21.15.7:0/3043605441 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01580c2960 con 0x7f01580c24f0 2023-11-28T18:47:28.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d899640 1 --2- 172.21.15.7:0/3043605441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01580c24f0 0x7f01581575c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01480097c0 tx=0x7f014801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7019ad2c0b0b4df3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:28.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d098640 1 -- 172.21.15.7:0/3043605441 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150001a00 con 0x7f01580c24f0 2023-11-28T18:47:28.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d098640 1 -- 172.21.15.7:0/3043605441 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01480a8610 con 0x7f01580c24f0 2023-11-28T18:47:28.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.209+0000 7f015d098640 1 -- 172.21.15.7:0/3043605441 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480a89a0 con 0x7f01580c24f0 2023-11-28T18:47:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- 172.21.15.7:0/3043605441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01580c24f0 msgr2=0x7f01581575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 --2- 172.21.15.7:0/3043605441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01580c24f0 0x7f01581575c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01480097c0 tx=0x7f014801eb90 comp rx=0 tx=0).stop 2023-11-28T18:47:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- 172.21.15.7:0/3043605441 shutdown_connections 2023-11-28T18:47:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 --2- 172.21.15.7:0/3043605441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01580c24f0 0x7f01581575c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- 172.21.15.7:0/3043605441 >> 172.21.15.7:0/3043605441 conn(0x7f01580662e0 msgr2=0x7f01580666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- 172.21.15.7:0/3043605441 shutdown_connections 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- 172.21.15.7:0/3043605441 wait complete. 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 Processor -- start 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- start start 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01580c24f0 0x7f015814a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01580c7980 0x7f015814a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0158150b30 0x7f0158153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0158070ae0 con 0x7f01580c24f0 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01581ef540 con 0x7f0158150b30 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f01602fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01581ef7c0 con 0x7f01580c7980 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e09a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01580c24f0 0x7f015814a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e09a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01580c24f0 0x7f015814a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51902/0 (socket says 172.21.15.7:51902) 2023-11-28T18:47:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e09a640 1 -- 172.21.15.7:0/3467209482 learned_addr learned my addr 172.21.15.7:0/3467209482 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e89b640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0158150b30 0x7f0158153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015d899640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01580c7980 0x7f015814a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e09a640 1 -- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01580c7980 msgr2=0x7f015814a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e09a640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01580c7980 0x7f015814a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e09a640 1 -- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0158150b30 msgr2=0x7f0158153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e09a640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0158150b30 0x7f0158153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e09a640 1 -- 172.21.15.7:0/3467209482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0148009420 con 0x7f01580c24f0 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015e89b640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0158150b30 0x7f0158153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.213+0000 7f015d899640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01580c7980 0x7f015814a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f015e09a640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01580c24f0 0x7f015814a060 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f015400c430 tx=0x7f015400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f014effd640 1 -- 172.21.15.7:0/3467209482 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540052d0 con 0x7f01580c24f0 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f014effd640 1 -- 172.21.15.7:0/3467209482 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154005450 con 0x7f01580c24f0 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01581efaa0 con 0x7f01580c24f0 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f014effd640 1 -- 172.21.15.7:0/3467209482 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154003840 con 0x7f01580c24f0 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01581f0060 con 0x7f01580c24f0 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015804e890 con 0x7f01580c24f0 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f014effd640 1 -- 172.21.15.7:0/3467209482 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f01540039a0 con 0x7f01580c24f0 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f014effd640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f014effd640 1 -- 172.21.15.7:0/3467209482 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f0154083fe0 con 0x7f01580c24f0 2023-11-28T18:47:28.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.217+0000 7f015d899640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:28.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.221+0000 7f015d899640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f01480b2000 tx=0x7f0148008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.221+0000 7f014effd640 1 -- 172.21.15.7:0/3467209482 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0154050cb0 con 0x7f01580c24f0 2023-11-28T18:47:28.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.417+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f015814f340 con 0x7f01580c24f0 2023-11-28T18:47:28.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.417+0000 7f014effd640 1 -- 172.21.15.7:0/3467209482 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f0154017020 con 0x7f01580c24f0 2023-11-28T18:47:28.418 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:28.419 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:28.419 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.417+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 msgr2=0x7f013c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.417+0000 7f01602fe640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f01480b2000 tx=0x7f0148008040 comp rx=0 tx=0).stop 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.417+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01580c24f0 msgr2=0x7f015814a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.417+0000 7f01602fe640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01580c24f0 0x7f015814a060 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f015400c430 tx=0x7f015400c8f0 comp rx=0 tx=0).stop 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.421+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 shutdown_connections 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.421+0000 7f01602fe640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f013c0668d0 0x7f013c068d90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.421+0000 7f01602fe640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0158150b30 0x7f0158153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.421+0000 7f01602fe640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f01580c7980 0x7f015814a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.421+0000 7f01602fe640 1 --2- 172.21.15.7:0/3467209482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01580c24f0 0x7f015814a060 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.421+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 >> 172.21.15.7:0/3467209482 conn(0x7f01580662e0 msgr2=0x7f01580c18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:28.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.421+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 shutdown_connections 2023-11-28T18:47:28.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:28.421+0000 7f01602fe640 1 -- 172.21.15.7:0/3467209482 wait complete. 2023-11-28T18:47:28.441 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:30.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f305032a640 1 Processor -- start 2023-11-28T18:47:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f305032a640 1 -- start start 2023-11-28T18:47:30.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f305032a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3048151cd0 0x7f304815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f305032a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3048071e60 con 0x7f304806e560 2023-11-28T18:47:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f305032a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3048070930 con 0x7f3048157b90 2023-11-28T18:47:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f305032a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3048157fa0 con 0x7f3048151cd0 2023-11-28T18:47:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f304e0c6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3048151cd0 0x7f304815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f304d8c5640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f304806e560 0x7f30480c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59090/0 (socket says 172.21.15.7:59090) 2023-11-28T18:47:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f304d8c5640 1 -- 172.21.15.7:0/3122089958 learned_addr learned my addr 172.21.15.7:0/3122089958 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f304e0c6640 1 -- 172.21.15.7:0/3122089958 >> v1:172.21.15.193:6789/0 conn(0x7f3048157b90 legacy=0x7f3048162510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f304e0c6640 1 -- 172.21.15.7:0/3122089958 >> v1:172.21.15.7:6789/0 conn(0x7f304806e560 legacy=0x7f30480c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f304e0c6640 1 -- 172.21.15.7:0/3122089958 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048070bf0 con 0x7f3048151cd0 2023-11-28T18:47:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.597+0000 7f304d0c4640 1 -- 172.21.15.7:0/3122089958 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756898115 0 0) 0x7f3048071e60 con 0x7f304806e560 2023-11-28T18:47:30.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f304e0c6640 1 --2- 172.21.15.7:0/3122089958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304815bd00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f30440097c0 tx=0x7f304401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e7f9b8c1d8e308 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:30.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f304d0c4640 1 -- 172.21.15.7:0/3122089958 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30440ab600 con 0x7f3048151cd0 2023-11-28T18:47:30.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f304d0c4640 1 -- 172.21.15.7:0/3122089958 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30440a7050 con 0x7f3048151cd0 2023-11-28T18:47:30.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f304d0c4640 1 -- 172.21.15.7:0/3122089958 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30440ab980 con 0x7f3048151cd0 2023-11-28T18:47:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f305032a640 1 -- 172.21.15.7:0/3122089958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 msgr2=0x7f304815bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f305032a640 1 --2- 172.21.15.7:0/3122089958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304815bd00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f30440097c0 tx=0x7f304401c3f0 comp rx=0 tx=0).stop 2023-11-28T18:47:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f305032a640 1 -- 172.21.15.7:0/3122089958 shutdown_connections 2023-11-28T18:47:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f305032a640 1 --2- 172.21.15.7:0/3122089958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304815bd00 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f305032a640 1 -- 172.21.15.7:0/3122089958 >> 172.21.15.7:0/3122089958 conn(0x7f30480662e0 msgr2=0x7f30480666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f305032a640 1 -- 172.21.15.7:0/3122089958 shutdown_connections 2023-11-28T18:47:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.601+0000 7f305032a640 1 -- 172.21.15.7:0/3122089958 wait complete. 2023-11-28T18:47:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 Processor -- start 2023-11-28T18:47:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 -- start start 2023-11-28T18:47:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f304806e560 0x7f30480c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304814a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3048157b90 0x7f30481f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30481f5f40 con 0x7f3048151cd0 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30480c85c0 con 0x7f304806e560 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30480c7d20 con 0x7f3048157b90 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304814a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304814a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51916/0 (socket says 172.21.15.7:51916) 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 -- 172.21.15.7:0/3417989054 learned_addr learned my addr 172.21.15.7:0/3417989054 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304e8c7640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3048157b90 0x7f30481f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304e0c6640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f304806e560 0x7f30480c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 -- 172.21.15.7:0/3417989054 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3048157b90 msgr2=0x7f30481f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3048157b90 0x7f30481f63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 -- 172.21.15.7:0/3417989054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f304806e560 msgr2=0x7f30480c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f304806e560 0x7f30480c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 -- 172.21.15.7:0/3417989054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3044009420 con 0x7f3048151cd0 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f304d8c5640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304814a740 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f303800c4a0 tx=0x7f303800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f303effd640 1 -- 172.21.15.7:0/3417989054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30380074e0 con 0x7f3048151cd0 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f303effd640 1 -- 172.21.15.7:0/3417989054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038007640 con 0x7f3048151cd0 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f303effd640 1 -- 172.21.15.7:0/3417989054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038004a30 con 0x7f3048151cd0 2023-11-28T18:47:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30480c8000 con 0x7f3048151cd0 2023-11-28T18:47:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.605+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30481f5bd0 con 0x7f3048151cd0 2023-11-28T18:47:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.609+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30480c26c0 con 0x7f3048151cd0 2023-11-28T18:47:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.609+0000 7f303effd640 1 -- 172.21.15.7:0/3417989054 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3038004b90 con 0x7f3048151cd0 2023-11-28T18:47:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.609+0000 7f303effd640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f302c0668d0 0x7f302c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.609+0000 7f303effd640 1 -- 172.21.15.7:0/3417989054 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f3038083bd0 con 0x7f3048151cd0 2023-11-28T18:47:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.609+0000 7f304e0c6640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f302c0668d0 0x7f302c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.609+0000 7f304e0c6640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f302c0668d0 0x7f302c068d90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f30440ad000 tx=0x7f3044008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:30.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.613+0000 7f303effd640 1 -- 172.21.15.7:0/3417989054 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3038050910 con 0x7f3048151cd0 2023-11-28T18:47:30.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f304804e890 con 0x7f3048151cd0 2023-11-28T18:47:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f303effd640 1 -- 172.21.15.7:0/3417989054 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f3038018070 con 0x7f3048151cd0 2023-11-28T18:47:30.806 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:30.806 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:30.807 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f302c0668d0 msgr2=0x7f302c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f302c0668d0 0x7f302c068d90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f30440ad000 tx=0x7f3044008040 comp rx=0 tx=0).stop 2023-11-28T18:47:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 msgr2=0x7f304814a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304814a740 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f303800c4a0 tx=0x7f303800c960 comp rx=0 tx=0).stop 2023-11-28T18:47:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 shutdown_connections 2023-11-28T18:47:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f302c0668d0 0x7f302c068d90 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3048157b90 0x7f30481f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:30.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3048151cd0 0x7f304814a740 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:30.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 --2- 172.21.15.7:0/3417989054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f304806e560 0x7f30480c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:30.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 >> 172.21.15.7:0/3417989054 conn(0x7f30480662e0 msgr2=0x7f30480c5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:30.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 shutdown_connections 2023-11-28T18:47:30.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:30.805+0000 7f305032a640 1 -- 172.21.15.7:0/3417989054 wait complete. 2023-11-28T18:47:30.829 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:32.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:32.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32f5fb8640 1 Processor -- start 2023-11-28T18:47:32.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32f5fb8640 1 -- start start 2023-11-28T18:47:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32f5fb8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f32f00c24f0 0x7f32f01575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32f5fb8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32f00c7980 0x7f32f00c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32f5fb8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f00c6140 con 0x7f32f0150b30 2023-11-28T18:47:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32f5fb8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f00c72b0 con 0x7f32f00c7980 2023-11-28T18:47:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32f5fb8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f0070c00 con 0x7f32f00c24f0 2023-11-28T18:47:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32effff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32f00c7980 0x7f32f00c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32effff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32f00c7980 0x7f32f00c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51928/0 (socket says 172.21.15.7:51928) 2023-11-28T18:47:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32effff640 1 -- 172.21.15.7:0/3245080094 learned_addr learned my addr 172.21.15.7:0/3245080094 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32eeffd640 1 --2- 172.21.15.7:0/3245080094 >> v2:172.21.15.193:3300/0 conn(0x7f32f00c24f0 0x7f32f01575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32effff640 1 -- 172.21.15.7:0/3245080094 >> v2:172.21.15.193:3300/0 conn(0x7f32f00c24f0 msgr2=0x7f32f01575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32effff640 1 --2- 172.21.15.7:0/3245080094 >> v2:172.21.15.193:3300/0 conn(0x7f32f00c24f0 0x7f32f01575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32effff640 1 -- 172.21.15.7:0/3245080094 >> v1:172.21.15.193:6789/0 conn(0x7f32f0150b30 legacy=0x7f32f00c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.985+0000 7f32effff640 1 -- 172.21.15.7:0/3245080094 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f0157b00 con 0x7f32f00c7980 2023-11-28T18:47:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32effff640 1 --2- 172.21.15.7:0/3245080094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f00c7980 0x7f32f00c7dd0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f32f006fee0 tx=0x7f32e801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e27ad796563128 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32ee7fc640 1 -- 172.21.15.7:0/3245080094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80af400 con 0x7f32f00c7980 2023-11-28T18:47:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32ee7fc640 1 -- 172.21.15.7:0/3245080094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e80ab050 con 0x7f32f00c7980 2023-11-28T18:47:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32ee7fc640 1 -- 172.21.15.7:0/3245080094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80af780 con 0x7f32f00c7980 2023-11-28T18:47:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3245080094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f00c7980 msgr2=0x7f32f00c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3245080094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f00c7980 0x7f32f00c7dd0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f32f006fee0 tx=0x7f32e801f1c0 comp rx=0 tx=0).stop 2023-11-28T18:47:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3245080094 shutdown_connections 2023-11-28T18:47:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3245080094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f00c7980 0x7f32f00c7dd0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3245080094 >> v2:172.21.15.193:3300/0 conn(0x7f32f00c24f0 0x7f32f01575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3245080094 >> 172.21.15.7:0/3245080094 conn(0x7f32f00662e0 msgr2=0x7f32f00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3245080094 shutdown_connections 2023-11-28T18:47:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.989+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3245080094 wait complete. 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 Processor -- start 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 -- start start 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f32f00c24f0 0x7f32f0072620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32f00c7980 0x7f32f0072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f0150b30 0x7f32f00717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0152b90 con 0x7f32f0150b30 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f014dc70 con 0x7f32f00c7980 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01f06c0 con 0x7f32f00c24f0 2023-11-28T18:47:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f0150b30 0x7f32f00717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f0150b30 0x7f32f00717c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51932/0 (socket says 172.21.15.7:51932) 2023-11-28T18:47:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 -- 172.21.15.7:0/3000215489 learned_addr learned my addr 172.21.15.7:0/3000215489 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32eeffd640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32f00c7980 0x7f32f0072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32ef7fe640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f32f00c24f0 0x7f32f0072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 -- 172.21.15.7:0/3000215489 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f32f00c24f0 msgr2=0x7f32f0072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f32f00c24f0 0x7f32f0072620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 -- 172.21.15.7:0/3000215489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32f00c7980 msgr2=0x7f32f0072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32f00c7980 0x7f32f0072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 -- 172.21.15.7:0/3000215489 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e8009b30 con 0x7f32f0150b30 2023-11-28T18:47:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32effff640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f0150b30 0x7f32f00717c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f32e80b2040 tx=0x7f32e801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32d7fff640 1 -- 172.21.15.7:0/3000215489 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80afd10 con 0x7f32f0150b30 2023-11-28T18:47:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32d7fff640 1 -- 172.21.15.7:0/3000215489 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e80afe90 con 0x7f32f0150b30 2023-11-28T18:47:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32d7fff640 1 -- 172.21.15.7:0/3000215489 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80ba660 con 0x7f32f0150b30 2023-11-28T18:47:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f01f09a0 con 0x7f32f0150b30 2023-11-28T18:47:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f01542c0 con 0x7f32f0150b30 2023-11-28T18:47:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.993+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f004e890 con 0x7f32f0150b30 2023-11-28T18:47:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.997+0000 7f32d7fff640 1 -- 172.21.15.7:0/3000215489 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f32e80b6040 con 0x7f32f0150b30 2023-11-28T18:47:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.997+0000 7f32d7fff640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f32dc066900 0x7f32dc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.997+0000 7f32d7fff640 1 -- 172.21.15.7:0/3000215489 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f32e81229f0 con 0x7f32f0150b30 2023-11-28T18:47:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.997+0000 7f32ef7fe640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f32dc066900 0x7f32dc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:32.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:32.997+0000 7f32ef7fe640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f32dc066900 0x7f32dc068dc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000f70 tx=0x7f32e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:33.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.001+0000 7f32d7fff640 1 -- 172.21.15.7:0/3000215489 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f32e80ef900 con 0x7f32f0150b30 2023-11-28T18:47:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32f00735d0 con 0x7f32f0150b30 2023-11-28T18:47:33.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32d7fff640 1 -- 172.21.15.7:0/3000215489 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f32e8009510 con 0x7f32f0150b30 2023-11-28T18:47:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:33.201 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:33.202 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:33.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f32dc066900 msgr2=0x7f32dc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f32dc066900 0x7f32dc068dc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000f70 tx=0x7f32e0009040 comp rx=0 tx=0).stop 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f0150b30 msgr2=0x7f32f00717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f0150b30 0x7f32f00717c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f32e80b2040 tx=0x7f32e801bf60 comp rx=0 tx=0).stop 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 shutdown_connections 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f32dc066900 0x7f32dc068dc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f0150b30 0x7f32f00717c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32f00c7980 0x7f32f0072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 --2- 172.21.15.7:0/3000215489 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f32f00c24f0 0x7f32f0072620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 >> 172.21.15.7:0/3000215489 conn(0x7f32f00662e0 msgr2=0x7f32f00caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 shutdown_connections 2023-11-28T18:47:33.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:33.193+0000 7f32f5fb8640 1 -- 172.21.15.7:0/3000215489 wait complete. 2023-11-28T18:47:33.222 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:34.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~135s 2023-11-28T18:47:35.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:35.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e5d51d640 1 Processor -- start 2023-11-28T18:47:35.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e5d51d640 1 -- start start 2023-11-28T18:47:35.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e5d51d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e581542e0 0x7f2e5815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:35.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e5d51d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e580ca920 con 0x7f2e580cbb70 2023-11-28T18:47:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e5d51d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5815e550 con 0x7f2e58072450 2023-11-28T18:47:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e5d51d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e58071d50 con 0x7f2e581542e0 2023-11-28T18:47:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e56ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e581542e0 0x7f2e5815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e4ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2e580cbb70 0x7f2e580cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57292/0 (socket says 172.21.15.7:57292) 2023-11-28T18:47:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e4ffff640 1 -- 172.21.15.7:0/1041304372 learned_addr learned my addr 172.21.15.7:0/1041304372 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e56ffd640 1 -- 172.21.15.7:0/1041304372 >> v1:172.21.15.193:6789/0 conn(0x7f2e58072450 legacy=0x7f2e58164990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e56ffd640 1 -- 172.21.15.7:0/1041304372 >> v1:172.21.15.7:6789/0 conn(0x7f2e580cbb70 legacy=0x7f2e580cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.377+0000 7f2e56ffd640 1 -- 172.21.15.7:0/1041304372 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e580c4cd0 con 0x7f2e581542e0 2023-11-28T18:47:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e56ffd640 1 --2- 172.21.15.7:0/1041304372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e581542e0 0x7f2e5815e010 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2e480097c0 tx=0x7f2e4801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2559cb49c1e688 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e567fc640 1 -- 172.21.15.7:0/1041304372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e480ab680 con 0x7f2e581542e0 2023-11-28T18:47:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e567fc640 1 -- 172.21.15.7:0/1041304372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e480ab7e0 con 0x7f2e581542e0 2023-11-28T18:47:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e567fc640 1 -- 172.21.15.7:0/1041304372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e480abb70 con 0x7f2e581542e0 2023-11-28T18:47:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1041304372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e581542e0 msgr2=0x7f2e5815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e5d51d640 1 --2- 172.21.15.7:0/1041304372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e581542e0 0x7f2e5815e010 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2e480097c0 tx=0x7f2e4801c480 comp rx=0 tx=0).stop 2023-11-28T18:47:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1041304372 shutdown_connections 2023-11-28T18:47:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e5d51d640 1 --2- 172.21.15.7:0/1041304372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e581542e0 0x7f2e5815e010 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1041304372 >> 172.21.15.7:0/1041304372 conn(0x7f2e580717e0 msgr2=0x7f2e580662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1041304372 shutdown_connections 2023-11-28T18:47:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.381+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1041304372 wait complete. 2023-11-28T18:47:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 Processor -- start 2023-11-28T18:47:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 -- start start 2023-11-28T18:47:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e58072450 0x7f2e581f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e580cbb70 0x7f2e58150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e581542e0 0x7f2e5815c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e58072030 con 0x7f2e580cbb70 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580ce5f0 con 0x7f2e58072450 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e581f3a00 con 0x7f2e581542e0 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e580cbb70 0x7f2e58150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e580cbb70 0x7f2e58150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38024/0 (socket says 172.21.15.7:38024) 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 -- 172.21.15.7:0/1471643813 learned_addr learned my addr 172.21.15.7:0/1471643813 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e56ffd640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e58072450 0x7f2e581f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e577fe640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e581542e0 0x7f2e5815c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 -- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e581542e0 msgr2=0x7f2e5815c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e581542e0 0x7f2e5815c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 -- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e58072450 msgr2=0x7f2e581f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e58072450 0x7f2e581f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 -- 172.21.15.7:0/1471643813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e48009420 con 0x7f2e580cbb70 2023-11-28T18:47:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e56ffd640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e58072450 0x7f2e581f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4ffff640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e580cbb70 0x7f2e58150610 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000c500 tx=0x7f2e4000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4f7fe640 1 -- 172.21.15.7:0/1471643813 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40005cf0 con 0x7f2e580cbb70 2023-11-28T18:47:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4f7fe640 1 -- 172.21.15.7:0/1471643813 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e40005e50 con 0x7f2e580cbb70 2023-11-28T18:47:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e58157440 con 0x7f2e580cbb70 2023-11-28T18:47:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e4f7fe640 1 -- 172.21.15.7:0/1471643813 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e400072f0 con 0x7f2e580cbb70 2023-11-28T18:47:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.385+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e581576e0 con 0x7f2e580cbb70 2023-11-28T18:47:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.389+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5806c670 con 0x7f2e580cbb70 2023-11-28T18:47:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.389+0000 7f2e4f7fe640 1 -- 172.21.15.7:0/1471643813 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2e40007450 con 0x7f2e580cbb70 2023-11-28T18:47:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.389+0000 7f2e4f7fe640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e380668d0 0x7f2e38068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.389+0000 7f2e4f7fe640 1 -- 172.21.15.7:0/1471643813 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f2e40083e20 con 0x7f2e580cbb70 2023-11-28T18:47:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.389+0000 7f2e56ffd640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e380668d0 0x7f2e38068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.393+0000 7f2e56ffd640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e380668d0 0x7f2e38068d90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2e480af000 tx=0x7f2e480a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.393+0000 7f2e4f7fe640 1 -- 172.21.15.7:0/1471643813 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2e40050c80 con 0x7f2e580cbb70 2023-11-28T18:47:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.585+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e5804e890 con 0x7f2e580cbb70 2023-11-28T18:47:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.585+0000 7f2e4f7fe640 1 -- 172.21.15.7:0/1471643813 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f2e40017020 con 0x7f2e580cbb70 2023-11-28T18:47:35.587 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:35.587 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e380668d0 msgr2=0x7f2e38068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e380668d0 0x7f2e38068d90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2e480af000 tx=0x7f2e480a7000 comp rx=0 tx=0).stop 2023-11-28T18:47:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e580cbb70 msgr2=0x7f2e58150610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e580cbb70 0x7f2e58150610 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000c500 tx=0x7f2e4000c9c0 comp rx=0 tx=0).stop 2023-11-28T18:47:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 shutdown_connections 2023-11-28T18:47:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e380668d0 0x7f2e38068d90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e581542e0 0x7f2e5815c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e580cbb70 0x7f2e58150610 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 --2- 172.21.15.7:0/1471643813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e58072450 0x7f2e581f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 >> 172.21.15.7:0/1471643813 conn(0x7f2e580717e0 msgr2=0x7f2e581535a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 shutdown_connections 2023-11-28T18:47:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:35.589+0000 7f2e5d51d640 1 -- 172.21.15.7:0/1471643813 wait complete. 2023-11-28T18:47:35.611 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:47:37.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3559f8640 1 Processor -- start 2023-11-28T18:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3559f8640 1 -- start start 2023-11-28T18:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3559f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 0x7fe3500ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3559f8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe35015a8a0 0x7fe35015dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3559f8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35015d7e0 con 0x7fe3500cf6c0 2023-11-28T18:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3559f8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3500cab80 con 0x7fe35015a8a0 2023-11-28T18:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3559f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3500cb850 con 0x7fe3501540e0 2023-11-28T18:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3467fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe35015a8a0 0x7fe35015dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3467fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe35015a8a0 0x7fe35015dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38040/0 (socket says 172.21.15.7:38040) 2023-11-28T18:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3467fc640 1 -- 172.21.15.7:0/2492604989 learned_addr learned my addr 172.21.15.7:0/2492604989 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:37.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe34effd640 1 --2- 172.21.15.7:0/2492604989 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 0x7fe3500ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:37.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3467fc640 1 -- 172.21.15.7:0/2492604989 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 msgr2=0x7fe3500ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:37.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3467fc640 1 --2- 172.21.15.7:0/2492604989 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 0x7fe3500ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3467fc640 1 -- 172.21.15.7:0/2492604989 >> v1:172.21.15.193:6789/0 conn(0x7fe3500cf6c0 legacy=0x7fe3500cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:37.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.765+0000 7fe3467fc640 1 -- 172.21.15.7:0/2492604989 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3500cbb30 con 0x7fe35015a8a0 2023-11-28T18:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3467fc640 1 --2- 172.21.15.7:0/2492604989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe35015dc70 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe340009870 tx=0x7fe34001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb322726a68bc804 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe34e7fc640 1 -- 172.21.15.7:0/2492604989 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400ab650 con 0x7fe35015a8a0 2023-11-28T18:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe34e7fc640 1 -- 172.21.15.7:0/2492604989 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3400ab7b0 con 0x7fe35015a8a0 2023-11-28T18:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe34e7fc640 1 -- 172.21.15.7:0/2492604989 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400abb40 con 0x7fe35015a8a0 2023-11-28T18:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3559f8640 1 -- 172.21.15.7:0/2492604989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 msgr2=0x7fe35015dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3559f8640 1 --2- 172.21.15.7:0/2492604989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe35015dc70 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe340009870 tx=0x7fe34001c540 comp rx=0 tx=0).stop 2023-11-28T18:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3559f8640 1 -- 172.21.15.7:0/2492604989 shutdown_connections 2023-11-28T18:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3559f8640 1 --2- 172.21.15.7:0/2492604989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe35015dc70 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3559f8640 1 --2- 172.21.15.7:0/2492604989 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 0x7fe3500ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3559f8640 1 -- 172.21.15.7:0/2492604989 >> 172.21.15.7:0/2492604989 conn(0x7fe3500662e0 msgr2=0x7fe3500666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3559f8640 1 -- 172.21.15.7:0/2492604989 shutdown_connections 2023-11-28T18:47:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.769+0000 7fe3559f8640 1 -- 172.21.15.7:0/2492604989 wait complete. 2023-11-28T18:47:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 Processor -- start 2023-11-28T18:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 -- start start 2023-11-28T18:47:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3500cf6c0 0x7fe35015cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 0x7fe3500b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe3500b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35015afa0 con 0x7fe35015a8a0 2023-11-28T18:47:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3500cf180 con 0x7fe3500cf6c0 2023-11-28T18:47:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35015d0e0 con 0x7fe3501540e0 2023-11-28T18:47:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe3500b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe3500b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38054/0 (socket says 172.21.15.7:38054) 2023-11-28T18:47:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 -- 172.21.15.7:0/515004686 learned_addr learned my addr 172.21.15.7:0/515004686 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34effd640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3500cf6c0 0x7fe35015cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:37.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3467fc640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 0x7fe3500b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:37.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 -- 172.21.15.7:0/515004686 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 msgr2=0x7fe3500b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:37.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 0x7fe3500b5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 -- 172.21.15.7:0/515004686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3500cf6c0 msgr2=0x7fe35015cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:37.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3500cf6c0 0x7fe35015cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 -- 172.21.15.7:0/515004686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3400094d0 con 0x7fe35015a8a0 2023-11-28T18:47:37.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34f7fe640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe3500b3a80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c7a0 tx=0x7fe34800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:37.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34cff9640 1 -- 172.21.15.7:0/515004686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348003b80 con 0x7fe35015a8a0 2023-11-28T18:47:37.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34cff9640 1 -- 172.21.15.7:0/515004686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348003ce0 con 0x7fe35015a8a0 2023-11-28T18:47:37.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35015b1f0 con 0x7fe35015a8a0 2023-11-28T18:47:37.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe34cff9640 1 -- 172.21.15.7:0/515004686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34800aa90 con 0x7fe35015a8a0 2023-11-28T18:47:37.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3500ce850 con 0x7fe35015a8a0 2023-11-28T18:47:37.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.773+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35004e890 con 0x7fe35015a8a0 2023-11-28T18:47:37.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.777+0000 7fe34cff9640 1 -- 172.21.15.7:0/515004686 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe34800abf0 con 0x7fe35015a8a0 2023-11-28T18:47:37.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.777+0000 7fe34cff9640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3300668d0 0x7fe330068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:37.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.777+0000 7fe34cff9640 1 -- 172.21.15.7:0/515004686 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fe348084660 con 0x7fe35015a8a0 2023-11-28T18:47:37.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.777+0000 7fe34effd640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3300668d0 0x7fe330068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:37.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.777+0000 7fe34effd640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3300668d0 0x7fe330068d90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe338007a60 tx=0x7fe338008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:37.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.781+0000 7fe34cff9640 1 -- 172.21.15.7:0/515004686 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe348051430 con 0x7fe35015a8a0 2023-11-28T18:47:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.969+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3500cf2c0 con 0x7fe35015a8a0 2023-11-28T18:47:37.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.969+0000 7fe34cff9640 1 -- 172.21.15.7:0/515004686 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fe348054a50 con 0x7fe35015a8a0 2023-11-28T18:47:37.973 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:37.973 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:37.975 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3300668d0 msgr2=0x7fe330068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3300668d0 0x7fe330068d90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe338007a60 tx=0x7fe338008040 comp rx=0 tx=0).stop 2023-11-28T18:47:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 msgr2=0x7fe3500b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe3500b3a80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c7a0 tx=0x7fe34800cc60 comp rx=0 tx=0).stop 2023-11-28T18:47:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 shutdown_connections 2023-11-28T18:47:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe3300668d0 0x7fe330068d90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe35015a8a0 0x7fe3500b3a80 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe3501540e0 0x7fe3500b5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 --2- 172.21.15.7:0/515004686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3500cf6c0 0x7fe35015cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 >> 172.21.15.7:0/515004686 conn(0x7fe3500662e0 msgr2=0x7fe3500666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:37.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 shutdown_connections 2023-11-28T18:47:37.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:37.973+0000 7fe3559f8640 1 -- 172.21.15.7:0/515004686 wait complete. 2023-11-28T18:47:37.994 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:39.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.149+0000 7f074545a640 1 Processor -- start 2023-11-28T18:47:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.149+0000 7f074545a640 1 -- start start 2023-11-28T18:47:40.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f074545a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0740151f00 0x7f07400cab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f074545a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07400c4740 0x7f07400cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f074545a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f07401522e0 0x7f07400cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f074545a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07400cb0d0 con 0x7f07400c4740 2023-11-28T18:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f074545a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07400c9f00 con 0x7f07401522e0 2023-11-28T18:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f074545a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0740072950 con 0x7f0740151f00 2023-11-28T18:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07400c4740 0x7f07400cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07400c4740 0x7f07400cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38068/0 (socket says 172.21.15.7:38068) 2023-11-28T18:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 -- 172.21.15.7:0/4184095354 learned_addr learned my addr 172.21.15.7:0/4184095354 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f073f7fe640 1 --2- 172.21.15.7:0/4184095354 >> v2:172.21.15.193:3300/0 conn(0x7f07401522e0 0x7f07400cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f073effd640 1 --2- 172.21.15.7:0/4184095354 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0740151f00 0x7f07400cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 -- 172.21.15.7:0/4184095354 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0740151f00 msgr2=0x7f07400cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 --2- 172.21.15.7:0/4184095354 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0740151f00 0x7f07400cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 -- 172.21.15.7:0/4184095354 >> v2:172.21.15.193:3300/0 conn(0x7f07401522e0 msgr2=0x7f07400cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 --2- 172.21.15.7:0/4184095354 >> v2:172.21.15.193:3300/0 conn(0x7f07401522e0 0x7f07400cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 -- 172.21.15.7:0/4184095354 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0740072c30 con 0x7f07400c4740 2023-11-28T18:47:40.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f073effd640 1 --2- 172.21.15.7:0/4184095354 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0740151f00 0x7f07400cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:47:40.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.153+0000 7f0737fff640 1 --2- 172.21.15.7:0/4184095354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f07400cd6b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f07400c5fb0 tx=0x7f072801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5469ca210904adb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:40.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f073e7fc640 1 -- 172.21.15.7:0/4184095354 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07280ab600 con 0x7f07400c4740 2023-11-28T18:47:40.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f073e7fc640 1 -- 172.21.15.7:0/4184095354 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07280a7050 con 0x7f07400c4740 2023-11-28T18:47:40.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f073e7fc640 1 -- 172.21.15.7:0/4184095354 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07280ab980 con 0x7f07400c4740 2023-11-28T18:47:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- 172.21.15.7:0/4184095354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 msgr2=0x7f07400cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 --2- 172.21.15.7:0/4184095354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f07400cd6b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f07400c5fb0 tx=0x7f072801c470 comp rx=0 tx=0).stop 2023-11-28T18:47:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- 172.21.15.7:0/4184095354 shutdown_connections 2023-11-28T18:47:40.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 --2- 172.21.15.7:0/4184095354 >> v2:172.21.15.193:3300/0 conn(0x7f07401522e0 0x7f07400cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 --2- 172.21.15.7:0/4184095354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f07400cd6b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 --2- 172.21.15.7:0/4184095354 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0740151f00 0x7f07400cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- 172.21.15.7:0/4184095354 >> 172.21.15.7:0/4184095354 conn(0x7f0740073cd0 msgr2=0x7f07400662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:40.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- 172.21.15.7:0/4184095354 shutdown_connections 2023-11-28T18:47:40.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- 172.21.15.7:0/4184095354 wait complete. 2023-11-28T18:47:40.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 Processor -- start 2023-11-28T18:47:40.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- start start 2023-11-28T18:47:40.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f074014b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:40.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0740151f00 0x7f07400cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:40.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f074014b8d0 0x7f074014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:40.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07400cc6a0 con 0x7f07400c4740 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07400ccf80 con 0x7f0740151f00 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f074545a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074014b620 con 0x7f074014b8d0 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f073effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f074014b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f073effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f074014b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38072/0 (socket says 172.21.15.7:38072) 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.157+0000 7f073effd640 1 -- 172.21.15.7:0/1478764262 learned_addr learned my addr 172.21.15.7:0/1478764262 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f0737fff640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0740151f00 0x7f07400cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f073f7fe640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f074014b8d0 0x7f074014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f073effd640 1 -- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f074014b8d0 msgr2=0x7f074014ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f073effd640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f074014b8d0 0x7f074014ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f073effd640 1 -- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0740151f00 msgr2=0x7f07400cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f073effd640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0740151f00 0x7f07400cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f073effd640 1 -- 172.21.15.7:0/1478764262 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07280094d0 con 0x7f07400c4740 2023-11-28T18:47:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f073f7fe640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f074014b8d0 0x7f074014ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f073effd640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f074014b0e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f073000c7c0 tx=0x7f073000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f07377fe640 1 -- 172.21.15.7:0/1478764262 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730007a30 con 0x7f07400c4740 2023-11-28T18:47:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f07377fe640 1 -- 172.21.15.7:0/1478764262 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0730007b90 con 0x7f07400c4740 2023-11-28T18:47:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f07377fe640 1 -- 172.21.15.7:0/1478764262 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073000ce70 con 0x7f07400c4740 2023-11-28T18:47:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074015ae10 con 0x7f07400c4740 2023-11-28T18:47:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07401f41e0 con 0x7f07400c4740 2023-11-28T18:47:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074004e890 con 0x7f07400c4740 2023-11-28T18:47:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f07377fe640 1 -- 172.21.15.7:0/1478764262 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f07300113f0 con 0x7f07400c4740 2023-11-28T18:47:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.161+0000 7f07377fe640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0720066900 0x7f0720068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.165+0000 7f07377fe640 1 -- 172.21.15.7:0/1478764262 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f0730084980 con 0x7f07400c4740 2023-11-28T18:47:40.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.165+0000 7f0737fff640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0720066900 0x7f0720068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:40.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.165+0000 7f0737fff640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0720066900 0x7f0720068dc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f07400741d0 tx=0x7f0728008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:40.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.169+0000 7f07377fe640 1 -- 172.21.15.7:0/1478764262 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f07300517e0 con 0x7f07400c4740 2023-11-28T18:47:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0740162440 con 0x7f07400c4740 2023-11-28T18:47:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f07377fe640 1 -- 172.21.15.7:0/1478764262 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f0730054e00 con 0x7f07400c4740 2023-11-28T18:47:40.361 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:40.361 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0720066900 msgr2=0x7f0720068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0720066900 0x7f0720068dc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f07400741d0 tx=0x7f0728008040 comp rx=0 tx=0).stop 2023-11-28T18:47:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 msgr2=0x7f074014b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f074014b0e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f073000c7c0 tx=0x7f073000cc80 comp rx=0 tx=0).stop 2023-11-28T18:47:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 shutdown_connections 2023-11-28T18:47:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0720066900 0x7f0720068dc0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f074014b8d0 0x7f074014ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0740151f00 0x7f07400cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 --2- 172.21.15.7:0/1478764262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07400c4740 0x7f074014b0e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:40.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 >> 172.21.15.7:0/1478764262 conn(0x7f0740073cd0 msgr2=0x7f07400c7610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:40.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 shutdown_connections 2023-11-28T18:47:40.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:40.361+0000 7f074545a640 1 -- 172.21.15.7:0/1478764262 wait complete. 2023-11-28T18:47:40.383 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:40.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~141s 2023-11-28T18:47:41.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:47:41.699 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:47:42.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc65e6640 1 Processor -- start 2023-11-28T18:47:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc65e6640 1 -- start start 2023-11-28T18:47:42.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc65e6640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4cc00cae40 0x7f4cc00ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:42.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc65e6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4cc00cf9f0 0x7f4cc00cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:42.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc65e6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc015d4e0 con 0x7f4cc00cb220 2023-11-28T18:47:42.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc65e6640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc00d03c0 con 0x7f4cc00cf9f0 2023-11-28T18:47:42.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc65e6640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc00717f0 con 0x7f4cc00cae40 2023-11-28T18:47:42.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc4b83640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4cc00cf9f0 0x7f4cc00cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:42.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc4b83640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4cc00cf9f0 0x7f4cc00cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38074/0 (socket says 172.21.15.7:38074) 2023-11-28T18:47:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc4b83640 1 -- 172.21.15.7:0/2218619332 learned_addr learned my addr 172.21.15.7:0/2218619332 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cbffff640 1 --2- 172.21.15.7:0/2218619332 >> v2:172.21.15.193:3300/0 conn(0x7f4cc00cae40 0x7f4cc00ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc4b83640 1 -- 172.21.15.7:0/2218619332 >> v2:172.21.15.193:3300/0 conn(0x7f4cc00cae40 msgr2=0x7f4cc00ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc4b83640 1 --2- 172.21.15.7:0/2218619332 >> v2:172.21.15.193:3300/0 conn(0x7f4cc00cae40 0x7f4cc00ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc4b83640 1 -- 172.21.15.7:0/2218619332 >> v1:172.21.15.193:6789/0 conn(0x7f4cc00cb220 legacy=0x7f4cc0070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.536+0000 7f4cc4b83640 1 -- 172.21.15.7:0/2218619332 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc015d730 con 0x7f4cc00cf9f0 2023-11-28T18:47:42.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc4b83640 1 --2- 172.21.15.7:0/2218619332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cf9f0 0x7f4cc00cfe80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00ccb50 tx=0x7f4cb801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a4317f2ecd58f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:42.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cbeffd640 1 -- 172.21.15.7:0/2218619332 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a5860 con 0x7f4cc00cf9f0 2023-11-28T18:47:42.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cbeffd640 1 -- 172.21.15.7:0/2218619332 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb80a59c0 con 0x7f4cc00cf9f0 2023-11-28T18:47:42.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cbeffd640 1 -- 172.21.15.7:0/2218619332 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a5d50 con 0x7f4cc00cf9f0 2023-11-28T18:47:42.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc65e6640 1 -- 172.21.15.7:0/2218619332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cf9f0 msgr2=0x7f4cc00cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:42.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/2218619332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cf9f0 0x7f4cc00cfe80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00ccb50 tx=0x7f4cb801c9a0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc65e6640 1 -- 172.21.15.7:0/2218619332 shutdown_connections 2023-11-28T18:47:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/2218619332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cf9f0 0x7f4cc00cfe80 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/2218619332 >> v2:172.21.15.193:3300/0 conn(0x7f4cc00cae40 0x7f4cc00ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc65e6640 1 -- 172.21.15.7:0/2218619332 >> 172.21.15.7:0/2218619332 conn(0x7f4cc0072310 msgr2=0x7f4cc0072720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc65e6640 1 -- 172.21.15.7:0/2218619332 shutdown_connections 2023-11-28T18:47:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.540+0000 7f4cc65e6640 1 -- 172.21.15.7:0/2218619332 wait complete. 2023-11-28T18:47:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 Processor -- start 2023-11-28T18:47:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 -- start start 2023-11-28T18:47:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cae40 0x7f4cc014cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cb220 0x7f4cc015b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc00cf9f0 0x7f4cc015bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc014dfa0 con 0x7f4cc00cb220 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc01471c0 con 0x7f4cc00cf9f0 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc0146d30 con 0x7f4cc00cae40 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cb220 0x7f4cc015b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cb220 0x7f4cc015b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38086/0 (socket says 172.21.15.7:38086) 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 -- 172.21.15.7:0/3972650538 learned_addr learned my addr 172.21.15.7:0/3972650538 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc4b83640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc00cf9f0 0x7f4cc015bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbffff640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cae40 0x7f4cc014cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 -- 172.21.15.7:0/3972650538 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cae40 msgr2=0x7f4cc014cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cae40 0x7f4cc014cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 -- 172.21.15.7:0/3972650538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc00cf9f0 msgr2=0x7f4cc015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc00cf9f0 0x7f4cc015bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 -- 172.21.15.7:0/3972650538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb8009b30 con 0x7f4cc00cb220 2023-11-28T18:47:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbf7fe640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cb220 0x7f4cc015b710 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400c3c0 tx=0x7f4cb400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbcff9640 1 -- 172.21.15.7:0/3972650538 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40052d0 con 0x7f4cc00cb220 2023-11-28T18:47:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbcff9640 1 -- 172.21.15.7:0/3972650538 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4005430 con 0x7f4cc00cb220 2023-11-28T18:47:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cbcff9640 1 -- 172.21.15.7:0/3972650538 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4003840 con 0x7f4cc00cb220 2023-11-28T18:47:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.544+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc014f560 con 0x7f4cc00cb220 2023-11-28T18:47:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.548+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc014f920 con 0x7f4cc00cb220 2023-11-28T18:47:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.548+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc0067a90 con 0x7f4cc00cb220 2023-11-28T18:47:42.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.548+0000 7f4cbcff9640 1 -- 172.21.15.7:0/3972650538 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4cb40039a0 con 0x7f4cc00cb220 2023-11-28T18:47:42.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.548+0000 7f4cbcff9640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cac066900 0x7f4cac068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:42.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.548+0000 7f4cbcff9640 1 -- 172.21.15.7:0/3972650538 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f4cb4083e00 con 0x7f4cc00cb220 2023-11-28T18:47:42.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.548+0000 7f4cbffff640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cac066900 0x7f4cac068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:42.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.548+0000 7f4cbffff640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cac066900 0x7f4cac068dc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009ac0 tx=0x7f4cb0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:42.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.552+0000 7f4cbcff9640 1 -- 172.21.15.7:0/3972650538 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4cb4050bd0 con 0x7f4cc00cb220 2023-11-28T18:47:42.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.748+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cc0146ea0 con 0x7f4cc00cb220 2023-11-28T18:47:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.748+0000 7f4cbcff9640 1 -- 172.21.15.7:0/3972650538 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f4cb4017090 con 0x7f4cc00cb220 2023-11-28T18:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cac066900 msgr2=0x7f4cac068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cac066900 0x7f4cac068dc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009ac0 tx=0x7f4cb0009190 comp rx=0 tx=0).stop 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cb220 msgr2=0x7f4cc015b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cb220 0x7f4cc015b710 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400c3c0 tx=0x7f4cb400c880 comp rx=0 tx=0).stop 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 shutdown_connections 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4cac066900 0x7f4cac068dc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cc00cf9f0 0x7f4cc015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cc00cb220 0x7f4cc015b710 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 --2- 172.21.15.7:0/3972650538 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4cc00cae40 0x7f4cc014cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 >> 172.21.15.7:0/3972650538 conn(0x7f4cc0072310 msgr2=0x7f4cc00726f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 shutdown_connections 2023-11-28T18:47:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:42.752+0000 7f4cc65e6640 1 -- 172.21.15.7:0/3972650538 wait complete. 2023-11-28T18:47:42.774 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:44.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.928+0000 7f9673c09640 1 Processor -- start 2023-11-28T18:47:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.928+0000 7f9673c09640 1 -- start start 2023-11-28T18:47:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f9673c09640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c150b30 0x7f966c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f9673c09640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f966c1575c0 0x7f966c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f9673c09640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c0c6140 con 0x7f966c0c4020 2023-11-28T18:47:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f9673c09640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f966c0c72b0 con 0x7f966c1575c0 2023-11-28T18:47:44.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f9673c09640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c070c00 con 0x7f966c150b30 2023-11-28T18:47:44.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96711a4640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f966c1575c0 0x7f966c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:44.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96719a5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c150b30 0x7f966c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:44.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96721a6640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f966c0c4020 0x7f966c0c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:35022/0 (socket says 172.21.15.7:35022) 2023-11-28T18:47:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96721a6640 1 -- 172.21.15.7:0/2677324796 learned_addr learned my addr 172.21.15.7:0/2677324796 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96709a3640 1 -- 172.21.15.7:0/2677324796 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709163323 0 0) 0x7f966c0c6140 con 0x7f966c0c4020 2023-11-28T18:47:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96709a3640 1 -- 172.21.15.7:0/2677324796 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9658003530 con 0x7f966c0c4020 2023-11-28T18:47:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96711a4640 1 -- 172.21.15.7:0/2677324796 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c150b30 msgr2=0x7f966c0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96711a4640 1 --2- 172.21.15.7:0/2677324796 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c150b30 0x7f966c0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96711a4640 1 -- 172.21.15.7:0/2677324796 >> v1:172.21.15.193:6789/0 conn(0x7f966c0c4020 legacy=0x7f966c0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.932+0000 7f96711a4640 1 -- 172.21.15.7:0/2677324796 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c0c6370 con 0x7f966c1575c0 2023-11-28T18:47:44.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f96711a4640 1 --2- 172.21.15.7:0/2677324796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c1575c0 0x7f966c0c3ae0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f965c00ba60 tx=0x7f965c021340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6e7680d89266954 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:44.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f96709a3640 1 -- 172.21.15.7:0/2677324796 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c0b1600 con 0x7f966c1575c0 2023-11-28T18:47:44.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f96709a3640 1 -- 172.21.15.7:0/2677324796 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965c0ad050 con 0x7f966c1575c0 2023-11-28T18:47:44.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f96709a3640 1 -- 172.21.15.7:0/2677324796 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c0b1980 con 0x7f966c1575c0 2023-11-28T18:47:44.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 -- 172.21.15.7:0/2677324796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c1575c0 msgr2=0x7f966c0c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 --2- 172.21.15.7:0/2677324796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c1575c0 0x7f966c0c3ae0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f965c00ba60 tx=0x7f965c021340 comp rx=0 tx=0).stop 2023-11-28T18:47:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 -- 172.21.15.7:0/2677324796 shutdown_connections 2023-11-28T18:47:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 --2- 172.21.15.7:0/2677324796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c1575c0 0x7f966c0c3ae0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 --2- 172.21.15.7:0/2677324796 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c150b30 0x7f966c0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 -- 172.21.15.7:0/2677324796 >> 172.21.15.7:0/2677324796 conn(0x7f966c0662e0 msgr2=0x7f966c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 -- 172.21.15.7:0/2677324796 shutdown_connections 2023-11-28T18:47:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 -- 172.21.15.7:0/2677324796 wait complete. 2023-11-28T18:47:44.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 Processor -- start 2023-11-28T18:47:44.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.936+0000 7f9673c09640 1 -- start start 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f966c0c4020 0x7f966c1e69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c150b30 0x7f966c14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c1575c0 0x7f966c1f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c14d8c0 con 0x7f966c0c4020 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c14b190 con 0x7f966c150b30 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c14b430 con 0x7f966c1575c0 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f966c0c4020 0x7f966c1e69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f966c0c4020 0x7f966c1e69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50202/0 (socket says 172.21.15.7:50202) 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 -- 172.21.15.7:0/2222791049 learned_addr learned my addr 172.21.15.7:0/2222791049 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96721a6640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c1575c0 0x7f966c1f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96711a4640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c150b30 0x7f966c14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:44.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 -- 172.21.15.7:0/2222791049 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c1575c0 msgr2=0x7f966c1f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:44.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c1575c0 0x7f966c1f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:44.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 -- 172.21.15.7:0/2222791049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c150b30 msgr2=0x7f966c14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:44.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c150b30 0x7f966c14eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:44.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 -- 172.21.15.7:0/2222791049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c00b6c0 con 0x7f966c0c4020 2023-11-28T18:47:44.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96719a5640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f966c0c4020 0x7f966c1e69c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f966800e930 tx=0x7f966800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:44.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96627fc640 1 -- 172.21.15.7:0/2222791049 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668009690 con 0x7f966c0c4020 2023-11-28T18:47:44.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96627fc640 1 -- 172.21.15.7:0/2222791049 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966800ca00 con 0x7f966c0c4020 2023-11-28T18:47:44.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f96627fc640 1 -- 172.21.15.7:0/2222791049 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966800cd70 con 0x7f966c0c4020 2023-11-28T18:47:44.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c14db10 con 0x7f966c0c4020 2023-11-28T18:47:44.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c1f3ad0 con 0x7f966c0c4020 2023-11-28T18:47:44.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.940+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c04e890 con 0x7f966c0c4020 2023-11-28T18:47:44.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.944+0000 7f96627fc640 1 -- 172.21.15.7:0/2222791049 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9668013470 con 0x7f966c0c4020 2023-11-28T18:47:44.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.944+0000 7f96627fc640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f96500668d0 0x7f9650068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:44.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.944+0000 7f96627fc640 1 -- 172.21.15.7:0/2222791049 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f9668084c40 con 0x7f966c0c4020 2023-11-28T18:47:44.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.944+0000 7f96711a4640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f96500668d0 0x7f9650068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:44.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.944+0000 7f96711a4640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f96500668d0 0x7f9650068d90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f965c0b3000 tx=0x7f965c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:44.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:44.948+0000 7f96627fc640 1 -- 172.21.15.7:0/2222791049 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9668051990 con 0x7f966c0c4020 2023-11-28T18:47:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.136+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f966c0c4500 con 0x7f966c0c4020 2023-11-28T18:47:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f96627fc640 1 -- 172.21.15.7:0/2222791049 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f966c0c4500 con 0x7f966c0c4020 2023-11-28T18:47:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:45.142 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:45.143 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:45.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f96500668d0 msgr2=0x7f9650068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:45.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f96500668d0 0x7f9650068d90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f965c0b3000 tx=0x7f965c008040 comp rx=0 tx=0).stop 2023-11-28T18:47:45.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f966c0c4020 msgr2=0x7f966c1e69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:45.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f966c0c4020 0x7f966c1e69c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f966800e930 tx=0x7f966800edf0 comp rx=0 tx=0).stop 2023-11-28T18:47:45.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 shutdown_connections 2023-11-28T18:47:45.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f96500668d0 0x7f9650068d90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:45.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f966c1575c0 0x7f966c1f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:45.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f966c150b30 0x7f966c14eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:45.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 --2- 172.21.15.7:0/2222791049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f966c0c4020 0x7f966c1e69c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:45.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 >> 172.21.15.7:0/2222791049 conn(0x7f966c0662e0 msgr2=0x7f966c0c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:45.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 shutdown_connections 2023-11-28T18:47:45.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:45.140+0000 7f9673c09640 1 -- 172.21.15.7:0/2222791049 wait complete. 2023-11-28T18:47:45.163 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:47.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:47.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa631279640 1 Processor -- start 2023-11-28T18:47:47.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa631279640 1 -- start start 2023-11-28T18:47:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa631279640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa62c150b30 0x7fa62c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa631279640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa62c1575c0 0x7fa62c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa631279640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62c0c6140 con 0x7fa62c0c4020 2023-11-28T18:47:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa631279640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62c0c72b0 con 0x7fa62c1575c0 2023-11-28T18:47:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa631279640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62c070c00 con 0x7fa62c150b30 2023-11-28T18:47:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa62b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa62c0c4020 0x7fa62c0c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39050/0 (socket says 172.21.15.7:39050) 2023-11-28T18:47:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa623fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa62c1575c0 0x7fa62c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.316+0000 7fa62b7fe640 1 -- 172.21.15.7:0/3832658153 learned_addr learned my addr 172.21.15.7:0/3832658153 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa62affd640 1 --2- 172.21.15.7:0/3832658153 >> v2:172.21.15.193:3300/0 conn(0x7fa62c150b30 0x7fa62c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:47.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa623fff640 1 -- 172.21.15.7:0/3832658153 >> v2:172.21.15.193:3300/0 conn(0x7fa62c150b30 msgr2=0x7fa62c0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:47.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa623fff640 1 --2- 172.21.15.7:0/3832658153 >> v2:172.21.15.193:3300/0 conn(0x7fa62c150b30 0x7fa62c0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa623fff640 1 -- 172.21.15.7:0/3832658153 >> v1:172.21.15.7:6789/0 conn(0x7fa62c0c4020 legacy=0x7fa62c0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:47.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa623fff640 1 -- 172.21.15.7:0/3832658153 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c0c6370 con 0x7fa62c1575c0 2023-11-28T18:47:47.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa62a7fc640 1 -- 172.21.15.7:0/3832658153 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114964523 0 0) 0x7fa62c0c6140 con 0x7fa62c0c4020 2023-11-28T18:47:47.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa623fff640 1 --2- 172.21.15.7:0/3832658153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c0c3ae0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa62c06ec80 tx=0x7fa61401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337c59a303d10a3a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:47.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa62a7fc640 1 -- 172.21.15.7:0/3832658153 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6140a8950 con 0x7fa62c1575c0 2023-11-28T18:47:47.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa62a7fc640 1 -- 172.21.15.7:0/3832658153 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6140a8ab0 con 0x7fa62c1575c0 2023-11-28T18:47:47.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.320+0000 7fa62a7fc640 1 -- 172.21.15.7:0/3832658153 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6140a8e40 con 0x7fa62c1575c0 2023-11-28T18:47:47.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- 172.21.15.7:0/3832658153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 msgr2=0x7fa62c0c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:47.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 --2- 172.21.15.7:0/3832658153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c0c3ae0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa62c06ec80 tx=0x7fa61401ec50 comp rx=0 tx=0).stop 2023-11-28T18:47:47.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- 172.21.15.7:0/3832658153 shutdown_connections 2023-11-28T18:47:47.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 --2- 172.21.15.7:0/3832658153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c0c3ae0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 --2- 172.21.15.7:0/3832658153 >> v2:172.21.15.193:3300/0 conn(0x7fa62c150b30 0x7fa62c0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- 172.21.15.7:0/3832658153 >> 172.21.15.7:0/3832658153 conn(0x7fa62c0662e0 msgr2=0x7fa62c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:47.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- 172.21.15.7:0/3832658153 shutdown_connections 2023-11-28T18:47:47.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- 172.21.15.7:0/3832658153 wait complete. 2023-11-28T18:47:47.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 Processor -- start 2023-11-28T18:47:47.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- start start 2023-11-28T18:47:47.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c0c4020 0x7fa62c0721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:47.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa62c150b30 0x7fa62c1eb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:47.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:47.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c14dd60 con 0x7fa62c1575c0 2023-11-28T18:47:47.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c14b190 con 0x7fa62c0c4020 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa631279640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c14f040 con 0x7fa62c150b30 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c149fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50220/0 (socket says 172.21.15.7:50220) 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62b7fe640 1 -- 172.21.15.7:0/2634760834 learned_addr learned my addr 172.21.15.7:0/2634760834 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62affd640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c0c4020 0x7fa62c0721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa623fff640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa62c150b30 0x7fa62c1eb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62b7fe640 1 -- 172.21.15.7:0/2634760834 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa62c150b30 msgr2=0x7fa62c1eb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62b7fe640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa62c150b30 0x7fa62c1eb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62b7fe640 1 -- 172.21.15.7:0/2634760834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c0c4020 msgr2=0x7fa62c0721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62b7fe640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c0c4020 0x7fa62c0721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.324+0000 7fa62b7fe640 1 -- 172.21.15.7:0/2634760834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa614009420 con 0x7fa62c1575c0 2023-11-28T18:47:47.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa62b7fe640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c149fd0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa624001570 tx=0x7fa62400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:47.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa6237fe640 1 -- 172.21.15.7:0/2634760834 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62400a640 con 0x7fa62c1575c0 2023-11-28T18:47:47.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa6237fe640 1 -- 172.21.15.7:0/2634760834 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62400a7c0 con 0x7fa62c1575c0 2023-11-28T18:47:47.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa6237fe640 1 -- 172.21.15.7:0/2634760834 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62400ab70 con 0x7fa62c1575c0 2023-11-28T18:47:47.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62c14f320 con 0x7fa62c1575c0 2023-11-28T18:47:47.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62c0c5900 con 0x7fa62c1575c0 2023-11-28T18:47:47.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62c04e890 con 0x7fa62c1575c0 2023-11-28T18:47:47.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa6237fe640 1 -- 172.21.15.7:0/2634760834 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa62400acd0 con 0x7fa62c1575c0 2023-11-28T18:47:47.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa6237fe640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa60c0668d0 0x7fa60c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:47.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa6237fe640 1 -- 172.21.15.7:0/2634760834 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fa624083e40 con 0x7fa62c1575c0 2023-11-28T18:47:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.328+0000 7fa62affd640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa60c0668d0 0x7fa60c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.332+0000 7fa62affd640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa60c0668d0 0x7fa60c068d90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00b9d0 tx=0x7fa61c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:47.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.332+0000 7fa6237fe640 1 -- 172.21.15.7:0/2634760834 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa624050c00 con 0x7fa62c1575c0 2023-11-28T18:47:47.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.524+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa62c0c2d80 con 0x7fa62c1575c0 2023-11-28T18:47:47.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.524+0000 7fa6237fe640 1 -- 172.21.15.7:0/2634760834 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fa624018020 con 0x7fa62c1575c0 2023-11-28T18:47:47.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:47.527 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa60c0668d0 msgr2=0x7fa60c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa60c0668d0 0x7fa60c068d90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00b9d0 tx=0x7fa61c008040 comp rx=0 tx=0).stop 2023-11-28T18:47:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 msgr2=0x7fa62c149fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c149fd0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa624001570 tx=0x7fa62400cca0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 shutdown_connections 2023-11-28T18:47:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa60c0668d0 0x7fa60c068d90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa62c1575c0 0x7fa62c149fd0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa62c150b30 0x7fa62c1eb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 --2- 172.21.15.7:0/2634760834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c0c4020 0x7fa62c0721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:47.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 >> 172.21.15.7:0/2634760834 conn(0x7fa62c0662e0 msgr2=0x7fa62c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:47.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 shutdown_connections 2023-11-28T18:47:47.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:47.528+0000 7fa631279640 1 -- 172.21.15.7:0/2634760834 wait complete. 2023-11-28T18:47:47.550 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:47:47.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~148s 2023-11-28T18:47:49.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.704+0000 7f04f27f8640 1 Processor -- start 2023-11-28T18:47:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.704+0000 7f04f27f8640 1 -- start start 2023-11-28T18:47:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.704+0000 7f04f27f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec071250 0x7f04ec0c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.704+0000 7f04f27f8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04ec0ccbb0 0x7f04ec0c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f27f8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ec0c48e0 con 0x7f04ec070e70 2023-11-28T18:47:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f27f8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec15e330 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f27f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec15d7b0 con 0x7f04ec071250 2023-11-28T18:47:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04ebfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f04ec070e70 0x7f04ec15a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39054/0 (socket says 172.21.15.7:39054) 2023-11-28T18:47:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04ebfff640 1 -- 172.21.15.7:0/690652667 learned_addr learned my addr 172.21.15.7:0/690652667 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f0d95640 1 --2- 172.21.15.7:0/690652667 >> v2:172.21.15.7:3300/0 conn(0x7f04ec0ccbb0 0x7f04ec0c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04eb7fe640 1 --2- 172.21.15.7:0/690652667 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec071250 0x7f04ec0c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04eaffd640 1 -- 172.21.15.7:0/690652667 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361226838 0 0) 0x7f04ec0c48e0 con 0x7f04ec070e70 2023-11-28T18:47:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04eaffd640 1 -- 172.21.15.7:0/690652667 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d4003530 con 0x7f04ec070e70 2023-11-28T18:47:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f0d95640 1 -- 172.21.15.7:0/690652667 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec071250 msgr2=0x7f04ec0c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f0d95640 1 --2- 172.21.15.7:0/690652667 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec071250 0x7f04ec0c4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f0d95640 1 -- 172.21.15.7:0/690652667 >> v1:172.21.15.7:6789/0 conn(0x7f04ec070e70 legacy=0x7f04ec15a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f0d95640 1 -- 172.21.15.7:0/690652667 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec15da90 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.708+0000 7f04f0d95640 1 --2- 172.21.15.7:0/690652667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec0c52b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009d90 tx=0x7f04e401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffed8d3589f14c44 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04eaffd640 1 -- 172.21.15.7:0/690652667 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e40af600 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04eaffd640 1 -- 172.21.15.7:0/690652667 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e40ab050 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04eaffd640 1 -- 172.21.15.7:0/690652667 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e40af960 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- 172.21.15.7:0/690652667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 msgr2=0x7f04ec0c52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:49.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 --2- 172.21.15.7:0/690652667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec0c52b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009d90 tx=0x7f04e401f130 comp rx=0 tx=0).stop 2023-11-28T18:47:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- 172.21.15.7:0/690652667 shutdown_connections 2023-11-28T18:47:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 --2- 172.21.15.7:0/690652667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec0c52b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 --2- 172.21.15.7:0/690652667 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec071250 0x7f04ec0c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- 172.21.15.7:0/690652667 >> 172.21.15.7:0/690652667 conn(0x7f04ec0662e0 msgr2=0x7f04ec0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- 172.21.15.7:0/690652667 shutdown_connections 2023-11-28T18:47:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- 172.21.15.7:0/690652667 wait complete. 2023-11-28T18:47:49.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 Processor -- start 2023-11-28T18:47:49.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- start start 2023-11-28T18:47:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec070e70 0x7f04ec1ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04ec071250 0x7f04ec14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec15ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec1efd20 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0ce610 con 0x7f04ec071250 2023-11-28T18:47:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f27f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec14e190 con 0x7f04ec070e70 2023-11-28T18:47:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.712+0000 7f04f0d95640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec15ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f0d95640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec15ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50246/0 (socket says 172.21.15.7:50246) 2023-11-28T18:47:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f0d95640 1 -- 172.21.15.7:0/2884934635 learned_addr learned my addr 172.21.15.7:0/2884934635 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04ebfff640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec070e70 0x7f04ec1ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:49.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04eb7fe640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04ec071250 0x7f04ec14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:49.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f0d95640 1 -- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec070e70 msgr2=0x7f04ec1ef7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:49.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f0d95640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec070e70 0x7f04ec1ef7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f0d95640 1 -- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04ec071250 msgr2=0x7f04ec14feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:49.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f0d95640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04ec071250 0x7f04ec14feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f0d95640 1 -- 172.21.15.7:0/2884934635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e4009a70 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04eb7fe640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04ec071250 0x7f04ec14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:49.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f0d95640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec15ca80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f04e401cd60 tx=0x7f04e401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:49.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04e8ff9640 1 -- 172.21.15.7:0/2884934635 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e40bf070 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04e8ff9640 1 -- 172.21.15.7:0/2884934635 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e4004250 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04e8ff9640 1 -- 172.21.15.7:0/2884934635 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4004600 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec14e430 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec0ceb40 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec04e890 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.716+0000 7f04e8ff9640 1 -- 172.21.15.7:0/2884934635 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f04e4004760 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.720+0000 7f04e8ff9640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f04d80668d0 0x7f04d8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:49.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.720+0000 7f04e8ff9640 1 -- 172.21.15.7:0/2884934635 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f04e4122bc0 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.720+0000 7f04ebfff640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f04d80668d0 0x7f04d8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:49.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.720+0000 7f04ebfff640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f04d80668d0 0x7f04d8068d90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f04dc007fd0 tx=0x7f04dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:49.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.724+0000 7f04e8ff9640 1 -- 172.21.15.7:0/2884934635 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f04e40f3370 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.912+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04ec1503f0 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04e8ff9640 1 -- 172.21.15.7:0/2884934635 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f04e40b4020 con 0x7f04ec0ccbb0 2023-11-28T18:47:49.917 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:49.917 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f04d80668d0 msgr2=0x7f04d8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f04d80668d0 0x7f04d8068d90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f04dc007fd0 tx=0x7f04dc009040 comp rx=0 tx=0).stop 2023-11-28T18:47:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 msgr2=0x7f04ec15ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec15ca80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f04e401cd60 tx=0x7f04e401bed0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 shutdown_connections 2023-11-28T18:47:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f04d80668d0 0x7f04d8068d90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04ec0ccbb0 0x7f04ec15ca80 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04ec071250 0x7f04ec14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 --2- 172.21.15.7:0/2884934635 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f04ec070e70 0x7f04ec1ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 >> 172.21.15.7:0/2884934635 conn(0x7f04ec0662e0 msgr2=0x7f04ec0cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 shutdown_connections 2023-11-28T18:47:49.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:49.916+0000 7f04f27f8640 1 -- 172.21.15.7:0/2884934635 wait complete. 2023-11-28T18:47:49.938 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:51.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.092+0000 7f09db957640 1 Processor -- start 2023-11-28T18:47:52.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.092+0000 7f09db957640 1 -- start start 2023-11-28T18:47:52.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09db957640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d40c68c0 0x7f09d40c2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09db957640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f09d415e300 0x7f09d4157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09db957640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d415e820 con 0x7f09d40c6ca0 2023-11-28T18:47:52.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09db957640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d415eb00 con 0x7f09d415e300 2023-11-28T18:47:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09db957640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415ede0 con 0x7f09d40c68c0 2023-11-28T18:47:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d8ef2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f09d40c6ca0 0x7f09d4161f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39058/0 (socket says 172.21.15.7:39058) 2023-11-28T18:47:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d8ef2640 1 -- 172.21.15.7:0/455507869 learned_addr learned my addr 172.21.15.7:0/455507869 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d9ef4640 1 --2- 172.21.15.7:0/455507869 >> v2:172.21.15.193:3300/0 conn(0x7f09d415e300 0x7f09d4157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d96f3640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d40c68c0 0x7f09d40c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971273004 0 0) 0x7f09d415e820 con 0x7f09d40c6ca0 2023-11-28T18:47:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c0003530 con 0x7f09d40c6ca0 2023-11-28T18:47:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3609474050 0 0) 0x7f09c0003530 con 0x7f09d40c6ca0 2023-11-28T18:47:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d40c68c0 msgr2=0x7f09d40c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:52.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d40c68c0 0x7f09d40c2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 >> v2:172.21.15.193:3300/0 conn(0x7f09d415e300 msgr2=0x7f09d4157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 --2- 172.21.15.7:0/455507869 >> v2:172.21.15.193:3300/0 conn(0x7f09d415e300 0x7f09d4157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d415f0c0 con 0x7f09d40c6ca0 2023-11-28T18:47:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f09c4001680 con 0x7f09d40c6ca0 2023-11-28T18:47:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 >> v1:172.21.15.7:6789/0 conn(0x7f09d40c6ca0 legacy=0x7f09d4161f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:52.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09c0003980 0x7f09c0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09c0004370 0x7f09c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09c000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415eb00 con 0x7f09c0004ec0 2023-11-28T18:47:52.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415ede0 con 0x7f09c0004370 2023-11-28T18:47:52.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c000bed0 con 0x7f09c0003980 2023-11-28T18:47:52.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d9ef4640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09c000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d8ef2640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09c0004370 0x7f09c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d96f3640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09c0003980 0x7f09c0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d9ef4640 1 -- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09c0003980 msgr2=0x7f09c0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:52.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d9ef4640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09c0003980 0x7f09c0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d9ef4640 1 -- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09c0004370 msgr2=0x7f09c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:52.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d9ef4640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09c0004370 0x7f09c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d9ef4640 1 -- 172.21.15.7:0/455507869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0003530 con 0x7f09c0004ec0 2023-11-28T18:47:52.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.096+0000 7f09d8ef2640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09c0004370 0x7f09c0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:52.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09d9ef4640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09c000b990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00e170 tx=0x7f09cc01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71c8bc349c8de27 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:52.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0a5c90 con 0x7f09c0004ec0 2023-11-28T18:47:52.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc00f040 con 0x7f09c0004ec0 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09cbfff640 1 -- 172.21.15.7:0/455507869 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0ad440 con 0x7f09c0004ec0 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09db957640 1 -- 172.21.15.7:0/455507869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 msgr2=0x7f09c000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09db957640 1 --2- 172.21.15.7:0/455507869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09c000b990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00e170 tx=0x7f09cc01cc80 comp rx=0 tx=0).stop 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09d9ef4640 1 -- 172.21.15.7:0/455507869 reap_dead start 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09db957640 1 -- 172.21.15.7:0/455507869 shutdown_connections 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09db957640 1 -- 172.21.15.7:0/455507869 >> 172.21.15.7:0/455507869 conn(0x7f09d40662e0 msgr2=0x7f09d40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09db957640 1 -- 172.21.15.7:0/455507869 shutdown_connections 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09db957640 1 -- 172.21.15.7:0/455507869 wait complete. 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.100+0000 7f09db957640 1 Processor -- start 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 -- start start 2023-11-28T18:47:52.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09d40ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d4160d50 0x7f09d415f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09d415fa00 0x7f09d4149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415b1a0 con 0x7f09c0004ec0 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415a100 con 0x7f09d415fa00 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415a3e0 con 0x7f09d4160d50 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09d40ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09d40ca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50268/0 (socket says 172.21.15.7:50268) 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 -- 172.21.15.7:0/3450807257 learned_addr learned my addr 172.21.15.7:0/3450807257 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d8ef2640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d4160d50 0x7f09d415f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d9ef4640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09d415fa00 0x7f09d4149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 -- 172.21.15.7:0/3450807257 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d4160d50 msgr2=0x7f09d415f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d4160d50 0x7f09d415f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 -- 172.21.15.7:0/3450807257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09d415fa00 msgr2=0x7f09d4149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09d415fa00 0x7f09d4149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 -- 172.21.15.7:0/3450807257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d415a6c0 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09d96f3640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09d40ca7f0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f09d00114c0 tx=0x7f09d0011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09c9ffb640 1 -- 172.21.15.7:0/3450807257 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0015870 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09c9ffb640 1 -- 172.21.15.7:0/3450807257 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d00159d0 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09c9ffb640 1 -- 172.21.15.7:0/3450807257 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0015d80 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d415a9a0 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d4161510 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.104+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d415fde0 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.108+0000 7f09c9ffb640 1 -- 172.21.15.7:0/3450807257 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f09d001d400 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.108+0000 7f09c9ffb640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f09b80668d0 0x7f09b8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.108+0000 7f09c9ffb640 1 -- 172.21.15.7:0/3450807257 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f09d0085110 con 0x7f09c0004ec0 2023-11-28T18:47:52.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.108+0000 7f09d8ef2640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f09b80668d0 0x7f09b8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:52.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.108+0000 7f09d8ef2640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f09b80668d0 0x7f09b8068d90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f09d41f8150 tx=0x7f09c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:52.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.112+0000 7f09c9ffb640 1 -- 172.21.15.7:0/3450807257 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f09d0051da0 con 0x7f09c0004ec0 2023-11-28T18:47:52.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.300+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09d404e890 con 0x7f09c0004ec0 2023-11-28T18:47:52.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09c9ffb640 1 -- 172.21.15.7:0/3450807257 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f09d0022090 con 0x7f09c0004ec0 2023-11-28T18:47:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:52.305 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:52.305 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:52.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f09b80668d0 msgr2=0x7f09b8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:52.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f09b80668d0 0x7f09b8068d90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f09d41f8150 tx=0x7f09c4009040 comp rx=0 tx=0).stop 2023-11-28T18:47:52.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 msgr2=0x7f09d40ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:52.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09d40ca7f0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f09d00114c0 tx=0x7f09d0011980 comp rx=0 tx=0).stop 2023-11-28T18:47:52.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 shutdown_connections 2023-11-28T18:47:52.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f09b80668d0 0x7f09b8068d90 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09d415fa00 0x7f09d4149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f09d4160d50 0x7f09d415f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 --2- 172.21.15.7:0/3450807257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c0004ec0 0x7f09d40ca7f0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:52.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 >> 172.21.15.7:0/3450807257 conn(0x7f09d40662e0 msgr2=0x7f09d415ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:52.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 shutdown_connections 2023-11-28T18:47:52.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:52.304+0000 7f09db957640 1 -- 172.21.15.7:0/3450807257 wait complete. 2023-11-28T18:47:52.326 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:54.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:54.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~155s 2023-11-28T18:47:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f6661980640 1 Processor -- start 2023-11-28T18:47:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f6661980640 1 -- start start 2023-11-28T18:47:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f6661980640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f665c072950 0x7f665c074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f6661980640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c073070 0x7f665c0c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f6661980640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c0c8490 con 0x7f665c0c1ef0 2023-11-28T18:47:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f6661980640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c0c81f0 con 0x7f665c072950 2023-11-28T18:47:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f6661980640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c15ab00 con 0x7f665c073070 2023-11-28T18:47:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f665affd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f665c072950 0x7f665c074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f6653fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c073070 0x7f665c0c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f665affd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f665c072950 0x7f665c074030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:55336/0 (socket says 172.21.15.7:55336) 2023-11-28T18:47:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f665affd640 1 -- 172.21.15.7:0/788330051 learned_addr learned my addr 172.21.15.7:0/788330051 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.480+0000 7f665b7fe640 1 --1- 172.21.15.7:0/788330051 >> v1:172.21.15.193:6789/0 conn(0x7f665c0c1ef0 0x7f665c06c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:56790/0 (socket says 172.21.15.7:56790) 2023-11-28T18:47:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.484+0000 7f665affd640 1 -- 172.21.15.7:0/788330051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c073070 msgr2=0x7f665c0c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.484+0000 7f665affd640 1 --2- 172.21.15.7:0/788330051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c073070 0x7f665c0c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.484+0000 7f665affd640 1 -- 172.21.15.7:0/788330051 >> v1:172.21.15.193:6789/0 conn(0x7f665c0c1ef0 legacy=0x7f665c06c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.484+0000 7f665affd640 1 -- 172.21.15.7:0/788330051 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c15ade0 con 0x7f665c072950 2023-11-28T18:47:54.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.484+0000 7f665affd640 1 --2- 172.21.15.7:0/788330051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c072950 0x7f665c074030 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f664c009870 tx=0x7f664c01c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d28bebc476404bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:54.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.484+0000 7f665a7fc640 1 -- 172.21.15.7:0/788330051 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c0ab600 con 0x7f665c072950 2023-11-28T18:47:54.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.484+0000 7f665a7fc640 1 -- 172.21.15.7:0/788330051 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c0a7050 con 0x7f665c072950 2023-11-28T18:47:54.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.484+0000 7f665a7fc640 1 -- 172.21.15.7:0/788330051 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c0ab980 con 0x7f665c072950 2023-11-28T18:47:54.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- 172.21.15.7:0/788330051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c072950 msgr2=0x7f665c074030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:54.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 --2- 172.21.15.7:0/788330051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c072950 0x7f665c074030 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f664c009870 tx=0x7f664c01c420 comp rx=0 tx=0).stop 2023-11-28T18:47:54.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- 172.21.15.7:0/788330051 shutdown_connections 2023-11-28T18:47:54.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 --2- 172.21.15.7:0/788330051 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c073070 0x7f665c0c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 --2- 172.21.15.7:0/788330051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c072950 0x7f665c074030 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- 172.21.15.7:0/788330051 >> 172.21.15.7:0/788330051 conn(0x7f665c0cbf80 msgr2=0x7f665c0cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:54.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- 172.21.15.7:0/788330051 shutdown_connections 2023-11-28T18:47:54.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- 172.21.15.7:0/788330051 wait complete. 2023-11-28T18:47:54.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 Processor -- start 2023-11-28T18:47:54.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- start start 2023-11-28T18:47:54.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c072950 0x7f665c1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:54.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f665c073070 0x7f665c153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:54.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c0c1ef0 0x7f665c1f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:54.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c151ce0 con 0x7f665c073070 2023-11-28T18:47:54.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c144be0 con 0x7f665c0c1ef0 2023-11-28T18:47:54.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6661980640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c151a90 con 0x7f665c072950 2023-11-28T18:47:54.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6653fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f665c073070 0x7f665c153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:54.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6653fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f665c073070 0x7f665c153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48626/0 (socket says 172.21.15.7:48626) 2023-11-28T18:47:54.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6653fff640 1 -- 172.21.15.7:0/1392106369 learned_addr learned my addr 172.21.15.7:0/1392106369 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f665b7fe640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c0c1ef0 0x7f665c1f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f665affd640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c072950 0x7f665c1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6653fff640 1 -- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c072950 msgr2=0x7f665c1526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6653fff640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c072950 0x7f665c1526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6653fff640 1 -- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c0c1ef0 msgr2=0x7f665c1f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6653fff640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c0c1ef0 0x7f665c1f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f6653fff640 1 -- 172.21.15.7:0/1392106369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0094d0 con 0x7f665c073070 2023-11-28T18:47:54.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f665b7fe640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c0c1ef0 0x7f665c1f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:54.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.488+0000 7f665affd640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c072950 0x7f665c1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:54.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6653fff640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f665c073070 0x7f665c153390 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f664400e900 tx=0x7f664400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:54.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6658ff9640 1 -- 172.21.15.7:0/1392106369 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6644009690 con 0x7f665c073070 2023-11-28T18:47:54.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6658ff9640 1 -- 172.21.15.7:0/1392106369 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664400ca00 con 0x7f665c073070 2023-11-28T18:47:54.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c13fd80 con 0x7f665c073070 2023-11-28T18:47:54.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6658ff9640 1 -- 172.21.15.7:0/1392106369 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664400cd90 con 0x7f665c073070 2023-11-28T18:47:54.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c144520 con 0x7f665c073070 2023-11-28T18:47:54.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c144d20 con 0x7f665c073070 2023-11-28T18:47:54.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6658ff9640 1 -- 172.21.15.7:0/1392106369 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6644013470 con 0x7f665c073070 2023-11-28T18:47:54.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6658ff9640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f663c066900 0x7f663c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:54.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f6658ff9640 1 -- 172.21.15.7:0/1392106369 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f6644084b00 con 0x7f665c073070 2023-11-28T18:47:54.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.492+0000 7f665affd640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f663c066900 0x7f663c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:54.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.496+0000 7f665affd640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f663c066900 0x7f663c068dc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f664c0ad000 tx=0x7f664c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:54.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.496+0000 7f6658ff9640 1 -- 172.21.15.7:0/1392106369 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f665c144d20 con 0x7f665c073070 2023-11-28T18:47:54.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.684+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f665c144d20 con 0x7f665c073070 2023-11-28T18:47:54.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.684+0000 7f6658ff9640 1 -- 172.21.15.7:0/1392106369 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f665c144d20 con 0x7f665c073070 2023-11-28T18:47:54.688 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:54.688 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:54.689 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f663c066900 msgr2=0x7f663c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f663c066900 0x7f663c068dc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f664c0ad000 tx=0x7f664c008040 comp rx=0 tx=0).stop 2023-11-28T18:47:54.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f665c073070 msgr2=0x7f665c153390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f665c073070 0x7f665c153390 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f664400e900 tx=0x7f664400edc0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 shutdown_connections 2023-11-28T18:47:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f663c066900 0x7f663c068dc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f665c0c1ef0 0x7f665c1f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f665c073070 0x7f665c153390 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 --2- 172.21.15.7:0/1392106369 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f665c072950 0x7f665c1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:54.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 >> 172.21.15.7:0/1392106369 conn(0x7f665c0cbf80 msgr2=0x7f665c0cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:54.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 shutdown_connections 2023-11-28T18:47:54.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:54.688+0000 7f6661980640 1 -- 172.21.15.7:0/1392106369 wait complete. 2023-11-28T18:47:54.710 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:56.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:56.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.864+0000 7f28fa8dd640 1 Processor -- start 2023-11-28T18:47:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.864+0000 7f28fa8dd640 1 -- start start 2023-11-28T18:47:56.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.864+0000 7f28fa8dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40c3490 0x7f28f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.864+0000 7f28fa8dd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f28f406df00 0x7f28f40c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28fa8dd640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f40c2ce0 con 0x7f28f40c3bc0 2023-11-28T18:47:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28fa8dd640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f4157240 con 0x7f28f406df00 2023-11-28T18:47:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28fa8dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40c2f80 con 0x7f28f40c3490 2023-11-28T18:47:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f28f40c3bc0 0x7f28f41503c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39342/0 (socket says 172.21.15.7:39342) 2023-11-28T18:47:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f3fff640 1 -- 172.21.15.7:0/2249317764 learned_addr learned my addr 172.21.15.7:0/2249317764 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f8e7a640 1 --2- 172.21.15.7:0/2249317764 >> v2:172.21.15.193:3300/0 conn(0x7f28f406df00 0x7f28f40c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f37fe640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40c3490 0x7f28f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912188790 0 0) 0x7f28f40c2ce0 con 0x7f28f40c3bc0 2023-11-28T18:47:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28dc003530 con 0x7f28f40c3bc0 2023-11-28T18:47:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2087020076 0 0) 0x7f28dc003530 con 0x7f28f40c3bc0 2023-11-28T18:47:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40c3490 msgr2=0x7f28f406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40c3490 0x7f28f406d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 >> v2:172.21.15.193:3300/0 conn(0x7f28f406df00 msgr2=0x7f28f40c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 --2- 172.21.15.7:0/2249317764 >> v2:172.21.15.193:3300/0 conn(0x7f28f406df00 0x7f28f40c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f4156f60 con 0x7f28f40c3bc0 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f28e4001680 con 0x7f28f40c3bc0 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 >> v1:172.21.15.7:6789/0 conn(0x7f28f40c3bc0 legacy=0x7f28f41503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc003980 0x7f28dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40c3490 0x7f28dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4157240 con 0x7f28dc004d10 2023-11-28T18:47:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40c2f80 con 0x7f28dc003980 2023-11-28T18:47:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc00dba0 con 0x7f28f40c3490 2023-11-28T18:47:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f8e7a640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f3fff640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc003980 0x7f28dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f37fe640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40c3490 0x7f28dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f8e7a640 1 -- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40c3490 msgr2=0x7f28dc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f8e7a640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40c3490 0x7f28dc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f8e7a640 1 -- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc003980 msgr2=0x7f28dc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f8e7a640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc003980 0x7f28dc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f8e7a640 1 -- 172.21.15.7:0/2249317764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc003530 con 0x7f28dc004d10 2023-11-28T18:47:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.868+0000 7f28f3fff640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc003980 0x7f28dc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28f8e7a640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28dc005190 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009fc0 tx=0x7f28ec01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e7b861d255dbe3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec0a5cb0 con 0x7f28dc004d10 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec00f040 con 0x7f28dc004d10 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28f2ffd640 1 -- 172.21.15.7:0/2249317764 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec0ad680 con 0x7f28dc004d10 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28fa8dd640 1 -- 172.21.15.7:0/2249317764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 msgr2=0x7f28dc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28fa8dd640 1 --2- 172.21.15.7:0/2249317764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28dc005190 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009fc0 tx=0x7f28ec01cca0 comp rx=0 tx=0).stop 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28f8e7a640 1 -- 172.21.15.7:0/2249317764 reap_dead start 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28fa8dd640 1 -- 172.21.15.7:0/2249317764 shutdown_connections 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28fa8dd640 1 -- 172.21.15.7:0/2249317764 >> 172.21.15.7:0/2249317764 conn(0x7f28f40662e0 msgr2=0x7f28f40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28fa8dd640 1 -- 172.21.15.7:0/2249317764 shutdown_connections 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28fa8dd640 1 -- 172.21.15.7:0/2249317764 wait complete. 2023-11-28T18:47:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28fa8dd640 1 Processor -- start 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.872+0000 7f28fa8dd640 1 -- start start 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28f4146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40cafe0 0x7f28f41469d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28f4148310 0x7f28f4148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f41578d0 con 0x7f28dc004d10 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4147260 con 0x7f28f4148310 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4072490 con 0x7f28f40cafe0 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28f4146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28f4146490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48636/0 (socket says 172.21.15.7:48636) 2023-11-28T18:47:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 -- 172.21.15.7:0/3214771118 learned_addr learned my addr 172.21.15.7:0/3214771118 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f37fe640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40cafe0 0x7f28f41469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f8e7a640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28f4148310 0x7f28f4148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 -- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40cafe0 msgr2=0x7f28f41469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40cafe0 0x7f28f41469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 -- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28f4148310 msgr2=0x7f28f4148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28f4148310 0x7f28f4148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:56.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 -- 172.21.15.7:0/3214771118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f4148cd0 con 0x7f28dc004d10 2023-11-28T18:47:56.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f8e7a640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28f4148310 0x7f28f4148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:47:56.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f3fff640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28f4146490 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f28e4001210 tx=0x7f28e400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:56.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f0ff9640 1 -- 172.21.15.7:0/3214771118 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e400c8d0 con 0x7f28dc004d10 2023-11-28T18:47:56.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f0ff9640 1 -- 172.21.15.7:0/3214771118 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e400ca30 con 0x7f28dc004d10 2023-11-28T18:47:56.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28f0ff9640 1 -- 172.21.15.7:0/3214771118 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e400cdc0 con 0x7f28dc004d10 2023-11-28T18:47:56.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f414ea60 con 0x7f28dc004d10 2023-11-28T18:47:56.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f4147790 con 0x7f28dc004d10 2023-11-28T18:47:56.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.876+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f404e890 con 0x7f28dc004d10 2023-11-28T18:47:56.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.880+0000 7f28f0ff9640 1 -- 172.21.15.7:0/3214771118 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f28e401d460 con 0x7f28dc004d10 2023-11-28T18:47:56.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.880+0000 7f28f0ff9640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28e00668d0 0x7f28e0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:56.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.880+0000 7f28f0ff9640 1 -- 172.21.15.7:0/3214771118 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f28e4084980 con 0x7f28dc004d10 2023-11-28T18:47:56.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.880+0000 7f28f37fe640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28e00668d0 0x7f28e0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:56.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.880+0000 7f28f37fe640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28e00668d0 0x7f28e0068d90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f28f4157490 tx=0x7f28e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:56.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:56.884+0000 7f28f0ff9640 1 -- 172.21.15.7:0/3214771118 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f28e40516a0 con 0x7f28dc004d10 2023-11-28T18:47:57.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.072+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28f41e48b0 con 0x7f28dc004d10 2023-11-28T18:47:57.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.072+0000 7f28f0ff9640 1 -- 172.21.15.7:0/3214771118 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f28e4054cc0 con 0x7f28dc004d10 2023-11-28T18:47:57.077 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:57.078 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:57.079 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:57.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28e00668d0 msgr2=0x7f28e0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:57.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28e00668d0 0x7f28e0068d90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f28f4157490 tx=0x7f28e8008040 comp rx=0 tx=0).stop 2023-11-28T18:47:57.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 msgr2=0x7f28f4146490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:57.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28f4146490 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f28e4001210 tx=0x7f28e400eca0 comp rx=0 tx=0).stop 2023-11-28T18:47:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 shutdown_connections 2023-11-28T18:47:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f28e00668d0 0x7f28e0068d90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28f4148310 0x7f28f4148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f28f40cafe0 0x7f28f41469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 --2- 172.21.15.7:0/3214771118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc004d10 0x7f28f4146490 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 >> 172.21.15.7:0/3214771118 conn(0x7f28f40662e0 msgr2=0x7f28f40c45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:57.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 shutdown_connections 2023-11-28T18:47:57.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:57.076+0000 7f28fa8dd640 1 -- 172.21.15.7:0/3214771118 wait complete. 2023-11-28T18:47:57.098 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:47:59.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:47:59.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a594fe640 1 Processor -- start 2023-11-28T18:47:59.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a594fe640 1 -- start start 2023-11-28T18:47:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a594fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3a541503c0 0x7f3a540c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a594fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a540c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a594fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a540c3920 con 0x7f3a540c3bc0 2023-11-28T18:47:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a594fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a54157240 con 0x7f3a541503c0 2023-11-28T18:47:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a594fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540c2ce0 con 0x7f3a540c3490 2023-11-28T18:47:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a52ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3a541503c0 0x7f3a540c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:59.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a4bfff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f3a540c3bc0 0x7f3a5406d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:56802/0 (socket says 172.21.15.7:56802) 2023-11-28T18:47:59.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a537fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a540c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:59.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.252+0000 7f3a4bfff640 1 -- 172.21.15.7:0/3755433336 learned_addr learned my addr 172.21.15.7:0/3755433336 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:47:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a52ffd640 1 -- 172.21.15.7:0/3755433336 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 msgr2=0x7f3a540c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:47:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a52ffd640 1 --2- 172.21.15.7:0/3755433336 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a540c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a52ffd640 1 -- 172.21.15.7:0/3755433336 >> v1:172.21.15.193:6789/0 conn(0x7f3a540c3bc0 legacy=0x7f3a5406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a52ffd640 1 -- 172.21.15.7:0/3755433336 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a540c2f80 con 0x7f3a541503c0 2023-11-28T18:47:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a527fc640 1 -- 172.21.15.7:0/3755433336 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484230533 0 0) 0x7f3a3c001120 con 0x7f3a540c3bc0 2023-11-28T18:47:59.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a537fe640 1 --2- 172.21.15.7:0/3755433336 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a540c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:47:59.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a52ffd640 1 --2- 172.21.15.7:0/3755433336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 0x7f3a540c7c60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3a440097c0 tx=0x7f3a4401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9122c984412f6e38 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:59.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a527fc640 1 -- 172.21.15.7:0/3755433336 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a440a8970 con 0x7f3a541503c0 2023-11-28T18:47:59.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a527fc640 1 -- 172.21.15.7:0/3755433336 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a440a8ad0 con 0x7f3a541503c0 2023-11-28T18:47:59.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.256+0000 7f3a527fc640 1 -- 172.21.15.7:0/3755433336 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a440a8e60 con 0x7f3a541503c0 2023-11-28T18:47:59.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- 172.21.15.7:0/3755433336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 msgr2=0x7f3a540c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 --2- 172.21.15.7:0/3755433336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 0x7f3a540c7c60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3a440097c0 tx=0x7f3a4401ec70 comp rx=0 tx=0).stop 2023-11-28T18:47:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- 172.21.15.7:0/3755433336 shutdown_connections 2023-11-28T18:47:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 --2- 172.21.15.7:0/3755433336 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a540c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 --2- 172.21.15.7:0/3755433336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 0x7f3a540c7c60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- 172.21.15.7:0/3755433336 >> 172.21.15.7:0/3755433336 conn(0x7f3a540662e0 msgr2=0x7f3a540666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- 172.21.15.7:0/3755433336 shutdown_connections 2023-11-28T18:47:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- 172.21.15.7:0/3755433336 wait complete. 2023-11-28T18:47:59.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 Processor -- start 2023-11-28T18:47:59.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- start start 2023-11-28T18:47:59.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a5414edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:59.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a540c3bc0 0x7f3a541dea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:59.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 0x7f3a541ef460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:59.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541e66e0 con 0x7f3a540c3bc0 2023-11-28T18:47:59.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54153620 con 0x7f3a541503c0 2023-11-28T18:47:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a594fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541f31c0 con 0x7f3a540c3490 2023-11-28T18:47:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a4bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a540c3bc0 0x7f3a541dea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a4bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a540c3bc0 0x7f3a541dea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48638/0 (socket says 172.21.15.7:48638) 2023-11-28T18:47:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a4bfff640 1 -- 172.21.15.7:0/2784462822 learned_addr learned my addr 172.21.15.7:0/2784462822 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:47:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a52ffd640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a5414edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:59.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a537fe640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 0x7f3a541ef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a4bfff640 1 -- 172.21.15.7:0/2784462822 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 msgr2=0x7f3a5414edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a4bfff640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a5414edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a4bfff640 1 -- 172.21.15.7:0/2784462822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 msgr2=0x7f3a541ef460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a4bfff640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 0x7f3a541ef460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.260+0000 7f3a4bfff640 1 -- 172.21.15.7:0/2784462822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c00ab80 con 0x7f3a540c3bc0 2023-11-28T18:47:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a4bfff640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a540c3bc0 0x7f3a541dea00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c780 tx=0x7f3a3c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a4b7fe640 1 -- 172.21.15.7:0/2784462822 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0072d0 con 0x7f3a540c3bc0 2023-11-28T18:47:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a4b7fe640 1 -- 172.21.15.7:0/2784462822 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c007450 con 0x7f3a540c3bc0 2023-11-28T18:47:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a4b7fe640 1 -- 172.21.15.7:0/2784462822 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0047f0 con 0x7f3a540c3bc0 2023-11-28T18:47:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a44009420 con 0x7f3a540c3bc0 2023-11-28T18:47:59.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a541e4510 con 0x7f3a540c3bc0 2023-11-28T18:47:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a541f3330 con 0x7f3a540c3bc0 2023-11-28T18:47:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a4b7fe640 1 -- 172.21.15.7:0/2784462822 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3a3c004950 con 0x7f3a540c3bc0 2023-11-28T18:47:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a4b7fe640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3a34066900 0x7f3a34068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:47:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a4b7fe640 1 -- 172.21.15.7:0/2784462822 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f3a3c0838a0 con 0x7f3a540c3bc0 2023-11-28T18:47:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.264+0000 7f3a52ffd640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3a34066900 0x7f3a34068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:47:59.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.268+0000 7f3a52ffd640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3a34066900 0x7f3a34068dc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3a440b2000 tx=0x7f3a44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:47:59.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.268+0000 7f3a4b7fe640 1 -- 172.21.15.7:0/2784462822 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3a3c050700 con 0x7f3a540c3bc0 2023-11-28T18:47:59.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.460+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a540c3920 con 0x7f3a540c3bc0 2023-11-28T18:47:59.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.460+0000 7f3a4b7fe640 1 -- 172.21.15.7:0/2784462822 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f3a540c3920 con 0x7f3a540c3bc0 2023-11-28T18:47:59.463 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:47:59.463 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:47:59.464 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:47:59.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3a34066900 msgr2=0x7f3a34068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:59.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3a34066900 0x7f3a34068dc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3a440b2000 tx=0x7f3a44008040 comp rx=0 tx=0).stop 2023-11-28T18:47:59.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a540c3bc0 msgr2=0x7f3a541dea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:47:59.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a540c3bc0 0x7f3a541dea00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c780 tx=0x7f3a3c00cc40 comp rx=0 tx=0).stop 2023-11-28T18:47:59.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 shutdown_connections 2023-11-28T18:47:59.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f3a34066900 0x7f3a34068dc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a541503c0 0x7f3a541ef460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a540c3bc0 0x7f3a541dea00 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 --2- 172.21.15.7:0/2784462822 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3a540c3490 0x7f3a5414edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:47:59.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 >> 172.21.15.7:0/2784462822 conn(0x7f3a540662e0 msgr2=0x7f3a541e17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:47:59.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 shutdown_connections 2023-11-28T18:47:59.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:47:59.464+0000 7f3a594fe640 1 -- 172.21.15.7:0/2784462822 wait complete. 2023-11-28T18:47:59.485 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:48:01.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~162s 2023-11-28T18:48:01.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:01.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.640+0000 7f9c1c8e3640 1 Processor -- start 2023-11-28T18:48:01.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.640+0000 7f9c1c8e3640 1 -- start start 2023-11-28T18:48:01.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c1c8e3640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9c18073070 0x7f9c18074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:01.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c1c8e3640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c180c8490 con 0x7f9c18072950 2023-11-28T18:48:01.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c1c8e3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c180c81f0 con 0x7f9c1806c920 2023-11-28T18:48:01.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c1c8e3640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1815ab00 con 0x7f9c18073070 2023-11-28T18:48:01.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c1659c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9c18072950 0x7f9c180c19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39356/0 (socket says 172.21.15.7:39356) 2023-11-28T18:48:01.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c1659c640 1 -- 172.21.15.7:0/495139129 learned_addr learned my addr 172.21.15.7:0/495139129 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:01.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c0ffff640 1 --2- 172.21.15.7:0/495139129 >> v2:172.21.15.193:3300/0 conn(0x7f9c18073070 0x7f9c18074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:01.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c15d9b640 1 -- 172.21.15.7:0/495139129 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931597118 0 0) 0x7f9c180c8490 con 0x7f9c18072950 2023-11-28T18:48:01.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c15d9b640 1 -- 172.21.15.7:0/495139129 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c04003530 con 0x7f9c18072950 2023-11-28T18:48:01.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c0ffff640 1 -- 172.21.15.7:0/495139129 >> v1:172.21.15.193:6789/0 conn(0x7f9c1806c920 legacy=0x7f9c181647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:01.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c0ffff640 1 -- 172.21.15.7:0/495139129 >> v1:172.21.15.7:6789/0 conn(0x7f9c18072950 legacy=0x7f9c180c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:01.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.644+0000 7f9c0ffff640 1 -- 172.21.15.7:0/495139129 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1815ade0 con 0x7f9c18073070 2023-11-28T18:48:01.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c0ffff640 1 --2- 172.21.15.7:0/495139129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 0x7f9c18074030 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9c000097c0 tx=0x7f9c0001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7733f33a0a7cbf48 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:01.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c15d9b640 1 -- 172.21.15.7:0/495139129 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c000ab600 con 0x7f9c18073070 2023-11-28T18:48:01.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c15d9b640 1 -- 172.21.15.7:0/495139129 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c000a7050 con 0x7f9c18073070 2023-11-28T18:48:01.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c15d9b640 1 -- 172.21.15.7:0/495139129 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c000ab980 con 0x7f9c18073070 2023-11-28T18:48:01.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/495139129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 msgr2=0x7f9c18074030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:01.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 --2- 172.21.15.7:0/495139129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 0x7f9c18074030 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9c000097c0 tx=0x7f9c0001c360 comp rx=0 tx=0).stop 2023-11-28T18:48:01.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/495139129 shutdown_connections 2023-11-28T18:48:01.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 --2- 172.21.15.7:0/495139129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 0x7f9c18074030 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:01.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/495139129 >> 172.21.15.7:0/495139129 conn(0x7f9c180cbf80 msgr2=0x7f9c180cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:01.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/495139129 shutdown_connections 2023-11-28T18:48:01.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/495139129 wait complete. 2023-11-28T18:48:01.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 Processor -- start 2023-11-28T18:48:01.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.648+0000 7f9c1c8e3640 1 -- start start 2023-11-28T18:48:01.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1806c920 0x7f9c18148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:01.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c18072950 0x7f9c18165830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:01.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 0x7f9c18149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:01.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c180aedb0 con 0x7f9c1806c920 2023-11-28T18:48:01.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c180af690 con 0x7f9c18073070 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c181487b0 con 0x7f9c18072950 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1806c920 0x7f9c18148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1806c920 0x7f9c18148d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48654/0 (socket says 172.21.15.7:48654) 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 -- 172.21.15.7:0/1215005853 learned_addr learned my addr 172.21.15.7:0/1215005853 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c0ffff640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c18072950 0x7f9c18165830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c16d9d640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 0x7f9c18149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 -- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c18072950 msgr2=0x7f9c18165830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c18072950 0x7f9c18165830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 -- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 msgr2=0x7f9c18149570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 0x7f9c18149570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 -- 172.21.15.7:0/1215005853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c00009420 con 0x7f9c1806c920 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c0ffff640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c18072950 0x7f9c18165830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c16d9d640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 0x7f9c18149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1659c640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1806c920 0x7f9c18148d70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c4d0 tx=0x7f9c0800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c0effd640 1 -- 172.21.15.7:0/1215005853 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08005cf0 con 0x7f9c1806c920 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c0effd640 1 -- 172.21.15.7:0/1215005853 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08005e50 con 0x7f9c1806c920 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c0effd640 1 -- 172.21.15.7:0/1215005853 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08007270 con 0x7f9c1806c920 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c18148a90 con 0x7f9c1806c920 2023-11-28T18:48:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1814b630 con 0x7f9c1806c920 2023-11-28T18:48:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.652+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1804e890 con 0x7f9c1806c920 2023-11-28T18:48:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.656+0000 7f9c0effd640 1 -- 172.21.15.7:0/1215005853 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9c080073d0 con 0x7f9c1806c920 2023-11-28T18:48:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.656+0000 7f9c0effd640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9bf80668d0 0x7f9bf8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.656+0000 7f9c0effd640 1 -- 172.21.15.7:0/1215005853 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f9c08083e00 con 0x7f9c1806c920 2023-11-28T18:48:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.656+0000 7f9c0ffff640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9bf80668d0 0x7f9bf8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:01.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.656+0000 7f9c0ffff640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9bf80668d0 0x7f9bf8068d90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9c000ad000 tx=0x7f9c00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:01.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.660+0000 7f9c0effd640 1 -- 172.21.15.7:0/1215005853 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9c08050be0 con 0x7f9c1806c920 2023-11-28T18:48:01.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.848+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c18072420 con 0x7f9c1806c920 2023-11-28T18:48:01.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.848+0000 7f9c0effd640 1 -- 172.21.15.7:0/1215005853 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f9c08017070 con 0x7f9c1806c920 2023-11-28T18:48:01.851 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:01.851 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:01.851 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:01.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.848+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9bf80668d0 msgr2=0x7f9bf8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.848+0000 7f9c1c8e3640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9bf80668d0 0x7f9bf8068d90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9c000ad000 tx=0x7f9c00008040 comp rx=0 tx=0).stop 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.848+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1806c920 msgr2=0x7f9c18148d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.848+0000 7f9c1c8e3640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1806c920 0x7f9c18148d70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c4d0 tx=0x7f9c0800c990 comp rx=0 tx=0).stop 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.848+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 shutdown_connections 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.852+0000 7f9c1c8e3640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9bf80668d0 0x7f9bf8068d90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.852+0000 7f9c1c8e3640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c18073070 0x7f9c18149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.852+0000 7f9c1c8e3640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9c18072950 0x7f9c18165830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.852+0000 7f9c1c8e3640 1 --2- 172.21.15.7:0/1215005853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1806c920 0x7f9c18148d70 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.852+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 >> 172.21.15.7:0/1215005853 conn(0x7f9c180cbf80 msgr2=0x7f9c180c9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.852+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 shutdown_connections 2023-11-28T18:48:01.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:01.852+0000 7f9c1c8e3640 1 -- 172.21.15.7:0/1215005853 wait complete. 2023-11-28T18:48:01.872 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:03.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02e0fc3640 1 Processor -- start 2023-11-28T18:48:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02e0fc3640 1 -- start start 2023-11-28T18:48:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02e0fc3640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f02dc150b30 0x7f02dc0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02e0fc3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02dc0c7980 0x7f02dc0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02e0fc3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc0c6140 con 0x7f02dc1575c0 2023-11-28T18:48:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02e0fc3640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02dc0c72b0 con 0x7f02dc0c7980 2023-11-28T18:48:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02e0fc3640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02dc070c00 con 0x7f02dc150b30 2023-11-28T18:48:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02dad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02dc0c7980 0x7f02dc0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02dad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02dc0c7980 0x7f02dc0c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40872/0 (socket says 172.21.15.7:40872) 2023-11-28T18:48:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02dad9d640 1 -- 172.21.15.7:0/995651006 learned_addr learned my addr 172.21.15.7:0/995651006 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.028+0000 7f02da59c640 1 --2- 172.21.15.7:0/995651006 >> v2:172.21.15.193:3300/0 conn(0x7f02dc150b30 0x7f02dc0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02dad9d640 1 -- 172.21.15.7:0/995651006 >> v2:172.21.15.193:3300/0 conn(0x7f02dc150b30 msgr2=0x7f02dc0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02dad9d640 1 --2- 172.21.15.7:0/995651006 >> v2:172.21.15.193:3300/0 conn(0x7f02dc150b30 0x7f02dc0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02dad9d640 1 -- 172.21.15.7:0/995651006 >> v1:172.21.15.193:6789/0 conn(0x7f02dc1575c0 legacy=0x7f02dc0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02dad9d640 1 -- 172.21.15.7:0/995651006 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc0c8310 con 0x7f02dc0c7980 2023-11-28T18:48:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02dad9d640 1 --2- 172.21.15.7:0/995651006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc0c7980 0x7f02dc0c7dd0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f02dc06ec80 tx=0x7f02d401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7dee680b69d972 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02d959a640 1 -- 172.21.15.7:0/995651006 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40af400 con 0x7f02dc0c7980 2023-11-28T18:48:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02d959a640 1 -- 172.21.15.7:0/995651006 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d40ab050 con 0x7f02dc0c7980 2023-11-28T18:48:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02d959a640 1 -- 172.21.15.7:0/995651006 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40af780 con 0x7f02dc0c7980 2023-11-28T18:48:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02e0fc3640 1 -- 172.21.15.7:0/995651006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc0c7980 msgr2=0x7f02dc0c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/995651006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc0c7980 0x7f02dc0c7dd0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f02dc06ec80 tx=0x7f02d401f1c0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02e0fc3640 1 -- 172.21.15.7:0/995651006 shutdown_connections 2023-11-28T18:48:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/995651006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc0c7980 0x7f02dc0c7dd0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/995651006 >> v2:172.21.15.193:3300/0 conn(0x7f02dc150b30 0x7f02dc0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.032+0000 7f02e0fc3640 1 -- 172.21.15.7:0/995651006 >> 172.21.15.7:0/995651006 conn(0x7f02dc0662e0 msgr2=0x7f02dc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 -- 172.21.15.7:0/995651006 shutdown_connections 2023-11-28T18:48:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 -- 172.21.15.7:0/995651006 wait complete. 2023-11-28T18:48:04.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 Processor -- start 2023-11-28T18:48:04.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 -- start start 2023-11-28T18:48:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f02dc0c7980 0x7f02dc072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02dc150b30 0x7f02dc14f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc1575c0 0x7f02dc1f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc1e3bc0 con 0x7f02dc1575c0 2023-11-28T18:48:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc073380 con 0x7f02dc150b30 2023-11-28T18:48:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02e0fc3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc0c2af0 con 0x7f02dc0c7980 2023-11-28T18:48:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc1575c0 0x7f02dc1f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc1575c0 0x7f02dc1f2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40884/0 (socket says 172.21.15.7:40884) 2023-11-28T18:48:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 -- 172.21.15.7:0/2921651259 learned_addr learned my addr 172.21.15.7:0/2921651259 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02d9d9b640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02dc150b30 0x7f02dc14f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02da59c640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f02dc0c7980 0x7f02dc072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 -- 172.21.15.7:0/2921651259 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f02dc0c7980 msgr2=0x7f02dc072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f02dc0c7980 0x7f02dc072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 -- 172.21.15.7:0/2921651259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02dc150b30 msgr2=0x7f02dc14f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02dc150b30 0x7f02dc14f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 -- 172.21.15.7:0/2921651259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d4009b30 con 0x7f02dc1575c0 2023-11-28T18:48:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.036+0000 7f02dad9d640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc1575c0 0x7f02dc1f2f30 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f02d401cdf0 tx=0x7f02d401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02caffd640 1 -- 172.21.15.7:0/2921651259 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40afd10 con 0x7f02dc1575c0 2023-11-28T18:48:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02caffd640 1 -- 172.21.15.7:0/2921651259 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d40afe90 con 0x7f02dc1575c0 2023-11-28T18:48:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02caffd640 1 -- 172.21.15.7:0/2921651259 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40b8670 con 0x7f02dc1575c0 2023-11-28T18:48:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc1e3040 con 0x7f02dc1575c0 2023-11-28T18:48:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02dc1e3600 con 0x7f02dc1575c0 2023-11-28T18:48:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02dc1f2c50 con 0x7f02dc1575c0 2023-11-28T18:48:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02caffd640 1 -- 172.21.15.7:0/2921651259 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f02d40b6040 con 0x7f02dc1575c0 2023-11-28T18:48:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02caffd640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f02bc0668d0 0x7f02bc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02caffd640 1 -- 172.21.15.7:0/2921651259 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f02d4122270 con 0x7f02dc1575c0 2023-11-28T18:48:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.040+0000 7f02da59c640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f02bc0668d0 0x7f02bc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.044+0000 7f02da59c640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f02bc0668d0 0x7f02bc068d90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009b60 tx=0x7f02cc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.048+0000 7f02caffd640 1 -- 172.21.15.7:0/2921651259 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f02dc1f2c50 con 0x7f02dc1575c0 2023-11-28T18:48:04.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.240+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02dc1f2c50 con 0x7f02dc1575c0 2023-11-28T18:48:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.240+0000 7f02caffd640 1 -- 172.21.15.7:0/2921651259 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f02dc1f2c50 con 0x7f02dc1575c0 2023-11-28T18:48:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:04.243 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:04.243 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:04.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.240+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f02bc0668d0 msgr2=0x7f02bc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:04.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.240+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f02bc0668d0 0x7f02bc068d90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009b60 tx=0x7f02cc009190 comp rx=0 tx=0).stop 2023-11-28T18:48:04.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.240+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc1575c0 msgr2=0x7f02dc1f2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.240+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc1575c0 0x7f02dc1f2f30 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f02d401cdf0 tx=0x7f02d401bf60 comp rx=0 tx=0).stop 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.244+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 shutdown_connections 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.244+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f02bc0668d0 0x7f02bc068d90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.244+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02dc1575c0 0x7f02dc1f2f30 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.244+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02dc150b30 0x7f02dc14f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.244+0000 7f02e0fc3640 1 --2- 172.21.15.7:0/2921651259 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f02dc0c7980 0x7f02dc072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.244+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 >> 172.21.15.7:0/2921651259 conn(0x7f02dc0662e0 msgr2=0x7f02dc14fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.244+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 shutdown_connections 2023-11-28T18:48:04.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:04.244+0000 7f02e0fc3640 1 -- 172.21.15.7:0/2921651259 wait complete. 2023-11-28T18:48:04.265 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:06.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:06.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22acfe640 1 Processor -- start 2023-11-28T18:48:06.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22acfe640 1 -- start start 2023-11-28T18:48:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22acfe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fb2240cbb70 0x7fb2240cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22acfe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2240cc810 0x7fb224072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22acfe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2240ca920 con 0x7fb2241542e0 2023-11-28T18:48:06.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22acfe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb224071d50 con 0x7fb2240cbb70 2023-11-28T18:48:06.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22acfe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb224072030 con 0x7fb2240cc810 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22929b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2240cc810 0x7fb224072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb223fff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fb2240cbb70 0x7fb2240cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22929b640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2240cc810 0x7fb224072450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:34466/0 (socket says 172.21.15.7:34466) 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22929b640 1 -- 172.21.15.7:0/549167341 learned_addr learned my addr 172.21.15.7:0/549167341 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb228a9a640 1 --1- 172.21.15.7:0/549167341 >> v1:172.21.15.193:6789/0 conn(0x7fb2241542e0 0x7fb22415e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:53366/0 (socket says 172.21.15.7:53366) 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb2237fe640 1 -- 172.21.15.7:0/549167341 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826959069 0 0) 0x7fb2240ca920 con 0x7fb2241542e0 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb2237fe640 1 -- 172.21.15.7:0/549167341 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb214003530 con 0x7fb2241542e0 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22929b640 1 -- 172.21.15.7:0/549167341 >> v2:172.21.15.193:3300/0 conn(0x7fb2240cbb70 msgr2=0x7fb2240cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22929b640 1 --2- 172.21.15.7:0/549167341 >> v2:172.21.15.193:3300/0 conn(0x7fb2240cbb70 0x7fb2240cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22929b640 1 -- 172.21.15.7:0/549167341 >> v1:172.21.15.193:6789/0 conn(0x7fb2241542e0 legacy=0x7fb22415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb22929b640 1 -- 172.21.15.7:0/549167341 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2240c4cd0 con 0x7fb2240cc810 2023-11-28T18:48:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.420+0000 7fb223fff640 1 --2- 172.21.15.7:0/549167341 >> v2:172.21.15.193:3300/0 conn(0x7fb2240cbb70 0x7fb2240cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:48:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22929b640 1 --2- 172.21.15.7:0/549167341 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2240cc810 0x7fb224072450 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009e50 tx=0x7fb21c01ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=363a209f6b7fc787 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb2237fe640 1 -- 172.21.15.7:0/549167341 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c0a57f0 con 0x7fb2240cc810 2023-11-28T18:48:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb2237fe640 1 -- 172.21.15.7:0/549167341 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21c0a5950 con 0x7fb2240cc810 2023-11-28T18:48:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb2237fe640 1 -- 172.21.15.7:0/549167341 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c0a5ce0 con 0x7fb2240cc810 2023-11-28T18:48:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22acfe640 1 -- 172.21.15.7:0/549167341 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2240cc810 msgr2=0x7fb224072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22acfe640 1 --2- 172.21.15.7:0/549167341 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2240cc810 0x7fb224072450 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009e50 tx=0x7fb21c01ca10 comp rx=0 tx=0).stop 2023-11-28T18:48:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22acfe640 1 -- 172.21.15.7:0/549167341 shutdown_connections 2023-11-28T18:48:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22acfe640 1 --2- 172.21.15.7:0/549167341 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2240cc810 0x7fb224072450 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22acfe640 1 --2- 172.21.15.7:0/549167341 >> v2:172.21.15.193:3300/0 conn(0x7fb2240cbb70 0x7fb2240cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22acfe640 1 -- 172.21.15.7:0/549167341 >> 172.21.15.7:0/549167341 conn(0x7fb2240717e0 msgr2=0x7fb2240662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22acfe640 1 -- 172.21.15.7:0/549167341 shutdown_connections 2023-11-28T18:48:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.424+0000 7fb22acfe640 1 -- 172.21.15.7:0/549167341 wait complete. 2023-11-28T18:48:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 Processor -- start 2023-11-28T18:48:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 -- start start 2023-11-28T18:48:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2240cbb70 0x7fb2241536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2240cc810 0x7fb2240cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2241542e0 0x7fb224143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240c7520 con 0x7fb2240cc810 2023-11-28T18:48:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2241f3400 con 0x7fb2240cbb70 2023-11-28T18:48:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240c6950 con 0x7fb2241542e0 2023-11-28T18:48:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2240cc810 0x7fb2240cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2240cc810 0x7fb2240cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40896/0 (socket says 172.21.15.7:40896) 2023-11-28T18:48:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 -- 172.21.15.7:0/266468448 learned_addr learned my addr 172.21.15.7:0/266468448 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb228a9a640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2240cbb70 0x7fb2241536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22929b640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2241542e0 0x7fb224143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 -- 172.21.15.7:0/266468448 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2241542e0 msgr2=0x7fb224143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2241542e0 0x7fb224143930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 -- 172.21.15.7:0/266468448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2240cbb70 msgr2=0x7fb2241536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2240cbb70 0x7fb2241536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 -- 172.21.15.7:0/266468448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210009420 con 0x7fb2240cc810 2023-11-28T18:48:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb223fff640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2240cc810 0x7fb2240cef90 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb21000ebe0 tx=0x7fb21000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb2217fa640 1 -- 172.21.15.7:0/266468448 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21000eee0 con 0x7fb2240cc810 2023-11-28T18:48:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb2217fa640 1 -- 172.21.15.7:0/266468448 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21000cc30 con 0x7fb2240cc810 2023-11-28T18:48:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c009b30 con 0x7fb2240cc810 2023-11-28T18:48:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb2217fa640 1 -- 172.21.15.7:0/266468448 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210013470 con 0x7fb2240cc810 2023-11-28T18:48:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.428+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2240c6ed0 con 0x7fb2240cc810 2023-11-28T18:48:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.432+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22406c670 con 0x7fb2240cc810 2023-11-28T18:48:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.432+0000 7fb2217fa640 1 -- 172.21.15.7:0/266468448 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb2100135d0 con 0x7fb2240cc810 2023-11-28T18:48:06.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.432+0000 7fb2217fa640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb2080668d0 0x7fb208068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:06.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.432+0000 7fb2217fa640 1 -- 172.21.15.7:0/266468448 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fb210084b30 con 0x7fb2240cc810 2023-11-28T18:48:06.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.432+0000 7fb228a9a640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb2080668d0 0x7fb208068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:06.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.432+0000 7fb228a9a640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb2080668d0 0x7fb208068d90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb2180011f0 tx=0x7fb218007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:06.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.436+0000 7fb2217fa640 1 -- 172.21.15.7:0/266468448 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb210051900 con 0x7fb2240cc810 2023-11-28T18:48:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.624+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb22404e890 con 0x7fb2240cc810 2023-11-28T18:48:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.624+0000 7fb2217fa640 1 -- 172.21.15.7:0/266468448 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fb210054f20 con 0x7fb2240cc810 2023-11-28T18:48:06.628 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:06.629 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:06.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb2080668d0 msgr2=0x7fb208068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:06.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb2080668d0 0x7fb208068d90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb2180011f0 tx=0x7fb218007d40 comp rx=0 tx=0).stop 2023-11-28T18:48:06.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2240cc810 msgr2=0x7fb2240cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:06.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2240cc810 0x7fb2240cef90 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb21000ebe0 tx=0x7fb21000c520 comp rx=0 tx=0).stop 2023-11-28T18:48:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 shutdown_connections 2023-11-28T18:48:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb2080668d0 0x7fb208068d90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb2241542e0 0x7fb224143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2240cc810 0x7fb2240cef90 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 --2- 172.21.15.7:0/266468448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2240cbb70 0x7fb2241536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 >> 172.21.15.7:0/266468448 conn(0x7fb2240717e0 msgr2=0x7fb224158a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 shutdown_connections 2023-11-28T18:48:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:06.628+0000 7fb22acfe640 1 -- 172.21.15.7:0/266468448 wait complete. 2023-11-28T18:48:06.650 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:07.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~168s 2023-11-28T18:48:08.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:08.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f805689c640 1 Processor -- start 2023-11-28T18:48:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f805689c640 1 -- start start 2023-11-28T18:48:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f805689c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80500c24f0 0x7f80501575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f805689c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f80500c7980 0x7f80500c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f805689c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80500c6140 con 0x7f8050150b30 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f805689c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80500c72b0 con 0x7f80500c24f0 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f805689c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8050070c00 con 0x7f80500c7980 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f804f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80500c24f0 0x7f80501575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f804f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80500c24f0 0x7f80501575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40908/0 (socket says 172.21.15.7:40908) 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.804+0000 7f804f7fe640 1 -- 172.21.15.7:0/380764756 learned_addr learned my addr 172.21.15.7:0/380764756 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f8054e39640 1 --2- 172.21.15.7:0/380764756 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f80500c7980 0x7f80500c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f804f7fe640 1 -- 172.21.15.7:0/380764756 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f80500c7980 msgr2=0x7f80500c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:08.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f804f7fe640 1 --2- 172.21.15.7:0/380764756 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f80500c7980 0x7f80500c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:08.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f804f7fe640 1 -- 172.21.15.7:0/380764756 >> v1:172.21.15.193:6789/0 conn(0x7f8050150b30 legacy=0x7f80500c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:08.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f804f7fe640 1 -- 172.21.15.7:0/380764756 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8050157b00 con 0x7f80500c24f0 2023-11-28T18:48:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f804f7fe640 1 --2- 172.21.15.7:0/380764756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f80501575c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8044009870 tx=0x7f804401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfa8e0e011ff31a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f804effd640 1 -- 172.21.15.7:0/380764756 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80440a8a30 con 0x7f80500c24f0 2023-11-28T18:48:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f804effd640 1 -- 172.21.15.7:0/380764756 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80440a8b90 con 0x7f80500c24f0 2023-11-28T18:48:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.808+0000 7f804effd640 1 -- 172.21.15.7:0/380764756 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80440b0440 con 0x7f80500c24f0 2023-11-28T18:48:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- 172.21.15.7:0/380764756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 msgr2=0x7f80501575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 --2- 172.21.15.7:0/380764756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f80501575c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8044009870 tx=0x7f804401ed30 comp rx=0 tx=0).stop 2023-11-28T18:48:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- 172.21.15.7:0/380764756 shutdown_connections 2023-11-28T18:48:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 --2- 172.21.15.7:0/380764756 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f80500c7980 0x7f80500c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 --2- 172.21.15.7:0/380764756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f80501575c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- 172.21.15.7:0/380764756 >> 172.21.15.7:0/380764756 conn(0x7f80500662e0 msgr2=0x7f80500666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- 172.21.15.7:0/380764756 shutdown_connections 2023-11-28T18:48:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- 172.21.15.7:0/380764756 wait complete. 2023-11-28T18:48:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 Processor -- start 2023-11-28T18:48:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- start start 2023-11-28T18:48:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f8050153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80500c7980 0x7f8050149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8050150b30 0x7f8050155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8050150200 con 0x7f80500c24f0 2023-11-28T18:48:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805014d4c0 con 0x7f80500c7980 2023-11-28T18:48:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f805689c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8050154540 con 0x7f8050150b30 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f8050153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f8050153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40920/0 (socket says 172.21.15.7:40920) 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804ffff640 1 -- 172.21.15.7:0/1297641224 learned_addr learned my addr 172.21.15.7:0/1297641224 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804f7fe640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80500c7980 0x7f8050149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f8054e39640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8050150b30 0x7f8050155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804ffff640 1 -- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8050150b30 msgr2=0x7f8050155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804ffff640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8050150b30 0x7f8050155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804ffff640 1 -- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80500c7980 msgr2=0x7f8050149d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804ffff640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80500c7980 0x7f8050149d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804ffff640 1 -- 172.21.15.7:0/1297641224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80440094d0 con 0x7f80500c24f0 2023-11-28T18:48:08.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.812+0000 7f804f7fe640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80500c7980 0x7f8050149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:48:08.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f804ffff640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f8050153eb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f804000c3f0 tx=0x7f804000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:08.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f804cff9640 1 -- 172.21.15.7:0/1297641224 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80400052d0 con 0x7f80500c24f0 2023-11-28T18:48:08.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f804cff9640 1 -- 172.21.15.7:0/1297641224 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8040005430 con 0x7f80500c24f0 2023-11-28T18:48:08.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80501ed540 con 0x7f80500c24f0 2023-11-28T18:48:08.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f804cff9640 1 -- 172.21.15.7:0/1297641224 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8040003840 con 0x7f80500c24f0 2023-11-28T18:48:08.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80501edb00 con 0x7f80500c24f0 2023-11-28T18:48:08.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805004e890 con 0x7f80500c24f0 2023-11-28T18:48:08.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f804cff9640 1 -- 172.21.15.7:0/1297641224 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f80400039a0 con 0x7f80500c24f0 2023-11-28T18:48:08.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f804cff9640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f803c0668d0 0x7f803c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:08.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f804cff9640 1 -- 172.21.15.7:0/1297641224 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f8040083e40 con 0x7f80500c24f0 2023-11-28T18:48:08.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.816+0000 7f804f7fe640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f803c0668d0 0x7f803c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:08.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.820+0000 7f804f7fe640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f803c0668d0 0x7f803c068d90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f80440b2000 tx=0x7f8044008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:08.820+0000 7f804cff9640 1 -- 172.21.15.7:0/1297641224 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8040050bf0 con 0x7f80500c24f0 2023-11-28T18:48:09.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80501e2a60 con 0x7f80500c24f0 2023-11-28T18:48:09.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f804cff9640 1 -- 172.21.15.7:0/1297641224 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f8040017090 con 0x7f80500c24f0 2023-11-28T18:48:09.018 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:09.018 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:09.019 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f803c0668d0 msgr2=0x7f803c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f803c0668d0 0x7f803c068d90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f80440b2000 tx=0x7f8044008040 comp rx=0 tx=0).stop 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 msgr2=0x7f8050153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f8050153eb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f804000c3f0 tx=0x7f804000c8b0 comp rx=0 tx=0).stop 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 shutdown_connections 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f803c0668d0 0x7f803c068d90 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8050150b30 0x7f8050155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80500c7980 0x7f8050149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 --2- 172.21.15.7:0/1297641224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80500c24f0 0x7f8050153eb0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 >> 172.21.15.7:0/1297641224 conn(0x7f80500662e0 msgr2=0x7f80500c3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 shutdown_connections 2023-11-28T18:48:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:09.016+0000 7f805689c640 1 -- 172.21.15.7:0/1297641224 wait complete. 2023-11-28T18:48:09.040 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:11.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.192+0000 7f6e03032640 1 Processor -- start 2023-11-28T18:48:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.192+0000 7f6e03032640 1 -- start start 2023-11-28T18:48:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e03032640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6dfc06e560 0x7f6dfc15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e03032640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6dfc157b90 0x7f6dfc0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e03032640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc071e60 con 0x7f6dfc151cd0 2023-11-28T18:48:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e03032640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dfc070730 con 0x7f6dfc157b90 2023-11-28T18:48:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e03032640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dfc070930 con 0x7f6dfc06e560 2023-11-28T18:48:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e015cf640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6dfc157b90 0x7f6dfc0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e00dce640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6dfc151cd0 0x7f6dfc0c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39758/0 (socket says 172.21.15.7:39758) 2023-11-28T18:48:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e00dce640 1 -- 172.21.15.7:0/3073311876 learned_addr learned my addr 172.21.15.7:0/3073311876 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6dfbfff640 1 --2- 172.21.15.7:0/3073311876 >> v2:172.21.15.193:3300/0 conn(0x7f6dfc06e560 0x7f6dfc15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e015cf640 1 -- 172.21.15.7:0/3073311876 >> v2:172.21.15.193:3300/0 conn(0x7f6dfc06e560 msgr2=0x7f6dfc15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e015cf640 1 --2- 172.21.15.7:0/3073311876 >> v2:172.21.15.193:3300/0 conn(0x7f6dfc06e560 0x7f6dfc15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e015cf640 1 -- 172.21.15.7:0/3073311876 >> v1:172.21.15.7:6789/0 conn(0x7f6dfc151cd0 legacy=0x7f6dfc0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6e015cf640 1 -- 172.21.15.7:0/3073311876 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc157fa0 con 0x7f6dfc157b90 2023-11-28T18:48:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6dfb7fe640 1 -- 172.21.15.7:0/3073311876 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032969712 0 0) 0x7f6dfc071e60 con 0x7f6dfc151cd0 2023-11-28T18:48:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.196+0000 7f6dfbfff640 1 --2- 172.21.15.7:0/3073311876 >> v2:172.21.15.193:3300/0 conn(0x7f6dfc06e560 0x7f6dfc15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:48:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e015cf640 1 --2- 172.21.15.7:0/3073311876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc0701f0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0671b0 tx=0x7f6df401c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8704e6ce8988b93b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6dfb7fe640 1 -- 172.21.15.7:0/3073311876 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df40a5830 con 0x7f6dfc157b90 2023-11-28T18:48:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6dfb7fe640 1 -- 172.21.15.7:0/3073311876 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df40a5990 con 0x7f6dfc157b90 2023-11-28T18:48:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6dfb7fe640 1 -- 172.21.15.7:0/3073311876 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df40a5d20 con 0x7f6dfc157b90 2023-11-28T18:48:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 -- 172.21.15.7:0/3073311876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 msgr2=0x7f6dfc0701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 --2- 172.21.15.7:0/3073311876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc0701f0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0671b0 tx=0x7f6df401c880 comp rx=0 tx=0).stop 2023-11-28T18:48:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 -- 172.21.15.7:0/3073311876 shutdown_connections 2023-11-28T18:48:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 --2- 172.21.15.7:0/3073311876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc0701f0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 --2- 172.21.15.7:0/3073311876 >> v2:172.21.15.193:3300/0 conn(0x7f6dfc06e560 0x7f6dfc15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 -- 172.21.15.7:0/3073311876 >> 172.21.15.7:0/3073311876 conn(0x7f6dfc0662e0 msgr2=0x7f6dfc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 -- 172.21.15.7:0/3073311876 shutdown_connections 2023-11-28T18:48:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 -- 172.21.15.7:0/3073311876 wait complete. 2023-11-28T18:48:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 Processor -- start 2023-11-28T18:48:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.200+0000 7f6e03032640 1 -- start start 2023-11-28T18:48:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6dfc06e560 0x7f6dfc150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dfc151cd0 0x7f6dfc1f6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc151520 con 0x7f6dfc157b90 2023-11-28T18:48:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc1f3ad0 con 0x7f6dfc151cd0 2023-11-28T18:48:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc157520 con 0x7f6dfc06e560 2023-11-28T18:48:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc146ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40942/0 (socket says 172.21.15.7:40942) 2023-11-28T18:48:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 -- 172.21.15.7:0/437094827 learned_addr learned my addr 172.21.15.7:0/437094827 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e00dce640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6dfc06e560 0x7f6dfc150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6dfbfff640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dfc151cd0 0x7f6dfc1f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 -- 172.21.15.7:0/437094827 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6dfc06e560 msgr2=0x7f6dfc150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6dfc06e560 0x7f6dfc150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 -- 172.21.15.7:0/437094827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dfc151cd0 msgr2=0x7f6dfc1f6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dfc151cd0 0x7f6dfc1f6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 -- 172.21.15.7:0/437094827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df4009a70 con 0x7f6dfc157b90 2023-11-28T18:48:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e015cf640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc146ae0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6df4019490 tx=0x7f6df4019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6df97fa640 1 -- 172.21.15.7:0/437094827 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df40b9860 con 0x7f6dfc157b90 2023-11-28T18:48:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6df97fa640 1 -- 172.21.15.7:0/437094827 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df40b99c0 con 0x7f6dfc157b90 2023-11-28T18:48:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6df97fa640 1 -- 172.21.15.7:0/437094827 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df40b9d70 con 0x7f6dfc157b90 2023-11-28T18:48:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc1f6060 con 0x7f6dfc157b90 2023-11-28T18:48:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc156130 con 0x7f6dfc157b90 2023-11-28T18:48:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc04e890 con 0x7f6dfc157b90 2023-11-28T18:48:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.204+0000 7f6df97fa640 1 -- 172.21.15.7:0/437094827 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6df40ad440 con 0x7f6dfc157b90 2023-11-28T18:48:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.208+0000 7f6df97fa640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6de00668d0 0x7f6de0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.208+0000 7f6df97fa640 1 -- 172.21.15.7:0/437094827 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f6df411feb0 con 0x7f6dfc157b90 2023-11-28T18:48:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.208+0000 7f6e00dce640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6de00668d0 0x7f6de0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.208+0000 7f6e00dce640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6de00668d0 0x7f6de0068d90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6df0000f70 tx=0x7f6df0007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.212+0000 7f6df97fa640 1 -- 172.21.15.7:0/437094827 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6df40eccf0 con 0x7f6dfc157b90 2023-11-28T18:48:11.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.400+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dfc1555b0 con 0x7f6dfc157b90 2023-11-28T18:48:11.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.400+0000 7f6df97fa640 1 -- 172.21.15.7:0/437094827 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f6df40bd090 con 0x7f6dfc157b90 2023-11-28T18:48:11.405 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:11.405 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:11.405 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:11.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6de00668d0 msgr2=0x7f6de0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:11.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6de00668d0 0x7f6de0068d90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6df0000f70 tx=0x7f6df0007b60 comp rx=0 tx=0).stop 2023-11-28T18:48:11.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 msgr2=0x7f6dfc146ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:11.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc146ae0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6df4019490 tx=0x7f6df4019620 comp rx=0 tx=0).stop 2023-11-28T18:48:11.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 shutdown_connections 2023-11-28T18:48:11.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6de00668d0 0x7f6de0068d90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dfc157b90 0x7f6dfc146ae0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dfc151cd0 0x7f6dfc1f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 --2- 172.21.15.7:0/437094827 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6dfc06e560 0x7f6dfc150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:11.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 >> 172.21.15.7:0/437094827 conn(0x7f6dfc0662e0 msgr2=0x7f6dfc14ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:11.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 shutdown_connections 2023-11-28T18:48:11.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:11.404+0000 7f6e03032640 1 -- 172.21.15.7:0/437094827 wait complete. 2023-11-28T18:48:11.427 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:48:11.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:48:11.761 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:48:13.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:13.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.580+0000 7fb3f5ac9640 1 Processor -- start 2023-11-28T18:48:13.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.580+0000 7fb3f5ac9640 1 -- start start 2023-11-28T18:48:13.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.580+0000 7fb3f5ac9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb3f0150b30 0x7fb3f00c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:13.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.580+0000 7fb3f5ac9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f00c7980 0x7fb3f00c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:13.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.580+0000 7fb3f5ac9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f00c6140 con 0x7fb3f01575c0 2023-11-28T18:48:13.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.580+0000 7fb3f5ac9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f00c72b0 con 0x7fb3f0150b30 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.580+0000 7fb3f5ac9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0070c00 con 0x7fb3f00c7980 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3e7fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb3f01575c0 0x7fb3f00c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57598/0 (socket says 172.21.15.7:57598) 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3ef7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb3f0150b30 0x7fb3f00c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3e7fff640 1 -- 172.21.15.7:0/3785854988 learned_addr learned my addr 172.21.15.7:0/3785854988 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3effff640 1 --2- 172.21.15.7:0/3785854988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f00c7980 0x7fb3f00c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3ef7fe640 1 -- 172.21.15.7:0/3785854988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f00c7980 msgr2=0x7fb3f00c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3ef7fe640 1 --2- 172.21.15.7:0/3785854988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f00c7980 0x7fb3f00c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3ef7fe640 1 -- 172.21.15.7:0/3785854988 >> v1:172.21.15.7:6789/0 conn(0x7fb3f01575c0 legacy=0x7fb3f00c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:13.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3ef7fe640 1 -- 172.21.15.7:0/3785854988 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f00c8350 con 0x7fb3f0150b30 2023-11-28T18:48:13.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.584+0000 7fb3ef7fe640 1 --2- 172.21.15.7:0/3785854988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f0150b30 0x7fb3f00c24f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00097c0 tx=0x7fb3e001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7293b804b93e6a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:13.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3eeffd640 1 -- 172.21.15.7:0/3785854988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e00a8970 con 0x7fb3f0150b30 2023-11-28T18:48:13.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3eeffd640 1 -- 172.21.15.7:0/3785854988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e00a8ad0 con 0x7fb3f0150b30 2023-11-28T18:48:13.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3eeffd640 1 -- 172.21.15.7:0/3785854988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e00a8e60 con 0x7fb3f0150b30 2023-11-28T18:48:13.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/3785854988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f0150b30 msgr2=0x7fb3f00c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:13.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/3785854988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f0150b30 0x7fb3f00c24f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00097c0 tx=0x7fb3e001ec70 comp rx=0 tx=0).stop 2023-11-28T18:48:13.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/3785854988 shutdown_connections 2023-11-28T18:48:13.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/3785854988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f00c7980 0x7fb3f00c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/3785854988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f0150b30 0x7fb3f00c24f0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/3785854988 >> 172.21.15.7:0/3785854988 conn(0x7fb3f00662e0 msgr2=0x7fb3f00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:13.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/3785854988 shutdown_connections 2023-11-28T18:48:13.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/3785854988 wait complete. 2023-11-28T18:48:13.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 Processor -- start 2023-11-28T18:48:13.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.588+0000 7fb3f5ac9640 1 -- start start 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f00c7980 0x7fb3f01f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f0150b30 0x7fb3f0072260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f01575c0 0x7fb3f0155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f01556a0 con 0x7fb3f01575c0 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0154970 con 0x7fb3f00c7980 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0154c50 con 0x7fb3f0150b30 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f01575c0 0x7fb3f0155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f01575c0 0x7fb3f0155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38434/0 (socket says 172.21.15.7:38434) 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 -- 172.21.15.7:0/1556061686 learned_addr learned my addr 172.21.15.7:0/1556061686 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3e7fff640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f0150b30 0x7fb3f0072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3ef7fe640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f00c7980 0x7fb3f01f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 -- 172.21.15.7:0/1556061686 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f0150b30 msgr2=0x7fb3f0072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f0150b30 0x7fb3f0072260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 -- 172.21.15.7:0/1556061686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f00c7980 msgr2=0x7fb3f01f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f00c7980 0x7fb3f01f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 -- 172.21.15.7:0/1556061686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e0009420 con 0x7fb3f01575c0 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3effff640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f01575c0 0x7fb3f0155dd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8007e40 tx=0x7fb3e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3ecff9640 1 -- 172.21.15.7:0/1556061686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8014620 con 0x7fb3f01575c0 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3ecff9640 1 -- 172.21.15.7:0/1556061686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e800ee90 con 0x7fb3f01575c0 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3ecff9640 1 -- 172.21.15.7:0/1556061686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e80149e0 con 0x7fb3f01575c0 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f0154f30 con 0x7fb3f01575c0 2023-11-28T18:48:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f01558b0 con 0x7fb3f01575c0 2023-11-28T18:48:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f004e890 con 0x7fb3f01575c0 2023-11-28T18:48:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.592+0000 7fb3ecff9640 1 -- 172.21.15.7:0/1556061686 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb3e8014b40 con 0x7fb3f01575c0 2023-11-28T18:48:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.596+0000 7fb3ecff9640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3d00668d0 0x7fb3d0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.596+0000 7fb3ecff9640 1 -- 172.21.15.7:0/1556061686 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fb3e80851d0 con 0x7fb3f01575c0 2023-11-28T18:48:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.596+0000 7fb3ef7fe640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3d00668d0 0x7fb3d0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.596+0000 7fb3ef7fe640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3d00668d0 0x7fb3d0068d90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00b2000 tx=0x7fb3e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.600+0000 7fb3ecff9640 1 -- 172.21.15.7:0/1556061686 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb3e8051fa0 con 0x7fb3f01575c0 2023-11-28T18:48:13.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.788+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3f00c2d80 con 0x7fb3f01575c0 2023-11-28T18:48:13.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3ecff9640 1 -- 172.21.15.7:0/1556061686 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fb3e80555c0 con 0x7fb3f01575c0 2023-11-28T18:48:13.793 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:13.793 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:13.793 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:13.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3d00668d0 msgr2=0x7fb3d0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3d00668d0 0x7fb3d0068d90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00b2000 tx=0x7fb3e0008040 comp rx=0 tx=0).stop 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f01575c0 msgr2=0x7fb3f0155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f01575c0 0x7fb3f0155dd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8007e40 tx=0x7fb3e800e4b0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 shutdown_connections 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb3d00668d0 0x7fb3d0068d90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3f01575c0 0x7fb3f0155dd0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb3f0150b30 0x7fb3f0072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 --2- 172.21.15.7:0/1556061686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f00c7980 0x7fb3f01f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 >> 172.21.15.7:0/1556061686 conn(0x7fb3f00662e0 msgr2=0x7fb3f00cabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 shutdown_connections 2023-11-28T18:48:13.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:13.792+0000 7fb3f5ac9640 1 -- 172.21.15.7:0/1556061686 wait complete. 2023-11-28T18:48:13.815 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:14.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~175s 2023-11-28T18:48:15.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:15.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.968+0000 7fb7b9632640 1 Processor -- start 2023-11-28T18:48:15.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.968+0000 7fb7b9632640 1 -- start start 2023-11-28T18:48:15.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b9632640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb7b415a5b0 0x7fb7b40c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:15.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b9632640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b406a8a0 0x7fb7b40c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:15.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b9632640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b40c8430 con 0x7fb7b4154040 2023-11-28T18:48:15.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b9632640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b406ae50 con 0x7fb7b415a5b0 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b9632640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b406b110 con 0x7fb7b406a8a0 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b2ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb7b4154040 0x7fb7b4159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57610/0 (socket says 172.21.15.7:57610) 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b2ffd640 1 -- 172.21.15.7:0/416930887 learned_addr learned my addr 172.21.15.7:0/416930887 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7abfff640 1 --2- 172.21.15.7:0/416930887 >> v2:172.21.15.7:3300/0 conn(0x7fb7b415a5b0 0x7fb7b40c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b37fe640 1 --2- 172.21.15.7:0/416930887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b406a8a0 0x7fb7b40c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7abfff640 1 -- 172.21.15.7:0/416930887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b406a8a0 msgr2=0x7fb7b40c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7abfff640 1 --2- 172.21.15.7:0/416930887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b406a8a0 0x7fb7b40c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7abfff640 1 -- 172.21.15.7:0/416930887 >> v1:172.21.15.7:6789/0 conn(0x7fb7b4154040 legacy=0x7fb7b4159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7abfff640 1 -- 172.21.15.7:0/416930887 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b40c7820 con 0x7fb7b415a5b0 2023-11-28T18:48:15.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7b27fc640 1 -- 172.21.15.7:0/416930887 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512973856 0 0) 0x7fb7b40c8430 con 0x7fb7b4154040 2023-11-28T18:48:15.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.972+0000 7fb7abfff640 1 --2- 172.21.15.7:0/416930887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b40c89b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0097f0 tx=0x7fb79c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dce8f5a51132d13 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:15.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b27fc640 1 -- 172.21.15.7:0/416930887 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0a8970 con 0x7fb7b415a5b0 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b27fc640 1 -- 172.21.15.7:0/416930887 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c0a8ad0 con 0x7fb7b415a5b0 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b27fc640 1 -- 172.21.15.7:0/416930887 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0a8e60 con 0x7fb7b415a5b0 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- 172.21.15.7:0/416930887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 msgr2=0x7fb7b40c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 --2- 172.21.15.7:0/416930887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b40c89b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0097f0 tx=0x7fb79c01ec70 comp rx=0 tx=0).stop 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- 172.21.15.7:0/416930887 shutdown_connections 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 --2- 172.21.15.7:0/416930887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b406a8a0 0x7fb7b40c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 --2- 172.21.15.7:0/416930887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b40c89b0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- 172.21.15.7:0/416930887 >> 172.21.15.7:0/416930887 conn(0x7fb7b406d4d0 msgr2=0x7fb7b406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- 172.21.15.7:0/416930887 shutdown_connections 2023-11-28T18:48:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- 172.21.15.7:0/416930887 wait complete. 2023-11-28T18:48:15.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 Processor -- start 2023-11-28T18:48:15.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- start start 2023-11-28T18:48:15.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b406a8a0 0x7fb7b41f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b4154040 0x7fb7b41f0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b414fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b415dbb0 con 0x7fb7b415a5b0 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b41572f0 con 0x7fb7b406a8a0 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b9632640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b4156d30 con 0x7fb7b4154040 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b414fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b414fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38454/0 (socket says 172.21.15.7:38454) 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.976+0000 7fb7b37fe640 1 -- 172.21.15.7:0/3636732093 learned_addr learned my addr 172.21.15.7:0/3636732093 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7abfff640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b4154040 0x7fb7b41f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b2ffd640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b406a8a0 0x7fb7b41f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b37fe640 1 -- 172.21.15.7:0/3636732093 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b4154040 msgr2=0x7fb7b41f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b37fe640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b4154040 0x7fb7b41f0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b37fe640 1 -- 172.21.15.7:0/3636732093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b406a8a0 msgr2=0x7fb7b41f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b37fe640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b406a8a0 0x7fb7b41f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b37fe640 1 -- 172.21.15.7:0/3636732093 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009420 con 0x7fb7b415a5b0 2023-11-28T18:48:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b37fe640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b414fb80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac007e60 tx=0x7fb7ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:15.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7ab7fe640 1 -- 172.21.15.7:0/3636732093 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac014420 con 0x7fb7b415a5b0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7ab7fe640 1 -- 172.21.15.7:0/3636732093 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ac0145a0 con 0x7fb7b415a5b0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b4157010 con 0x7fb7b415a5b0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7ab7fe640 1 -- 172.21.15.7:0/3636732093 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac014950 con 0x7fb7b415a5b0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b41f3270 con 0x7fb7b415a5b0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b404e890 con 0x7fb7b415a5b0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7ab7fe640 1 -- 172.21.15.7:0/3636732093 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb7ac014ab0 con 0x7fb7b415a5b0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7ab7fe640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7940668d0 0x7fb794068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.980+0000 7fb7ab7fe640 1 -- 172.21.15.7:0/3636732093 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fb7ac0850b0 con 0x7fb7b415a5b0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.984+0000 7fb7b2ffd640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7940668d0 0x7fb794068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.984+0000 7fb7b2ffd640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7940668d0 0x7fb794068d90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007fd0 tx=0x7fb7a4007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:15.984+0000 7fb7ab7fe640 1 -- 172.21.15.7:0/3636732093 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb7ac051f00 con 0x7fb7b415a5b0 2023-11-28T18:48:16.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.172+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7b4153a00 con 0x7fb7b415a5b0 2023-11-28T18:48:16.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.172+0000 7fb7ab7fe640 1 -- 172.21.15.7:0/3636732093 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fb7ac055520 con 0x7fb7b415a5b0 2023-11-28T18:48:16.178 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:16.178 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7940668d0 msgr2=0x7fb794068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7940668d0 0x7fb794068d90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007fd0 tx=0x7fb7a4007d60 comp rx=0 tx=0).stop 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 msgr2=0x7fb7b414fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b414fb80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac007e60 tx=0x7fb7ac00e4b0 comp rx=0 tx=0).stop 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 shutdown_connections 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fb7940668d0 0x7fb794068d90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b415a5b0 0x7fb7b414fb80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fb7b4154040 0x7fb7b41f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 --2- 172.21.15.7:0/3636732093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b406a8a0 0x7fb7b41f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:16.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 >> 172.21.15.7:0/3636732093 conn(0x7fb7b406d4d0 msgr2=0x7fb7b41593f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:16.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 shutdown_connections 2023-11-28T18:48:16.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:16.176+0000 7fb7b9632640 1 -- 172.21.15.7:0/3636732093 wait complete. 2023-11-28T18:48:16.198 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:18.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:18.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896f09a640 1 Processor -- start 2023-11-28T18:48:18.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896f09a640 1 -- start start 2023-11-28T18:48:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896f09a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896f09a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f89680cf6c0 0x7f89680cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896f09a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896815e380 con 0x7f89681540e0 2023-11-28T18:48:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896f09a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896815ade0 con 0x7f89680cf6c0 2023-11-28T18:48:18.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896f09a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896815d7e0 con 0x7f89680ca4d0 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896d637640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f89680cf6c0 0x7f89680cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f8967fff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896ce36640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f89681540e0 0x7f896815dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:41760/0 (socket says 172.21.15.7:41760) 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896ce36640 1 -- 172.21.15.7:0/89102406 learned_addr learned my addr 172.21.15.7:0/89102406 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f89677fe640 1 -- 172.21.15.7:0/89102406 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573336002 0 0) 0x7f895c001120 con 0x7f89681540e0 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f89677fe640 1 -- 172.21.15.7:0/89102406 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8958003530 con 0x7f89681540e0 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f8967fff640 1 -- 172.21.15.7:0/89102406 >> v2:172.21.15.193:3300/0 conn(0x7f89680cf6c0 msgr2=0x7f89680cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f8967fff640 1 --2- 172.21.15.7:0/89102406 >> v2:172.21.15.193:3300/0 conn(0x7f89680cf6c0 0x7f89680cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f8967fff640 1 -- 172.21.15.7:0/89102406 >> v1:172.21.15.193:6789/0 conn(0x7f89681540e0 legacy=0x7f896815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:18.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f8967fff640 1 -- 172.21.15.7:0/89102406 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89680ca8f0 con 0x7f89680ca4d0 2023-11-28T18:48:18.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.352+0000 7f896d637640 1 --2- 172.21.15.7:0/89102406 >> v2:172.21.15.193:3300/0 conn(0x7f89680cf6c0 0x7f89680cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:48:18.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f8967fff640 1 --2- 172.21.15.7:0/89102406 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a8a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f89540097c0 tx=0x7f895401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f99c2d99127e4b66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:18.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f89677fe640 1 -- 172.21.15.7:0/89102406 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89540ab600 con 0x7f89680ca4d0 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f89677fe640 1 -- 172.21.15.7:0/89102406 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89540a7050 con 0x7f89680ca4d0 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f89677fe640 1 -- 172.21.15.7:0/89102406 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89540ab980 con 0x7f89680ca4d0 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f896f09a640 1 -- 172.21.15.7:0/89102406 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 msgr2=0x7f896815a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f896f09a640 1 --2- 172.21.15.7:0/89102406 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a8a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f89540097c0 tx=0x7f895401c480 comp rx=0 tx=0).stop 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f896f09a640 1 -- 172.21.15.7:0/89102406 shutdown_connections 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f896f09a640 1 --2- 172.21.15.7:0/89102406 >> v2:172.21.15.193:3300/0 conn(0x7f89680cf6c0 0x7f89680cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f896f09a640 1 --2- 172.21.15.7:0/89102406 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a8a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f896f09a640 1 -- 172.21.15.7:0/89102406 >> 172.21.15.7:0/89102406 conn(0x7f89680662e0 msgr2=0x7f89680666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f896f09a640 1 -- 172.21.15.7:0/89102406 shutdown_connections 2023-11-28T18:48:18.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.356+0000 7f896f09a640 1 -- 172.21.15.7:0/89102406 wait complete. 2023-11-28T18:48:18.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 Processor -- start 2023-11-28T18:48:18.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 -- start start 2023-11-28T18:48:18.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:18.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89680cf6c0 0x7f896815b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:18.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89681540e0 0x7f896815bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:18.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89681f4d40 con 0x7f89680cf6c0 2023-11-28T18:48:18.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896815a5f0 con 0x7f89681540e0 2023-11-28T18:48:18.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8968152970 con 0x7f89680ca4d0 2023-11-28T18:48:18.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89680cf6c0 0x7f896815b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89680cf6c0 0x7f896815b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38460/0 (socket says 172.21.15.7:38460) 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 -- 172.21.15.7:0/3210045820 learned_addr learned my addr 172.21.15.7:0/3210045820 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896ce36640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896d637640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89681540e0 0x7f896815bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 -- 172.21.15.7:0/3210045820 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 msgr2=0x7f896815a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 -- 172.21.15.7:0/3210045820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89681540e0 msgr2=0x7f896815bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89681540e0 0x7f896815bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 -- 172.21.15.7:0/3210045820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8960009a70 con 0x7f89680cf6c0 2023-11-28T18:48:18.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f8967fff640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89680cf6c0 0x7f896815b790 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f89540ae040 tx=0x7f8954019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f89657fa640 1 -- 172.21.15.7:0/3210045820 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954006e20 con 0x7f89680cf6c0 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f89657fa640 1 -- 172.21.15.7:0/3210045820 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89540b4430 con 0x7f89680cf6c0 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f89657fa640 1 -- 172.21.15.7:0/3210045820 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89540b47c0 con 0x7f89680cf6c0 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8954009420 con 0x7f89680cf6c0 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.360+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8968152c30 con 0x7f89680cf6c0 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.364+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89681f4ef0 con 0x7f89680cf6c0 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.364+0000 7f89657fa640 1 -- 172.21.15.7:0/3210045820 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f89540b2040 con 0x7f89680cf6c0 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.364+0000 7f89657fa640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f894c0668d0 0x7f894c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:18.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.364+0000 7f89657fa640 1 -- 172.21.15.7:0/3210045820 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f895411e5e0 con 0x7f89680cf6c0 2023-11-28T18:48:18.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.364+0000 7f896ce36640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f894c0668d0 0x7f894c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:18.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.364+0000 7f896ce36640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f894c0668d0 0x7f894c068d90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f895c001280 tx=0x7f895c007bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:18.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.368+0000 7f89657fa640 1 -- 172.21.15.7:0/3210045820 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f89681f4ef0 con 0x7f89680cf6c0 2023-11-28T18:48:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.560+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89681f4ef0 con 0x7f89680cf6c0 2023-11-28T18:48:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.560+0000 7f89657fa640 1 -- 172.21.15.7:0/3210045820 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f89681f4ef0 con 0x7f89680cf6c0 2023-11-28T18:48:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:18.564 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f894c0668d0 msgr2=0x7f894c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f894c0668d0 0x7f894c068d90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f895c001280 tx=0x7f895c007bc0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89680cf6c0 msgr2=0x7f896815b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89680cf6c0 0x7f896815b790 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f89540ae040 tx=0x7f8954019220 comp rx=0 tx=0).stop 2023-11-28T18:48:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 shutdown_connections 2023-11-28T18:48:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f894c0668d0 0x7f894c068d90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89681540e0 0x7f896815bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89680cf6c0 0x7f896815b790 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 --2- 172.21.15.7:0/3210045820 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89680ca4d0 0x7f896815a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 >> 172.21.15.7:0/3210045820 conn(0x7f89680662e0 msgr2=0x7f89681453f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 shutdown_connections 2023-11-28T18:48:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:18.564+0000 7f896f09a640 1 -- 172.21.15.7:0/3210045820 wait complete. 2023-11-28T18:48:18.586 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:20.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:20.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.740+0000 7f188a71d640 1 Processor -- start 2023-11-28T18:48:20.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.740+0000 7f188a71d640 1 -- start start 2023-11-28T18:48:20.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f188a71d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18840c8090 0x7f18841575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:20.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f188a71d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18840c6140 con 0x7f1884150b30 2023-11-28T18:48:20.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f188a71d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1884157b00 con 0x7f18840c24f0 2023-11-28T18:48:20.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f188a71d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18840c72b0 con 0x7f18840c8090 2023-11-28T18:48:20.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1883fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1884150b30 0x7f18840c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57622/0 (socket says 172.21.15.7:57622) 2023-11-28T18:48:20.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1883fff640 1 -- 172.21.15.7:0/1585113961 learned_addr learned my addr 172.21.15.7:0/1585113961 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:20.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1888cba640 1 --2- 172.21.15.7:0/1585113961 >> v2:172.21.15.7:3300/0 conn(0x7f18840c8090 0x7f18841575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:20.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1882ffd640 1 -- 172.21.15.7:0/1585113961 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147031741 0 0) 0x7f18840c6140 con 0x7f1884150b30 2023-11-28T18:48:20.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1882ffd640 1 -- 172.21.15.7:0/1585113961 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186c005740 con 0x7f1884150b30 2023-11-28T18:48:20.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1888cba640 1 -- 172.21.15.7:0/1585113961 >> v1:172.21.15.193:6789/0 conn(0x7f18840c24f0 legacy=0x7f18840c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:20.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1888cba640 1 -- 172.21.15.7:0/1585113961 >> v1:172.21.15.7:6789/0 conn(0x7f1884150b30 legacy=0x7f18840c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:20.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1888cba640 1 -- 172.21.15.7:0/1585113961 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18840c2960 con 0x7f18840c8090 2023-11-28T18:48:20.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.744+0000 7f1888cba640 1 --2- 172.21.15.7:0/1585113961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18840c8090 0x7f18841575c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f187c009d90 tx=0x7f187c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f7ca8bda469b1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:20.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f1882ffd640 1 -- 172.21.15.7:0/1585113961 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c0af600 con 0x7f18840c8090 2023-11-28T18:48:20.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f1882ffd640 1 -- 172.21.15.7:0/1585113961 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187c0ab050 con 0x7f18840c8090 2023-11-28T18:48:20.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f1882ffd640 1 -- 172.21.15.7:0/1585113961 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c0af980 con 0x7f18840c8090 2023-11-28T18:48:20.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- 172.21.15.7:0/1585113961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18840c8090 msgr2=0x7f18841575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:20.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 --2- 172.21.15.7:0/1585113961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18840c8090 0x7f18841575c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f187c009d90 tx=0x7f187c01f130 comp rx=0 tx=0).stop 2023-11-28T18:48:20.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- 172.21.15.7:0/1585113961 shutdown_connections 2023-11-28T18:48:20.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 --2- 172.21.15.7:0/1585113961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18840c8090 0x7f18841575c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:20.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- 172.21.15.7:0/1585113961 >> 172.21.15.7:0/1585113961 conn(0x7f18840662e0 msgr2=0x7f18840666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:20.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- 172.21.15.7:0/1585113961 shutdown_connections 2023-11-28T18:48:20.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- 172.21.15.7:0/1585113961 wait complete. 2023-11-28T18:48:20.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 Processor -- start 2023-11-28T18:48:20.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- start start 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18840c24f0 0x7f1884148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18840c8090 0x7f18841eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1884150b30 0x7f1884146f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18841485e0 con 0x7f1884150b30 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1884147490 con 0x7f18840c24f0 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.748+0000 7f188a71d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1884155ec0 con 0x7f18840c8090 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1884150b30 0x7f1884146f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1884150b30 0x7f1884146f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38472/0 (socket says 172.21.15.7:38472) 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 -- 172.21.15.7:0/3042560067 learned_addr learned my addr 172.21.15.7:0/3042560067 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1883fff640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18840c24f0 0x7f1884148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f18837fe640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18840c8090 0x7f18841eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:20.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 -- 172.21.15.7:0/3042560067 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18840c8090 msgr2=0x7f18841eb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:20.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18840c8090 0x7f18841eb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:20.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 -- 172.21.15.7:0/3042560067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18840c24f0 msgr2=0x7f1884148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:20.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18840c24f0 0x7f1884148a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:20.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 -- 172.21.15.7:0/3042560067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c009a70 con 0x7f1884150b30 2023-11-28T18:48:20.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1888cba640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1884150b30 0x7f1884146f50 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f187c0b2040 tx=0x7f187c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:20.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1880ff9640 1 -- 172.21.15.7:0/3042560067 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c009450 con 0x7f1884150b30 2023-11-28T18:48:20.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1880ff9640 1 -- 172.21.15.7:0/3042560067 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187c0b9410 con 0x7f1884150b30 2023-11-28T18:48:20.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1880ff9640 1 -- 172.21.15.7:0/3042560067 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c0b97c0 con 0x7f1884150b30 2023-11-28T18:48:20.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18841561a0 con 0x7f1884150b30 2023-11-28T18:48:20.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1884156760 con 0x7f1884150b30 2023-11-28T18:48:20.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1884148720 con 0x7f1884150b30 2023-11-28T18:48:20.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.752+0000 7f1880ff9640 1 -- 172.21.15.7:0/3042560067 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f187c0b6050 con 0x7f1884150b30 2023-11-28T18:48:20.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.756+0000 7f1880ff9640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18700668d0 0x7f1870068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:20.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.756+0000 7f1880ff9640 1 -- 172.21.15.7:0/3042560067 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f187c122b40 con 0x7f1884150b30 2023-11-28T18:48:20.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.756+0000 7f1883fff640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18700668d0 0x7f1870068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:20.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.756+0000 7f1883fff640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18700668d0 0x7f1870068d90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f18740011f0 tx=0x7f1874007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:20.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.756+0000 7f1880ff9640 1 -- 172.21.15.7:0/3042560067 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1884148720 con 0x7f1884150b30 2023-11-28T18:48:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.952+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1884148720 con 0x7f1884150b30 2023-11-28T18:48:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.952+0000 7f1880ff9640 1 -- 172.21.15.7:0/3042560067 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f1884148720 con 0x7f1884150b30 2023-11-28T18:48:20.955 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:20.955 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18700668d0 msgr2=0x7f1870068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18700668d0 0x7f1870068d90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f18740011f0 tx=0x7f1874007d40 comp rx=0 tx=0).stop 2023-11-28T18:48:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1884150b30 msgr2=0x7f1884146f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1884150b30 0x7f1884146f50 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f187c0b2040 tx=0x7f187c01bed0 comp rx=0 tx=0).stop 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 shutdown_connections 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f18700668d0 0x7f1870068d90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1884150b30 0x7f1884146f50 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f18840c8090 0x7f18841eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 --2- 172.21.15.7:0/3042560067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18840c24f0 0x7f1884148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 >> 172.21.15.7:0/3042560067 conn(0x7f18840662e0 msgr2=0x7f188414fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 shutdown_connections 2023-11-28T18:48:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:20.956+0000 7f188a71d640 1 -- 172.21.15.7:0/3042560067 wait complete. 2023-11-28T18:48:20.977 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:21.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~182s 2023-11-28T18:48:22.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f212c6fe640 1 Processor -- start 2023-11-28T18:48:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f212c6fe640 1 -- start start 2023-11-28T18:48:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f212c6fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21241542e0 0x7f212415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f212c6fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f21240cbb70 0x7f21240cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f212c6fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21240ca920 con 0x7f21240cc810 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f212c6fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2124071d50 con 0x7f21240cbb70 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f212c6fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2124072030 con 0x7f21241542e0 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f2129c99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f21240cbb70 0x7f21240cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f2129c99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f21240cbb70 0x7f21240cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38480/0 (socket says 172.21.15.7:38480) 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f2129c99640 1 -- 172.21.15.7:0/2248570436 learned_addr learned my addr 172.21.15.7:0/2248570436 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f212a49a640 1 --2- 172.21.15.7:0/2248570436 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21241542e0 0x7f212415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f2129c99640 1 -- 172.21.15.7:0/2248570436 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21241542e0 msgr2=0x7f212415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f2129c99640 1 --2- 172.21.15.7:0/2248570436 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21241542e0 0x7f212415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f2129c99640 1 -- 172.21.15.7:0/2248570436 >> v1:172.21.15.193:6789/0 conn(0x7f21240cc810 legacy=0x7f2124072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.132+0000 7f2129c99640 1 -- 172.21.15.7:0/2248570436 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212415e550 con 0x7f21240cbb70 2023-11-28T18:48:23.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f2129c99640 1 --2- 172.21.15.7:0/2248570436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21240cc2d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f2114009870 tx=0x7f211401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1d1a4d98e2f8df server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f2129498640 1 -- 172.21.15.7:0/2248570436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21140a9400 con 0x7f21240cbb70 2023-11-28T18:48:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f2129498640 1 -- 172.21.15.7:0/2248570436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21140a9560 con 0x7f21240cbb70 2023-11-28T18:48:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f2129498640 1 -- 172.21.15.7:0/2248570436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21140a98f0 con 0x7f21240cbb70 2023-11-28T18:48:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f212c6fe640 1 -- 172.21.15.7:0/2248570436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 msgr2=0x7f21240cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f212c6fe640 1 --2- 172.21.15.7:0/2248570436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21240cc2d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f2114009870 tx=0x7f211401c540 comp rx=0 tx=0).stop 2023-11-28T18:48:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f212c6fe640 1 -- 172.21.15.7:0/2248570436 shutdown_connections 2023-11-28T18:48:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f212c6fe640 1 --2- 172.21.15.7:0/2248570436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21240cc2d0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f212c6fe640 1 --2- 172.21.15.7:0/2248570436 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21241542e0 0x7f212415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f212c6fe640 1 -- 172.21.15.7:0/2248570436 >> 172.21.15.7:0/2248570436 conn(0x7f21240717e0 msgr2=0x7f21240662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f212c6fe640 1 -- 172.21.15.7:0/2248570436 shutdown_connections 2023-11-28T18:48:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.136+0000 7f212c6fe640 1 -- 172.21.15.7:0/2248570436 wait complete. 2023-11-28T18:48:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 Processor -- start 2023-11-28T18:48:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 -- start start 2023-11-28T18:48:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21241472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21240cc810 0x7f21240c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21241542e0 0x7f212413b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2124141030 con 0x7f21240cbb70 2023-11-28T18:48:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212414b780 con 0x7f21241542e0 2023-11-28T18:48:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212414bef0 con 0x7f21240cc810 2023-11-28T18:48:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21241472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:23.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21241472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38486/0 (socket says 172.21.15.7:38486) 2023-11-28T18:48:23.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 -- 172.21.15.7:0/1084888069 learned_addr learned my addr 172.21.15.7:0/1084888069 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:23.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 -- 172.21.15.7:0/1084888069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21240cc810 msgr2=0x7f21240c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:23.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21240cc810 0x7f21240c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 -- 172.21.15.7:0/1084888069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21241542e0 msgr2=0x7f212413b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:23.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212ac9b640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21241542e0 0x7f212413b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21241542e0 0x7f212413b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 -- 172.21.15.7:0/1084888069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21140094d0 con 0x7f21240cbb70 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212ac9b640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21241542e0 0x7f212413b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212a49a640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21241472c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f212000e870 tx=0x7f212000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f211affd640 1 -- 172.21.15.7:0/1084888069 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212000eec0 con 0x7f21240cbb70 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f211affd640 1 -- 172.21.15.7:0/1084888069 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212000c940 con 0x7f21240cbb70 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21241414c0 con 0x7f21240cbb70 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.140+0000 7f211affd640 1 -- 172.21.15.7:0/1084888069 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212000ccb0 con 0x7f21240cbb70 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.144+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21240c6d60 con 0x7f21240cbb70 2023-11-28T18:48:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.144+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212406c670 con 0x7f21240cbb70 2023-11-28T18:48:23.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.144+0000 7f211affd640 1 -- 172.21.15.7:0/1084888069 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2120013470 con 0x7f21240cbb70 2023-11-28T18:48:23.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.144+0000 7f211affd640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21080668d0 0x7f2108068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:23.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.144+0000 7f211affd640 1 -- 172.21.15.7:0/1084888069 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f2120084e10 con 0x7f21240cbb70 2023-11-28T18:48:23.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.144+0000 7f2129c99640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21080668d0 0x7f2108068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:23.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.144+0000 7f2129c99640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21080668d0 0x7f2108068d90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f21140ae000 tx=0x7f2114008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:23.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.148+0000 7f211affd640 1 -- 172.21.15.7:0/1084888069 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2120088030 con 0x7f21240cbb70 2023-11-28T18:48:23.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f212404e890 con 0x7f21240cbb70 2023-11-28T18:48:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f211affd640 1 -- 172.21.15.7:0/1084888069 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f2120017090 con 0x7f21240cbb70 2023-11-28T18:48:23.341 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:23.341 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21080668d0 msgr2=0x7f2108068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21080668d0 0x7f2108068d90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f21140ae000 tx=0x7f2114008040 comp rx=0 tx=0).stop 2023-11-28T18:48:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 msgr2=0x7f21241472c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21241472c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f212000e870 tx=0x7f212000ec70 comp rx=0 tx=0).stop 2023-11-28T18:48:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 shutdown_connections 2023-11-28T18:48:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21080668d0 0x7f2108068d90 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21241542e0 0x7f212413b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21240cc810 0x7f21240c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 --2- 172.21.15.7:0/1084888069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21240cbb70 0x7f21241472c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 >> 172.21.15.7:0/1084888069 conn(0x7f21240717e0 msgr2=0x7f212415dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 shutdown_connections 2023-11-28T18:48:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:23.340+0000 7f212c6fe640 1 -- 172.21.15.7:0/1084888069 wait complete. 2023-11-28T18:48:23.364 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:48:25.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281be46640 1 Processor -- start 2023-11-28T18:48:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281be46640 1 -- start start 2023-11-28T18:48:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281be46640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f28141540a0 0x7f2814072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281be46640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28140c8750 con 0x7f28140c8ec0 2023-11-28T18:48:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281be46640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28140c8a30 con 0x7f281415d350 2023-11-28T18:48:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281be46640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281415d840 con 0x7f28141540a0 2023-11-28T18:48:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281a3e3640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f28140c8ec0 0x7f28141646e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33656/0 (socket says 172.21.15.7:33656) 2023-11-28T18:48:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281a3e3640 1 -- 172.21.15.7:0/2667377905 learned_addr learned my addr 172.21.15.7:0/2667377905 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2819be2640 1 --2- 172.21.15.7:0/2667377905 >> v2:172.21.15.193:3300/0 conn(0x7f28141540a0 0x7f2814072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837244281 0 0) 0x7f28140c8750 con 0x7f28140c8ec0 2023-11-28T18:48:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2800003530 con 0x7f28140c8ec0 2023-11-28T18:48:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 673462631 0 0) 0x7f2800003530 con 0x7f28140c8ec0 2023-11-28T18:48:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 >> v2:172.21.15.193:3300/0 conn(0x7f28141540a0 msgr2=0x7f2814072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 --2- 172.21.15.7:0/2667377905 >> v2:172.21.15.193:3300/0 conn(0x7f28141540a0 0x7f2814072ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 >> v1:172.21.15.193:6789/0 conn(0x7f281415d350 legacy=0x7f281406a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281415db20 con 0x7f28140c8ec0 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f280c001a00 con 0x7f28140c8ec0 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 >> v1:172.21.15.7:6789/0 conn(0x7f28140c8ec0 legacy=0x7f28141646e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000039d0 0x7f2800003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28000047d0 0x7f2800004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2800005190 0x7f280000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281415d840 con 0x7f28000047d0 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280000c090 con 0x7f28000039d0 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280000c250 con 0x7f2800005190 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f28193e1640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28000047d0 0x7f2800004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f2819be2640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000039d0 0x7f2800003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.520+0000 7f281a3e3640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2800005190 0x7f280000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f28193e1640 1 -- 172.21.15.7:0/2667377905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2800005190 msgr2=0x7f280000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f28193e1640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2800005190 0x7f280000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f28193e1640 1 -- 172.21.15.7:0/2667377905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000039d0 msgr2=0x7f2800003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f28193e1640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000039d0 0x7f2800003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f28193e1640 1 -- 172.21.15.7:0/2667377905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2800003530 con 0x7f28000047d0 2023-11-28T18:48:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f281a3e3640 1 -- 172.21.15.7:0/2667377905 reap_dead start 2023-11-28T18:48:25.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f28193e1640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28000047d0 0x7f2800004c50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2804001780 tx=0x7f280401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447675b36879eb90 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:25.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28040052d0 con 0x7f28000047d0 2023-11-28T18:48:25.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28040a7050 con 0x7f28000047d0 2023-11-28T18:48:25.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f2818be0640 1 -- 172.21.15.7:0/2667377905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804003600 con 0x7f28000047d0 2023-11-28T18:48:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f281be46640 1 -- 172.21.15.7:0/2667377905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28000047d0 msgr2=0x7f2800004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f281be46640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28000047d0 0x7f2800004c50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2804001780 tx=0x7f280401c400 comp rx=0 tx=0).stop 2023-11-28T18:48:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f281be46640 1 -- 172.21.15.7:0/2667377905 shutdown_connections 2023-11-28T18:48:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f281be46640 1 --2- 172.21.15.7:0/2667377905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28000047d0 0x7f2800004c50 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.524+0000 7f281be46640 1 -- 172.21.15.7:0/2667377905 >> 172.21.15.7:0/2667377905 conn(0x7f2814073cd0 msgr2=0x7f28140662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 -- 172.21.15.7:0/2667377905 shutdown_connections 2023-11-28T18:48:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 -- 172.21.15.7:0/2667377905 wait complete. 2023-11-28T18:48:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 Processor -- start 2023-11-28T18:48:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 -- start start 2023-11-28T18:48:25.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000047d0 0x7f28141f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28141504f0 0x7f2814159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f281415a170 0x7f281414d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28141f0a30 con 0x7f28141504f0 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28141590d0 con 0x7f28000047d0 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281be46640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2814158830 con 0x7f281415a170 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28141504f0 0x7f2814159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28141504f0 0x7f2814159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58120/0 (socket says 172.21.15.7:58120) 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 -- 172.21.15.7:0/1599796347 learned_addr learned my addr 172.21.15.7:0/1599796347 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f281a3e3640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f281415a170 0x7f281414d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f2819be2640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000047d0 0x7f28141f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 -- 172.21.15.7:0/1599796347 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f281415a170 msgr2=0x7f281414d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f281415a170 0x7f281414d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 -- 172.21.15.7:0/1599796347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000047d0 msgr2=0x7f28141f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000047d0 0x7f28141f04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 -- 172.21.15.7:0/1599796347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2814158b10 con 0x7f28141504f0 2023-11-28T18:48:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.528+0000 7f28193e1640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28141504f0 0x7f2814159c30 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2804000c10 tx=0x7f2800005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:25.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f280a7fc640 1 -- 172.21.15.7:0/1599796347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804007440 con 0x7f28141504f0 2023-11-28T18:48:25.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f280a7fc640 1 -- 172.21.15.7:0/1599796347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28040075a0 con 0x7f28141504f0 2023-11-28T18:48:25.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f280a7fc640 1 -- 172.21.15.7:0/1599796347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804006a20 con 0x7f28141504f0 2023-11-28T18:48:25.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2814158df0 con 0x7f28141504f0 2023-11-28T18:48:25.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2814159710 con 0x7f28141504f0 2023-11-28T18:48:25.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281415a550 con 0x7f28141504f0 2023-11-28T18:48:25.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f280a7fc640 1 -- 172.21.15.7:0/1599796347 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2804006b80 con 0x7f28141504f0 2023-11-28T18:48:25.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f280a7fc640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f27f8066950 0x7f27f8068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:25.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f280a7fc640 1 -- 172.21.15.7:0/1599796347 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f280411e940 con 0x7f28141504f0 2023-11-28T18:48:25.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f2819be2640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f27f8066950 0x7f27f8068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:25.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.532+0000 7f2819be2640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f27f8066950 0x7f27f8068e10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2814074f10 tx=0x7f2810009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:25.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.536+0000 7f280a7fc640 1 -- 172.21.15.7:0/1599796347 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f281415a550 con 0x7f28141504f0 2023-11-28T18:48:25.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f281415a550 con 0x7f28141504f0 2023-11-28T18:48:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f280a7fc640 1 -- 172.21.15.7:0/1599796347 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f281415a550 con 0x7f28141504f0 2023-11-28T18:48:25.734 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:25.734 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f27f8066950 msgr2=0x7f27f8068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f27f8066950 0x7f27f8068e10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2814074f10 tx=0x7f2810009190 comp rx=0 tx=0).stop 2023-11-28T18:48:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28141504f0 msgr2=0x7f2814159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28141504f0 0x7f2814159c30 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2804000c10 tx=0x7f2800005800 comp rx=0 tx=0).stop 2023-11-28T18:48:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 shutdown_connections 2023-11-28T18:48:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f27f8066950 0x7f27f8068e10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f281415a170 0x7f281414d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28141504f0 0x7f2814159c30 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 --2- 172.21.15.7:0/1599796347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28000047d0 0x7f28141f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:25.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 >> 172.21.15.7:0/1599796347 conn(0x7f2814073cd0 msgr2=0x7f281415bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:25.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 shutdown_connections 2023-11-28T18:48:25.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:25.732+0000 7f281be46640 1 -- 172.21.15.7:0/1599796347 wait complete. 2023-11-28T18:48:25.756 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:27.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:27.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff20ddb8640 1 Processor -- start 2023-11-28T18:48:27.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff20ddb8640 1 -- start start 2023-11-28T18:48:27.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff20ddb8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2080c3bc0 0x7ff2080c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:27.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff20ddb8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080c38e0 con 0x7ff2080c3490 2023-11-28T18:48:27.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff20ddb8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080c2ce0 con 0x7ff2081503c0 2023-11-28T18:48:27.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff20ddb8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff208157240 con 0x7ff2080c3bc0 2023-11-28T18:48:27.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff1fffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2080c3bc0 0x7ff2080c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:27.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff207fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff2080c3490 0x7ff2080c46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33660/0 (socket says 172.21.15.7:33660) 2023-11-28T18:48:27.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff207fff640 1 -- 172.21.15.7:0/2839444739 learned_addr learned my addr 172.21.15.7:0/2839444739 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:27.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff1fffff640 1 -- 172.21.15.7:0/2839444739 >> v1:172.21.15.193:6789/0 conn(0x7ff2081503c0 legacy=0x7ff20806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:27.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff1fffff640 1 -- 172.21.15.7:0/2839444739 >> v1:172.21.15.7:6789/0 conn(0x7ff2080c3490 legacy=0x7ff2080c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:27.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.912+0000 7ff1fffff640 1 -- 172.21.15.7:0/2839444739 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080c2f80 con 0x7ff2080c3bc0 2023-11-28T18:48:27.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff1fffff640 1 --2- 172.21.15.7:0/2839444739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2080c3bc0 0x7ff2080c7c60 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000a960 tx=0x7ff1f0020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71314ddcf31a553c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:27.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff206ffd640 1 -- 172.21.15.7:0/2839444739 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f00a9e50 con 0x7ff2080c3bc0 2023-11-28T18:48:27.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff206ffd640 1 -- 172.21.15.7:0/2839444739 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f00b1440 con 0x7ff2080c3bc0 2023-11-28T18:48:27.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff206ffd640 1 -- 172.21.15.7:0/2839444739 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f00b17d0 con 0x7ff2080c3bc0 2023-11-28T18:48:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff20ddb8640 1 -- 172.21.15.7:0/2839444739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2080c3bc0 msgr2=0x7ff2080c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff20ddb8640 1 --2- 172.21.15.7:0/2839444739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2080c3bc0 0x7ff2080c7c60 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000a960 tx=0x7ff1f0020040 comp rx=0 tx=0).stop 2023-11-28T18:48:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff20ddb8640 1 -- 172.21.15.7:0/2839444739 shutdown_connections 2023-11-28T18:48:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff20ddb8640 1 --2- 172.21.15.7:0/2839444739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2080c3bc0 0x7ff2080c7c60 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff20ddb8640 1 -- 172.21.15.7:0/2839444739 >> 172.21.15.7:0/2839444739 conn(0x7ff2080662e0 msgr2=0x7ff2080666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff20ddb8640 1 -- 172.21.15.7:0/2839444739 shutdown_connections 2023-11-28T18:48:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.916+0000 7ff20ddb8640 1 -- 172.21.15.7:0/2839444739 wait complete. 2023-11-28T18:48:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 Processor -- start 2023-11-28T18:48:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 -- start start 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2080c3490 0x7ff2081f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2080c3bc0 0x7ff2081e1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2081503c0 0x7ff20814d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2081edde0 con 0x7ff2081503c0 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080cbba0 con 0x7ff2080c3490 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2081e1da0 con 0x7ff2080c3bc0 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2081503c0 0x7ff20814d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2081503c0 0x7ff20814d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58146/0 (socket says 172.21.15.7:58146) 2023-11-28T18:48:27.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 -- 172.21.15.7:0/1289072296 learned_addr learned my addr 172.21.15.7:0/1289072296 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff1fffff640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2080c3bc0 0x7ff2081e1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff2077fe640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2080c3490 0x7ff2081f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 -- 172.21.15.7:0/1289072296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2080c3bc0 msgr2=0x7ff2081e1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2080c3bc0 0x7ff2081e1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 -- 172.21.15.7:0/1289072296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2080c3490 msgr2=0x7ff2081f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2080c3490 0x7ff2081f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 -- 172.21.15.7:0/1289072296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f000a5c0 con 0x7ff2081503c0 2023-11-28T18:48:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff207fff640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2081503c0 0x7ff20814d0f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff20000cc20 tx=0x7ff20000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff204ff9640 1 -- 172.21.15.7:0/1289072296 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000ad40 con 0x7ff2081503c0 2023-11-28T18:48:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff204ff9640 1 -- 172.21.15.7:0/1289072296 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff200012400 con 0x7ff2081503c0 2023-11-28T18:48:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff204ff9640 1 -- 172.21.15.7:0/1289072296 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000aea0 con 0x7ff2081503c0 2023-11-28T18:48:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2080cb5e0 con 0x7ff2081503c0 2023-11-28T18:48:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.920+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2081ed720 con 0x7ff2081503c0 2023-11-28T18:48:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.924+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2081ed580 con 0x7ff2081503c0 2023-11-28T18:48:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.924+0000 7ff204ff9640 1 -- 172.21.15.7:0/1289072296 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff200007070 con 0x7ff2081503c0 2023-11-28T18:48:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.924+0000 7ff204ff9640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e8066900 0x7ff1e8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.924+0000 7ff204ff9640 1 -- 172.21.15.7:0/1289072296 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff200084720 con 0x7ff2081503c0 2023-11-28T18:48:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.924+0000 7ff2077fe640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e8066900 0x7ff1e8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.924+0000 7ff2077fe640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e8066900 0x7ff1e8068dc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8007fd0 tx=0x7ff1f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:27.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:27.928+0000 7ff204ff9640 1 -- 172.21.15.7:0/1289072296 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2081ed580 con 0x7ff2081503c0 2023-11-28T18:48:27.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~188s 2023-11-28T18:48:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.116+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2081edf20 con 0x7ff2081503c0 2023-11-28T18:48:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff204ff9640 1 -- 172.21.15.7:0/1289072296 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7ff2081edf20 con 0x7ff2081503c0 2023-11-28T18:48:28.121 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:28.121 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:28.121 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:28.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e8066900 msgr2=0x7ff1e8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:28.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e8066900 0x7ff1e8068dc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8007fd0 tx=0x7ff1f8009040 comp rx=0 tx=0).stop 2023-11-28T18:48:28.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2081503c0 msgr2=0x7ff20814d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2081503c0 0x7ff20814d0f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff20000cc20 tx=0x7ff20000a430 comp rx=0 tx=0).stop 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 shutdown_connections 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff1e8066900 0x7ff1e8068dc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2081503c0 0x7ff20814d0f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff2080c3bc0 0x7ff2081e1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 --2- 172.21.15.7:0/1289072296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2080c3490 0x7ff2081f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 >> 172.21.15.7:0/1289072296 conn(0x7ff2080662e0 msgr2=0x7ff208155c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 shutdown_connections 2023-11-28T18:48:28.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:28.120+0000 7ff20ddb8640 1 -- 172.21.15.7:0/1289072296 wait complete. 2023-11-28T18:48:28.142 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:30.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.296+0000 7ff26d357640 1 Processor -- start 2023-11-28T18:48:30.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.296+0000 7ff26d357640 1 -- start start 2023-11-28T18:48:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.296+0000 7ff26d357640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 0x7ff26815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.296+0000 7ff26d357640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff268158000 con 0x7ff268157b90 2023-11-28T18:48:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.296+0000 7ff26d357640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff268071e60 con 0x7ff268151cd0 2023-11-28T18:48:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.296+0000 7ff26d357640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff268070930 con 0x7ff26806e560 2023-11-28T18:48:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2677fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff268157b90 0x7ff268162510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33674/0 (socket says 172.21.15.7:33674) 2023-11-28T18:48:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2677fe640 1 -- 172.21.15.7:0/2077825715 learned_addr learned my addr 172.21.15.7:0/2077825715 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff25ffff640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 0x7ff26815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757237754 0 0) 0x7ff268158000 con 0x7ff268157b90 2023-11-28T18:48:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff254003530 con 0x7ff268157b90 2023-11-28T18:48:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416617857 0 0) 0x7ff268071e60 con 0x7ff268151cd0 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff268158000 con 0x7ff268151cd0 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 690154668 0 0) 0x7ff254003530 con 0x7ff268157b90 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 msgr2=0x7ff26815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 0x7ff26815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 >> v1:172.21.15.193:6789/0 conn(0x7ff268151cd0 legacy=0x7ff2680c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268070bf0 con 0x7ff268157b90 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7ff260001a00 con 0x7ff268157b90 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff25ffff640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 0x7ff26815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 >> v1:172.21.15.7:6789/0 conn(0x7ff268157b90 legacy=0x7ff268162510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 0x7ff254003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff254004080 0x7ff2540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff254004d10 0x7ff254005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff268070930 con 0x7ff254004080 2023-11-28T18:48:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25400bad0 con 0x7ff254004d10 2023-11-28T18:48:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25400bc90 con 0x7ff26806e560 2023-11-28T18:48:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff25ffff640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff254004080 0x7ff2540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff2677fe640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff254004d10 0x7ff254005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff266ffd640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 0x7ff254003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff25ffff640 1 -- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 msgr2=0x7ff254003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff25ffff640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff26806e560 0x7ff254003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff25ffff640 1 -- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff254004d10 msgr2=0x7ff254005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff25ffff640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff254004d10 0x7ff254005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.300+0000 7ff25ffff640 1 -- 172.21.15.7:0/2077825715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250009420 con 0x7ff254004080 2023-11-28T18:48:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff25ffff640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff254004080 0x7ff2540047d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff250009c90 tx=0x7ff25001cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a5ed18364e0ff server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250004030 con 0x7ff254004080 2023-11-28T18:48:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff250004190 con 0x7ff254004080 2023-11-28T18:48:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff2667fc640 1 -- 172.21.15.7:0/2077825715 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2500a5690 con 0x7ff254004080 2023-11-28T18:48:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff26d357640 1 -- 172.21.15.7:0/2077825715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff254004080 msgr2=0x7ff2540047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff26d357640 1 --2- 172.21.15.7:0/2077825715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff254004080 0x7ff2540047d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff250009c90 tx=0x7ff25001cbf0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff2677fe640 1 -- 172.21.15.7:0/2077825715 reap_dead start 2023-11-28T18:48:30.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff26d357640 1 -- 172.21.15.7:0/2077825715 shutdown_connections 2023-11-28T18:48:30.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff26d357640 1 -- 172.21.15.7:0/2077825715 >> 172.21.15.7:0/2077825715 conn(0x7ff2680662e0 msgr2=0x7ff2680666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:30.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff26d357640 1 -- 172.21.15.7:0/2077825715 shutdown_connections 2023-11-28T18:48:30.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff26d357640 1 -- 172.21.15.7:0/2077825715 wait complete. 2023-11-28T18:48:30.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff26d357640 1 Processor -- start 2023-11-28T18:48:30.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.304+0000 7ff26d357640 1 -- start start 2023-11-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff254004080 0x7ff2680c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2680c6100 0x7ff2680c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2681f6660 0x7ff2681f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680c41a0 con 0x7ff2680c6100 2023-11-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680c7480 con 0x7ff2681f6660 2023-11-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680c3280 con 0x7ff254004080 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2680c6100 0x7ff2680c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2680c6100 0x7ff2680c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58168/0 (socket says 172.21.15.7:58168) 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 -- 172.21.15.7:0/1453509033 learned_addr learned my addr 172.21.15.7:0/1453509033 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff266ffd640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff254004080 0x7ff2680c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff2677fe640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2681f6660 0x7ff2681f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 -- 172.21.15.7:0/1453509033 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff254004080 msgr2=0x7ff2680c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff254004080 0x7ff2680c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 -- 172.21.15.7:0/1453509033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2681f6660 msgr2=0x7ff2681f6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2681f6660 0x7ff2681f6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 -- 172.21.15.7:0/1453509033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2680c3520 con 0x7ff2680c6100 2023-11-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25ffff640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2680c6100 0x7ff2680c6f40 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff250009c90 tx=0x7ff254005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25f7fe640 1 -- 172.21.15.7:0/1453509033 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250004300 con 0x7ff2680c6100 2023-11-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25f7fe640 1 -- 172.21.15.7:0/1453509033 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff250004460 con 0x7ff2680c6100 2023-11-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2680c3800 con 0x7ff2680c6100 2023-11-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25f7fe640 1 -- 172.21.15.7:0/1453509033 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2500a58d0 con 0x7ff2680c6100 2023-11-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2680c3dc0 con 0x7ff2680c6100 2023-11-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2680c42e0 con 0x7ff2680c6100 2023-11-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.308+0000 7ff25f7fe640 1 -- 172.21.15.7:0/1453509033 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff2500a5a30 con 0x7ff2680c6100 2023-11-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.312+0000 7ff25f7fe640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff248066900 0x7ff248068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.312+0000 7ff25f7fe640 1 -- 172.21.15.7:0/1453509033 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff25011efc0 con 0x7ff2680c6100 2023-11-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.312+0000 7ff266ffd640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff248066900 0x7ff248068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.312+0000 7ff266ffd640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff248066900 0x7ff248068dc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff258007fd0 tx=0x7ff258009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.316+0000 7ff25f7fe640 1 -- 172.21.15.7:0/1453509033 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2680c42e0 con 0x7ff2680c6100 2023-11-28T18:48:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2680c42e0 con 0x7ff2680c6100 2023-11-28T18:48:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff25f7fe640 1 -- 172.21.15.7:0/1453509033 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7ff2680c42e0 con 0x7ff2680c6100 2023-11-28T18:48:30.509 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:30.509 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff248066900 msgr2=0x7ff248068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff248066900 0x7ff248068dc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff258007fd0 tx=0x7ff258009040 comp rx=0 tx=0).stop 2023-11-28T18:48:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2680c6100 msgr2=0x7ff2680c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2680c6100 0x7ff2680c6f40 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff250009c90 tx=0x7ff254005800 comp rx=0 tx=0).stop 2023-11-28T18:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 shutdown_connections 2023-11-28T18:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff248066900 0x7ff248068dc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2681f6660 0x7ff2681f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2680c6100 0x7ff2680c6f40 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 --2- 172.21.15.7:0/1453509033 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff254004080 0x7ff2680c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 >> 172.21.15.7:0/1453509033 conn(0x7ff2680662e0 msgr2=0x7ff268072580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 shutdown_connections 2023-11-28T18:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:30.508+0000 7ff26d357640 1 -- 172.21.15.7:0/1453509033 wait complete. 2023-11-28T18:48:30.534 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:32.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:32.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.688+0000 7f2e40f3e640 1 Processor -- start 2023-11-28T18:48:32.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.688+0000 7f2e40f3e640 1 -- start start 2023-11-28T18:48:32.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.688+0000 7f2e40f3e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c150b30 0x7f2e3c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:32.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.688+0000 7f2e40f3e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e3c1575c0 0x7f2e3c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:32.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.688+0000 7f2e40f3e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f2e3c0c4020 0x7f2e3c0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:32.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.688+0000 7f2e40f3e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3c0c72b0 con 0x7f2e3c1575c0 2023-11-28T18:48:32.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.688+0000 7f2e40f3e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3c070c00 con 0x7f2e3c0c4020 2023-11-28T18:48:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.688+0000 7f2e40f3e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c0c6140 con 0x7f2e3c150b30 2023-11-28T18:48:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e3c1575c0 0x7f2e3c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e3c1575c0 0x7f2e3c0c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58180/0 (socket says 172.21.15.7:58180) 2023-11-28T18:48:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 -- 172.21.15.7:0/3403886020 learned_addr learned my addr 172.21.15.7:0/3403886020 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e3a59c640 1 --2- 172.21.15.7:0/3403886020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c150b30 0x7f2e3c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e3ad9d640 1 --2- 172.21.15.7:0/3403886020 >> v2:172.21.15.193:3300/0 conn(0x7f2e3c0c4020 0x7f2e3c0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 -- 172.21.15.7:0/3403886020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c150b30 msgr2=0x7f2e3c0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 --2- 172.21.15.7:0/3403886020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c150b30 0x7f2e3c0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 -- 172.21.15.7:0/3403886020 >> v2:172.21.15.193:3300/0 conn(0x7f2e3c0c4020 msgr2=0x7f2e3c0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 --2- 172.21.15.7:0/3403886020 >> v2:172.21.15.193:3300/0 conn(0x7f2e3c0c4020 0x7f2e3c0c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 -- 172.21.15.7:0/3403886020 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c0c6370 con 0x7f2e3c1575c0 2023-11-28T18:48:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e3a59c640 1 --2- 172.21.15.7:0/3403886020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c150b30 0x7f2e3c0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:48:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e3ad9d640 1 --2- 172.21.15.7:0/3403886020 >> v2:172.21.15.193:3300/0 conn(0x7f2e3c0c4020 0x7f2e3c0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:48:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e33fff640 1 --2- 172.21.15.7:0/3403886020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c1575c0 0x7f2e3c0c3ae0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0c95f0 tx=0x7f2e240213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd3d51bfb49afc9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e39d9b640 1 -- 172.21.15.7:0/3403886020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e240b1600 con 0x7f2e3c1575c0 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e39d9b640 1 -- 172.21.15.7:0/3403886020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e240ad050 con 0x7f2e3c1575c0 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.692+0000 7f2e39d9b640 1 -- 172.21.15.7:0/3403886020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e240b1980 con 0x7f2e3c1575c0 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3403886020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c1575c0 msgr2=0x7f2e3c0c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3403886020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c1575c0 0x7f2e3c0c3ae0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0c95f0 tx=0x7f2e240213c0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3403886020 shutdown_connections 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3403886020 >> v2:172.21.15.193:3300/0 conn(0x7f2e3c0c4020 0x7f2e3c0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3403886020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c1575c0 0x7f2e3c0c3ae0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3403886020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c150b30 0x7f2e3c0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3403886020 >> 172.21.15.7:0/3403886020 conn(0x7f2e3c0662e0 msgr2=0x7f2e3c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:32.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3403886020 shutdown_connections 2023-11-28T18:48:32.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3403886020 wait complete. 2023-11-28T18:48:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 Processor -- start 2023-11-28T18:48:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- start start 2023-11-28T18:48:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e3c0c4020 0x7f2e3c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c150b30 0x7f2e3c14acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c1575c0 0x7f2e3c1f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c1f3db0 con 0x7f2e3c150b30 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c154310 con 0x7f2e3c0c4020 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e40f3e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c152330 con 0x7f2e3c1575c0 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e33fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c150b30 0x7f2e3c14acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e33fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c150b30 0x7f2e3c14acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58194/0 (socket says 172.21.15.7:58194) 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e33fff640 1 -- 172.21.15.7:0/3367323222 learned_addr learned my addr 172.21.15.7:0/3367323222 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e3a59c640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e3c0c4020 0x7f2e3c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.696+0000 7f2e3ad9d640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c1575c0 0x7f2e3c1f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e33fff640 1 -- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c1575c0 msgr2=0x7f2e3c1f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e33fff640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c1575c0 0x7f2e3c1f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e33fff640 1 -- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e3c0c4020 msgr2=0x7f2e3c1527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e33fff640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e3c0c4020 0x7f2e3c1527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e33fff640 1 -- 172.21.15.7:0/3367323222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2400b770 con 0x7f2e3c150b30 2023-11-28T18:48:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e3ad9d640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c1575c0 0x7f2e3c1f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:48:32.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e33fff640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c150b30 0x7f2e3c14acb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b2c0 tx=0x7f2e2401fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:32.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e32ffd640 1 -- 172.21.15.7:0/3367323222 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e240bc650 con 0x7f2e3c150b30 2023-11-28T18:48:32.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e32ffd640 1 -- 172.21.15.7:0/3367323222 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e240bc7b0 con 0x7f2e3c150b30 2023-11-28T18:48:32.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3c153410 con 0x7f2e3c150b30 2023-11-28T18:48:32.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e32ffd640 1 -- 172.21.15.7:0/3367323222 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e240bcb60 con 0x7f2e3c150b30 2023-11-28T18:48:32.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3c154030 con 0x7f2e3c150b30 2023-11-28T18:48:32.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3c0c6bb0 con 0x7f2e3c150b30 2023-11-28T18:48:32.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e32ffd640 1 -- 172.21.15.7:0/3367323222 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2e240b8070 con 0x7f2e3c150b30 2023-11-28T18:48:32.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e32ffd640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e1c0668d0 0x7f2e1c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:32.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e32ffd640 1 -- 172.21.15.7:0/3367323222 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f2e24122fe0 con 0x7f2e3c150b30 2023-11-28T18:48:32.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.700+0000 7f2e3a59c640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e1c0668d0 0x7f2e1c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:32.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.704+0000 7f2e3a59c640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e1c0668d0 0x7f2e1c068d90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00ab60 tx=0x7f2e2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:32.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.704+0000 7f2e32ffd640 1 -- 172.21.15.7:0/3367323222 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2e240efdb0 con 0x7f2e3c150b30 2023-11-28T18:48:32.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.896+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e3c04e890 con 0x7f2e3c150b30 2023-11-28T18:48:32.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.896+0000 7f2e32ffd640 1 -- 172.21.15.7:0/3367323222 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f2e240b6070 con 0x7f2e3c150b30 2023-11-28T18:48:32.899 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:32.899 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:32.900 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:32.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e1c0668d0 msgr2=0x7f2e1c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:32.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e1c0668d0 0x7f2e1c068d90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00ab60 tx=0x7f2e2c008040 comp rx=0 tx=0).stop 2023-11-28T18:48:32.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c150b30 msgr2=0x7f2e3c14acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:32.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c150b30 0x7f2e3c14acb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b2c0 tx=0x7f2e2401fc00 comp rx=0 tx=0).stop 2023-11-28T18:48:32.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 shutdown_connections 2023-11-28T18:48:32.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2e1c0668d0 0x7f2e1c068d90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2e3c1575c0 0x7f2e3c1f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3c150b30 0x7f2e3c14acb0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 --2- 172.21.15.7:0/3367323222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e3c0c4020 0x7f2e3c1527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 >> 172.21.15.7:0/3367323222 conn(0x7f2e3c0662e0 msgr2=0x7f2e3c14ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 shutdown_connections 2023-11-28T18:48:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:32.900+0000 7f2e40f3e640 1 -- 172.21.15.7:0/3367323222 wait complete. 2023-11-28T18:48:32.921 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:34.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~195s 2023-11-28T18:48:34.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff64309a640 1 Processor -- start 2023-11-28T18:48:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff64309a640 1 -- start start 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff64309a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff63c150b30 0x7ff63c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff64309a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff63c0c7980 0x7ff63c0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff64309a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c0c6140 con 0x7ff63c1575c0 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff64309a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c0c72b0 con 0x7ff63c150b30 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff64309a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c070c00 con 0x7ff63c0c7980 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff63bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff63c1575c0 0x7ff63c0c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53960/0 (socket says 172.21.15.7:53960) 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff640e36640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff63c150b30 0x7ff63c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff63bfff640 1 -- 172.21.15.7:0/319185867 learned_addr learned my addr 172.21.15.7:0/319185867 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff641637640 1 --2- 172.21.15.7:0/319185867 >> v2:172.21.15.193:3300/0 conn(0x7ff63c0c7980 0x7ff63c0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff640e36640 1 -- 172.21.15.7:0/319185867 >> v2:172.21.15.193:3300/0 conn(0x7ff63c0c7980 msgr2=0x7ff63c0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff640e36640 1 --2- 172.21.15.7:0/319185867 >> v2:172.21.15.193:3300/0 conn(0x7ff63c0c7980 0x7ff63c0c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff640e36640 1 -- 172.21.15.7:0/319185867 >> v1:172.21.15.7:6789/0 conn(0x7ff63c1575c0 legacy=0x7ff63c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.076+0000 7ff640e36640 1 -- 172.21.15.7:0/319185867 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0c8310 con 0x7ff63c150b30 2023-11-28T18:48:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff640e36640 1 --2- 172.21.15.7:0/319185867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c150b30 0x7ff63c0c24f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff63c06d520 tx=0x7ff63001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b41f95b03cf97b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff63b7fe640 1 -- 172.21.15.7:0/319185867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300a8950 con 0x7ff63c150b30 2023-11-28T18:48:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff63b7fe640 1 -- 172.21.15.7:0/319185867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6300a8ab0 con 0x7ff63c150b30 2023-11-28T18:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff63b7fe640 1 -- 172.21.15.7:0/319185867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300a8e40 con 0x7ff63c150b30 2023-11-28T18:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff64309a640 1 -- 172.21.15.7:0/319185867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c150b30 msgr2=0x7ff63c0c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff64309a640 1 --2- 172.21.15.7:0/319185867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c150b30 0x7ff63c0c24f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff63c06d520 tx=0x7ff63001ec50 comp rx=0 tx=0).stop 2023-11-28T18:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff64309a640 1 -- 172.21.15.7:0/319185867 shutdown_connections 2023-11-28T18:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff64309a640 1 --2- 172.21.15.7:0/319185867 >> v2:172.21.15.193:3300/0 conn(0x7ff63c0c7980 0x7ff63c0c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff64309a640 1 --2- 172.21.15.7:0/319185867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c150b30 0x7ff63c0c24f0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff64309a640 1 -- 172.21.15.7:0/319185867 >> 172.21.15.7:0/319185867 conn(0x7ff63c0662e0 msgr2=0x7ff63c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff64309a640 1 -- 172.21.15.7:0/319185867 shutdown_connections 2023-11-28T18:48:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.080+0000 7ff64309a640 1 -- 172.21.15.7:0/319185867 wait complete. 2023-11-28T18:48:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 Processor -- start 2023-11-28T18:48:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 -- start start 2023-11-28T18:48:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff63c0c7980 0x7ff63c156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff63c150b30 0x7ff63c1f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c1575c0 0x7ff63c14cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c1554a0 con 0x7ff63c1575c0 2023-11-28T18:48:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c1f2e10 con 0x7ff63c150b30 2023-11-28T18:48:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c1ef670 con 0x7ff63c0c7980 2023-11-28T18:48:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c1575c0 0x7ff63c14cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff640e36640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff63c0c7980 0x7ff63c156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c1575c0 0x7ff63c14cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39042/0 (socket says 172.21.15.7:39042) 2023-11-28T18:48:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 -- 172.21.15.7:0/159317793 learned_addr learned my addr 172.21.15.7:0/159317793 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff63bfff640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff63c150b30 0x7ff63c1f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 -- 172.21.15.7:0/159317793 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff63c0c7980 msgr2=0x7ff63c156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff63c0c7980 0x7ff63c156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 -- 172.21.15.7:0/159317793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff63c150b30 msgr2=0x7ff63c1f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff63c150b30 0x7ff63c1f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 -- 172.21.15.7:0/159317793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630009420 con 0x7ff63c1575c0 2023-11-28T18:48:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff641637640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c1575c0 0x7ff63c14cd70 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff634007e40 tx=0x7ff63400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff6397fa640 1 -- 172.21.15.7:0/159317793 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634014620 con 0x7ff63c1575c0 2023-11-28T18:48:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff6397fa640 1 -- 172.21.15.7:0/159317793 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63400ee90 con 0x7ff63c1575c0 2023-11-28T18:48:35.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff6397fa640 1 -- 172.21.15.7:0/159317793 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6340149e0 con 0x7ff63c1575c0 2023-11-28T18:48:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c152f80 con 0x7ff63c1575c0 2023-11-28T18:48:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.084+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c155c90 con 0x7ff63c1575c0 2023-11-28T18:48:35.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.088+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c1f2be0 con 0x7ff63c1575c0 2023-11-28T18:48:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.088+0000 7ff6397fa640 1 -- 172.21.15.7:0/159317793 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff634014b40 con 0x7ff63c1575c0 2023-11-28T18:48:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.088+0000 7ff6397fa640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6200668d0 0x7ff620068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.088+0000 7ff6397fa640 1 -- 172.21.15.7:0/159317793 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff634085230 con 0x7ff63c1575c0 2023-11-28T18:48:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.088+0000 7ff640e36640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6200668d0 0x7ff620068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.088+0000 7ff640e36640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6200668d0 0x7ff620068d90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff6300b2000 tx=0x7ff630008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:35.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.092+0000 7ff6397fa640 1 -- 172.21.15.7:0/159317793 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff63c1f2be0 con 0x7ff63c1575c0 2023-11-28T18:48:35.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff63c1f2be0 con 0x7ff63c1575c0 2023-11-28T18:48:35.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff6397fa640 1 -- 172.21.15.7:0/159317793 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7ff63c1f2be0 con 0x7ff63c1575c0 2023-11-28T18:48:35.290 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:35.290 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:35.290 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:35.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6200668d0 msgr2=0x7ff620068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:35.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6200668d0 0x7ff620068d90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff6300b2000 tx=0x7ff630008040 comp rx=0 tx=0).stop 2023-11-28T18:48:35.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c1575c0 msgr2=0x7ff63c14cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:35.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c1575c0 0x7ff63c14cd70 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff634007e40 tx=0x7ff63400e4b0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 shutdown_connections 2023-11-28T18:48:35.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff6200668d0 0x7ff620068d90 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff63c1575c0 0x7ff63c14cd70 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff63c150b30 0x7ff63c1f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 --2- 172.21.15.7:0/159317793 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff63c0c7980 0x7ff63c156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:35.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 >> 172.21.15.7:0/159317793 conn(0x7ff63c0662e0 msgr2=0x7ff63c14fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:35.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 shutdown_connections 2023-11-28T18:48:35.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:35.288+0000 7ff64309a640 1 -- 172.21.15.7:0/159317793 wait complete. 2023-11-28T18:48:35.312 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:48:37.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:37.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd135301640 1 Processor -- start 2023-11-28T18:48:37.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd135301640 1 -- start start 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd135301640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd130150b30 0x7fd1300c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd135301640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1301575c0 0x7fd1300c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd135301640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1300c6140 con 0x7fd1300c4020 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd135301640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1300c72b0 con 0x7fd130150b30 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd135301640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd130070c00 con 0x7fd1301575c0 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd12effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd130150b30 0x7fd1300c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd12effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd130150b30 0x7fd1300c24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39050/0 (socket says 172.21.15.7:39050) 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd12effd640 1 -- 172.21.15.7:0/779486774 learned_addr learned my addr 172.21.15.7:0/779486774 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:37.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd12e7fc640 1 --2- 172.21.15.7:0/779486774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1301575c0 0x7fd1300c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:37.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd12effd640 1 -- 172.21.15.7:0/779486774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1301575c0 msgr2=0x7fd1300c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:37.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd12effd640 1 --2- 172.21.15.7:0/779486774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1301575c0 0x7fd1300c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd12effd640 1 -- 172.21.15.7:0/779486774 >> v1:172.21.15.193:6789/0 conn(0x7fd1300c4020 legacy=0x7fd1300c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:37.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.468+0000 7fd12effd640 1 -- 172.21.15.7:0/779486774 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1300c6370 con 0x7fd130150b30 2023-11-28T18:48:37.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd12effd640 1 --2- 172.21.15.7:0/779486774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1300c24f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd120009870 tx=0x7fd12001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99d2380109a952d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:37.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd12dffb640 1 -- 172.21.15.7:0/779486774 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1200a89b0 con 0x7fd130150b30 2023-11-28T18:48:37.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd12dffb640 1 -- 172.21.15.7:0/779486774 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1200a8b10 con 0x7fd130150b30 2023-11-28T18:48:37.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd12dffb640 1 -- 172.21.15.7:0/779486774 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1200a8ea0 con 0x7fd130150b30 2023-11-28T18:48:37.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd135301640 1 -- 172.21.15.7:0/779486774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 msgr2=0x7fd1300c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:37.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd135301640 1 --2- 172.21.15.7:0/779486774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1300c24f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd120009870 tx=0x7fd12001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd135301640 1 -- 172.21.15.7:0/779486774 shutdown_connections 2023-11-28T18:48:37.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd135301640 1 --2- 172.21.15.7:0/779486774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1301575c0 0x7fd1300c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd135301640 1 --2- 172.21.15.7:0/779486774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1300c24f0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd135301640 1 -- 172.21.15.7:0/779486774 >> 172.21.15.7:0/779486774 conn(0x7fd1300662e0 msgr2=0x7fd1300666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:37.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd135301640 1 -- 172.21.15.7:0/779486774 shutdown_connections 2023-11-28T18:48:37.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.472+0000 7fd135301640 1 -- 172.21.15.7:0/779486774 wait complete. 2023-11-28T18:48:37.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 Processor -- start 2023-11-28T18:48:37.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 -- start start 2023-11-28T18:48:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1300c4020 0x7fd1301e2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1301488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1301575c0 0x7fd130148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd130148150 con 0x7fd130150b30 2023-11-28T18:48:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1301e3210 con 0x7fd1301575c0 2023-11-28T18:48:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1301e34f0 con 0x7fd1300c4020 2023-11-28T18:48:37.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1301488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1301488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39054/0 (socket says 172.21.15.7:39054) 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 -- 172.21.15.7:0/3706126850 learned_addr learned my addr 172.21.15.7:0/3706126850 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12effd640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1300c4020 0x7fd1301e2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12f7fe640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1301575c0 0x7fd130148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 -- 172.21.15.7:0/3706126850 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1300c4020 msgr2=0x7fd1301e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1300c4020 0x7fd1301e2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 -- 172.21.15.7:0/3706126850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1301575c0 msgr2=0x7fd130148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1301575c0 0x7fd130148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 -- 172.21.15.7:0/3706126850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1200094d0 con 0x7fd130150b30 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd12e7fc640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1301488c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd12400d5f0 tx=0x7fd12400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:37.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd1177fe640 1 -- 172.21.15.7:0/3706126850 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1240125f0 con 0x7fd130150b30 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd1177fe640 1 -- 172.21.15.7:0/3706126850 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd124012750 con 0x7fd130150b30 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd1177fe640 1 -- 172.21.15.7:0/3706126850 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124012ae0 con 0x7fd130150b30 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1301e37d0 con 0x7fd130150b30 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.476+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1301483a0 con 0x7fd130150b30 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.480+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd130148560 con 0x7fd130150b30 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.480+0000 7fd1177fe640 1 -- 172.21.15.7:0/3706126850 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd124012c40 con 0x7fd130150b30 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.480+0000 7fd1177fe640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd11c0668d0 0x7fd11c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.480+0000 7fd1177fe640 1 -- 172.21.15.7:0/3706126850 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fd124084cb0 con 0x7fd130150b30 2023-11-28T18:48:37.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.480+0000 7fd12effd640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd11c0668d0 0x7fd11c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:37.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.480+0000 7fd12effd640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd11c0668d0 0x7fd11c068d90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd1200b2000 tx=0x7fd1200aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:37.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.484+0000 7fd1177fe640 1 -- 172.21.15.7:0/3706126850 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd124051a80 con 0x7fd130150b30 2023-11-28T18:48:37.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.672+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd13004e890 con 0x7fd130150b30 2023-11-28T18:48:37.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd1177fe640 1 -- 172.21.15.7:0/3706126850 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fd1240550a0 con 0x7fd130150b30 2023-11-28T18:48:37.676 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:37.677 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:37.677 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:37.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd11c0668d0 msgr2=0x7fd11c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:37.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd11c0668d0 0x7fd11c068d90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd1200b2000 tx=0x7fd1200aa070 comp rx=0 tx=0).stop 2023-11-28T18:48:37.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 msgr2=0x7fd1301488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:37.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1301488c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd12400d5f0 tx=0x7fd12400dab0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 shutdown_connections 2023-11-28T18:48:37.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd11c0668d0 0x7fd11c068d90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1301575c0 0x7fd130148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd130150b30 0x7fd1301488c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 --2- 172.21.15.7:0/3706126850 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd1300c4020 0x7fd1301e2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:37.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 >> 172.21.15.7:0/3706126850 conn(0x7fd1300662e0 msgr2=0x7fd1300c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:37.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 shutdown_connections 2023-11-28T18:48:37.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:37.676+0000 7fd135301640 1 -- 172.21.15.7:0/3706126850 wait complete. 2023-11-28T18:48:37.698 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:39.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa2c17640 1 Processor -- start 2023-11-28T18:48:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa2c17640 1 -- start start 2023-11-28T18:48:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa2c17640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a9c1542e0 0x7f9a9c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa2c17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa2c17640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f9a9c0cc810 0x7f9a9c072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa2c17640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c071d50 con 0x7f9a9c1542e0 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa2c17640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c072030 con 0x7f9a9c0cc810 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa2c17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0ca920 con 0x7f9a9c0cbb70 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa09b3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a9c1542e0 0x7f9a9c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa09b3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a9c1542e0 0x7f9a9c15e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39060/0 (socket says 172.21.15.7:39060) 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa09b3640 1 -- 172.21.15.7:0/3512760156 learned_addr learned my addr 172.21.15.7:0/3512760156 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9a9bfff640 1 --2- 172.21.15.7:0/3512760156 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa11b4640 1 --2- 172.21.15.7:0/3512760156 >> v2:172.21.15.193:3300/0 conn(0x7f9a9c0cc810 0x7f9a9c072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa09b3640 1 -- 172.21.15.7:0/3512760156 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 msgr2=0x7f9a9c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa09b3640 1 --2- 172.21.15.7:0/3512760156 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa09b3640 1 -- 172.21.15.7:0/3512760156 >> v2:172.21.15.193:3300/0 conn(0x7f9a9c0cc810 msgr2=0x7f9a9c072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa09b3640 1 --2- 172.21.15.7:0/3512760156 >> v2:172.21.15.193:3300/0 conn(0x7f9a9c0cc810 0x7f9a9c072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa09b3640 1 -- 172.21.15.7:0/3512760156 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c15e550 con 0x7f9a9c1542e0 2023-11-28T18:48:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.852+0000 7f9aa11b4640 1 --2- 172.21.15.7:0/3512760156 >> v2:172.21.15.193:3300/0 conn(0x7f9a9c0cc810 0x7f9a9c072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:48:39.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa09b3640 1 --2- 172.21.15.7:0/3512760156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15e010 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c066fc0 tx=0x7f9a9001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30edd879da87b6c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:39.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9a9b7fe640 1 -- 172.21.15.7:0/3512760156 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a900ac630 con 0x7f9a9c1542e0 2023-11-28T18:48:39.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9a9b7fe640 1 -- 172.21.15.7:0/3512760156 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a900ac790 con 0x7f9a9c1542e0 2023-11-28T18:48:39.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9a9b7fe640 1 -- 172.21.15.7:0/3512760156 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a900acb20 con 0x7f9a9c1542e0 2023-11-28T18:48:39.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3512760156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 msgr2=0x7f9a9c15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:39.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3512760156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15e010 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c066fc0 tx=0x7f9a9001c500 comp rx=0 tx=0).stop 2023-11-28T18:48:39.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3512760156 shutdown_connections 2023-11-28T18:48:39.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3512760156 >> v2:172.21.15.193:3300/0 conn(0x7f9a9c0cc810 0x7f9a9c072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:39.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3512760156 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:39.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3512760156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15e010 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3512760156 >> 172.21.15.7:0/3512760156 conn(0x7f9a9c0717e0 msgr2=0x7f9a9c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3512760156 shutdown_connections 2023-11-28T18:48:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.856+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3512760156 wait complete. 2023-11-28T18:48:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 Processor -- start 2023-11-28T18:48:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 -- start start 2023-11-28T18:48:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a9c0cc810 0x7f9a9c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c15cc40 con 0x7f9a9c1542e0 2023-11-28T18:48:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c1567f0 con 0x7f9a9c0cc810 2023-11-28T18:48:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c15bb40 con 0x7f9a9c0cbb70 2023-11-28T18:48:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39070/0 (socket says 172.21.15.7:39070) 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 -- 172.21.15.7:0/3317688859 learned_addr learned my addr 172.21.15.7:0/3317688859 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa09b3640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9a9bfff640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a9c0cc810 0x7f9a9c15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 -- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 msgr2=0x7f9a9c156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 -- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a9c0cc810 msgr2=0x7f9a9c15d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a9c0cc810 0x7f9a9c15d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 -- 172.21.15.7:0/3317688859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a900094d0 con 0x7f9a9c1542e0 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa09b3640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa11b4640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15d5a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9a94007ef0 tx=0x7f9a9400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9a997fa640 1 -- 172.21.15.7:0/3317688859 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94003ba0 con 0x7f9a9c1542e0 2023-11-28T18:48:39.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9a997fa640 1 -- 172.21.15.7:0/3317688859 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9400f040 con 0x7f9a9c1542e0 2023-11-28T18:48:39.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9a997fa640 1 -- 172.21.15.7:0/3317688859 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9400de20 con 0x7f9a9c1542e0 2023-11-28T18:48:39.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.860+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c15c2a0 con 0x7f9a9c1542e0 2023-11-28T18:48:39.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.864+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c15c860 con 0x7f9a9c1542e0 2023-11-28T18:48:39.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.864+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c06c670 con 0x7f9a9c1542e0 2023-11-28T18:48:39.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.864+0000 7f9a997fa640 1 -- 172.21.15.7:0/3317688859 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9a94004030 con 0x7f9a9c1542e0 2023-11-28T18:48:39.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.864+0000 7f9a997fa640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9a80066900 0x7f9a80068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:39.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.864+0000 7f9a997fa640 1 -- 172.21.15.7:0/3317688859 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f9a94084ac0 con 0x7f9a9c1542e0 2023-11-28T18:48:39.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.864+0000 7f9aa09b3640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9a80066900 0x7f9a80068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:39.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.864+0000 7f9aa09b3640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9a80066900 0x7f9a80068dc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9a900ae000 tx=0x7f9a90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:39.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:39.868+0000 7f9a997fa640 1 -- 172.21.15.7:0/3317688859 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9a94051890 con 0x7f9a9c1542e0 2023-11-28T18:48:40.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.060+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a9c15cd80 con 0x7f9a9c1542e0 2023-11-28T18:48:40.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.060+0000 7f9a997fa640 1 -- 172.21.15.7:0/3317688859 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f9a9c15cd80 con 0x7f9a9c1542e0 2023-11-28T18:48:40.063 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:40.063 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:40.063 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:40.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.060+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9a80066900 msgr2=0x7f9a80068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:40.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.060+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9a80066900 0x7f9a80068dc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9a900ae000 tx=0x7f9a90008040 comp rx=0 tx=0).stop 2023-11-28T18:48:40.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.060+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 msgr2=0x7f9a9c15d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.060+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15d5a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9a94007ef0 tx=0x7f9a9400d4a0 comp rx=0 tx=0).stop 2023-11-28T18:48:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.064+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 shutdown_connections 2023-11-28T18:48:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.064+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9a80066900 0x7f9a80068dc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.064+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a9c1542e0 0x7f9a9c15d5a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.064+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a9c0cc810 0x7f9a9c15d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.064+0000 7f9aa2c17640 1 --2- 172.21.15.7:0/3317688859 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9a9c0cbb70 0x7f9a9c156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.064+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 >> 172.21.15.7:0/3317688859 conn(0x7f9a9c0717e0 msgr2=0x7f9a9c0b4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:40.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.064+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 shutdown_connections 2023-11-28T18:48:40.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:40.064+0000 7f9aa2c17640 1 -- 172.21.15.7:0/3317688859 wait complete. 2023-11-28T18:48:40.085 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:41.378 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~202s 2023-11-28T18:48:41.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:48:41.818 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:48:42.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:42.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.236+0000 7f145c5aa640 1 Processor -- start 2023-11-28T18:48:42.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.236+0000 7f145c5aa640 1 -- start start 2023-11-28T18:48:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145c5aa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 0x7f14541575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145c5aa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f14540c7980 0x7f14540c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145c5aa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14540c6140 con 0x7f1454150b30 2023-11-28T18:48:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145c5aa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14540c72b0 con 0x7f14540c7980 2023-11-28T18:48:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145c5aa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1454070c00 con 0x7f14540c24f0 2023-11-28T18:48:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145ab47640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f14540c7980 0x7f14540c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145a346640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1454150b30 0x7f14540c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53968/0 (socket says 172.21.15.7:53968) 2023-11-28T18:48:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145a346640 1 -- 172.21.15.7:0/139090595 learned_addr learned my addr 172.21.15.7:0/139090595 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f1459b45640 1 --2- 172.21.15.7:0/139090595 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 0x7f14541575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145ab47640 1 -- 172.21.15.7:0/139090595 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 msgr2=0x7f14541575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145ab47640 1 --2- 172.21.15.7:0/139090595 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 0x7f14541575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145ab47640 1 -- 172.21.15.7:0/139090595 >> v1:172.21.15.7:6789/0 conn(0x7f1454150b30 legacy=0x7f14540c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.240+0000 7f145ab47640 1 -- 172.21.15.7:0/139090595 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1454157b00 con 0x7f14540c7980 2023-11-28T18:48:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145ab47640 1 --2- 172.21.15.7:0/139090595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14540c7980 0x7f14540c7e10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f144c009d90 tx=0x7f144c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdd04a40f455074 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:42.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f1459344640 1 -- 172.21.15.7:0/139090595 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c0af600 con 0x7f14540c7980 2023-11-28T18:48:42.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f1459344640 1 -- 172.21.15.7:0/139090595 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c0ab050 con 0x7f14540c7980 2023-11-28T18:48:42.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f1459344640 1 -- 172.21.15.7:0/139090595 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c0af980 con 0x7f14540c7980 2023-11-28T18:48:42.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 -- 172.21.15.7:0/139090595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14540c7980 msgr2=0x7f14540c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 --2- 172.21.15.7:0/139090595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14540c7980 0x7f14540c7e10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f144c009d90 tx=0x7f144c01f130 comp rx=0 tx=0).stop 2023-11-28T18:48:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 -- 172.21.15.7:0/139090595 shutdown_connections 2023-11-28T18:48:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 --2- 172.21.15.7:0/139090595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14540c7980 0x7f14540c7e10 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 --2- 172.21.15.7:0/139090595 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 0x7f14541575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 -- 172.21.15.7:0/139090595 >> 172.21.15.7:0/139090595 conn(0x7f14540662e0 msgr2=0x7f14540666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 -- 172.21.15.7:0/139090595 shutdown_connections 2023-11-28T18:48:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 -- 172.21.15.7:0/139090595 wait complete. 2023-11-28T18:48:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 Processor -- start 2023-11-28T18:48:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.244+0000 7f145c5aa640 1 -- start start 2023-11-28T18:48:42.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 0x7f1454153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14540c7980 0x7f1454154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1454150b30 0x7f1454155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1454156aa0 con 0x7f1454150b30 2023-11-28T18:48:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1454155c40 con 0x7f14540c7980 2023-11-28T18:48:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1454155f20 con 0x7f14540c24f0 2023-11-28T18:48:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1454150b30 0x7f1454155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1454150b30 0x7f1454155080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39088/0 (socket says 172.21.15.7:39088) 2023-11-28T18:48:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 -- 172.21.15.7:0/629686577 learned_addr learned my addr 172.21.15.7:0/629686577 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145a346640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 0x7f1454153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f1459b45640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14540c7980 0x7f1454154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 -- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 msgr2=0x7f1454153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 0x7f1454153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 -- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14540c7980 msgr2=0x7f1454154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14540c7980 0x7f1454154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 -- 172.21.15.7:0/629686577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c009a70 con 0x7f1454150b30 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f1459b45640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14540c7980 0x7f1454154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145ab47640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1454150b30 0x7f1454155080 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f144c0b2040 tx=0x7f144c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f144affd640 1 -- 172.21.15.7:0/629686577 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c0b9610 con 0x7f1454150b30 2023-11-28T18:48:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f144affd640 1 -- 172.21.15.7:0/629686577 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c0b9770 con 0x7f1454150b30 2023-11-28T18:48:42.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f144affd640 1 -- 172.21.15.7:0/629686577 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c0b9b20 con 0x7f1454150b30 2023-11-28T18:48:42.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1454156200 con 0x7f1454150b30 2023-11-28T18:48:42.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14541567c0 con 0x7f1454150b30 2023-11-28T18:48:42.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145404e890 con 0x7f1454150b30 2023-11-28T18:48:42.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.248+0000 7f144affd640 1 -- 172.21.15.7:0/629686577 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f144c0b6050 con 0x7f1454150b30 2023-11-28T18:48:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.252+0000 7f144affd640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14380668d0 0x7f1438068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.252+0000 7f144affd640 1 -- 172.21.15.7:0/629686577 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f144c122ed0 con 0x7f1454150b30 2023-11-28T18:48:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.252+0000 7f145a346640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14380668d0 0x7f1438068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.252+0000 7f145a346640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14380668d0 0x7f1438068d90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1450005890 tx=0x7f1450009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.256+0000 7f144affd640 1 -- 172.21.15.7:0/629686577 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f144c0efc80 con 0x7f1454150b30 2023-11-28T18:48:42.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.444+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14540c2d80 con 0x7f1454150b30 2023-11-28T18:48:42.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.444+0000 7f144affd640 1 -- 172.21.15.7:0/629686577 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f144c009450 con 0x7f1454150b30 2023-11-28T18:48:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:42.447 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:42.448 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14380668d0 msgr2=0x7f1438068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14380668d0 0x7f1438068d90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1450005890 tx=0x7f1450009040 comp rx=0 tx=0).stop 2023-11-28T18:48:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1454150b30 msgr2=0x7f1454155080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1454150b30 0x7f1454155080 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f144c0b2040 tx=0x7f144c01bed0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 shutdown_connections 2023-11-28T18:48:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f14380668d0 0x7f1438068d90 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1454150b30 0x7f1454155080 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14540c7980 0x7f1454154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 --2- 172.21.15.7:0/629686577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f14540c24f0 0x7f1454153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 >> 172.21.15.7:0/629686577 conn(0x7f14540662e0 msgr2=0x7f14540666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 shutdown_connections 2023-11-28T18:48:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:42.448+0000 7f145c5aa640 1 -- 172.21.15.7:0/629686577 wait complete. 2023-11-28T18:48:42.469 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:44.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:44.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.620+0000 7fbb1b1d1640 1 Processor -- start 2023-11-28T18:48:44.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.620+0000 7fbb1b1d1640 1 -- start start 2023-11-28T18:48:44.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb1b1d1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb1406e560 0x7fbb1415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:44.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb1b1d1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb14071e60 con 0x7fbb14157b90 2023-11-28T18:48:44.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb1b1d1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb14070930 con 0x7fbb14151cd0 2023-11-28T18:48:44.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb1b1d1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb14157fa0 con 0x7fbb1406e560 2023-11-28T18:48:44.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb13fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb1406e560 0x7fbb1415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:44.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb1976e640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbb14157b90 0x7fbb14162510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35366/0 (socket says 172.21.15.7:35366) 2023-11-28T18:48:44.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb1976e640 1 -- 172.21.15.7:0/504491177 learned_addr learned my addr 172.21.15.7:0/504491177 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:44.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb13fff640 1 -- 172.21.15.7:0/504491177 >> v1:172.21.15.193:6789/0 conn(0x7fbb14151cd0 legacy=0x7fbb140c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:44.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb13fff640 1 -- 172.21.15.7:0/504491177 >> v1:172.21.15.7:6789/0 conn(0x7fbb14157b90 legacy=0x7fbb14162510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:44.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.624+0000 7fbb13fff640 1 -- 172.21.15.7:0/504491177 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb14070bf0 con 0x7fbb1406e560 2023-11-28T18:48:44.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb13fff640 1 --2- 172.21.15.7:0/504491177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb1406e560 0x7fbb1415bd00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbb000097c0 tx=0x7fbb0001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e502d96ca83e095 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:44.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb137fe640 1 -- 172.21.15.7:0/504491177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb000ab600 con 0x7fbb1406e560 2023-11-28T18:48:44.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb137fe640 1 -- 172.21.15.7:0/504491177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb000a7050 con 0x7fbb1406e560 2023-11-28T18:48:44.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb137fe640 1 -- 172.21.15.7:0/504491177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb000ab980 con 0x7fbb1406e560 2023-11-28T18:48:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/504491177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb1406e560 msgr2=0x7fbb1415bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 --2- 172.21.15.7:0/504491177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb1406e560 0x7fbb1415bd00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbb000097c0 tx=0x7fbb0001c3f0 comp rx=0 tx=0).stop 2023-11-28T18:48:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/504491177 shutdown_connections 2023-11-28T18:48:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 --2- 172.21.15.7:0/504491177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb1406e560 0x7fbb1415bd00 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/504491177 >> 172.21.15.7:0/504491177 conn(0x7fbb140662e0 msgr2=0x7fbb140666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/504491177 shutdown_connections 2023-11-28T18:48:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/504491177 wait complete. 2023-11-28T18:48:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 Processor -- start 2023-11-28T18:48:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.628+0000 7fbb1b1d1640 1 -- start start 2023-11-28T18:48:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb1406e560 0x7fbb14150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb14151cd0 0x7fbb141529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb14157b90 0x7fbb141f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb14155070 con 0x7fbb14157b90 2023-11-28T18:48:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb14154150 con 0x7fbb1406e560 2023-11-28T18:48:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb141543f0 con 0x7fbb14151cd0 2023-11-28T18:48:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb14157b90 0x7fbb141f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb14157b90 0x7fbb141f45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38340/0 (socket says 172.21.15.7:38340) 2023-11-28T18:48:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 -- 172.21.15.7:0/3357826759 learned_addr learned my addr 172.21.15.7:0/3357826759 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb13fff640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb14151cd0 0x7fbb141529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb18f6d640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb1406e560 0x7fbb14150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 -- 172.21.15.7:0/3357826759 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb14151cd0 msgr2=0x7fbb141529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb14151cd0 0x7fbb141529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 -- 172.21.15.7:0/3357826759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb1406e560 msgr2=0x7fbb14150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb1406e560 0x7fbb14150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 -- 172.21.15.7:0/3357826759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb00009420 con 0x7fbb14157b90 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1976e640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb14157b90 0x7fbb141f45e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00ca80 tx=0x7fbb0c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb117fa640 1 -- 172.21.15.7:0/3357826759 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00ac30 con 0x7fbb14157b90 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb117fa640 1 -- 172.21.15.7:0/3357826759 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c00ad90 con 0x7fbb14157b90 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb141546d0 con 0x7fbb14157b90 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb117fa640 1 -- 172.21.15.7:0/3357826759 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c012600 con 0x7fbb14157b90 2023-11-28T18:48:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb14154c90 con 0x7fbb14157b90 2023-11-28T18:48:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.632+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb141551b0 con 0x7fbb14157b90 2023-11-28T18:48:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.636+0000 7fbb117fa640 1 -- 172.21.15.7:0/3357826759 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbb0c007070 con 0x7fbb14157b90 2023-11-28T18:48:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.636+0000 7fbb117fa640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbaf80668d0 0x7fbaf8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.636+0000 7fbb117fa640 1 -- 172.21.15.7:0/3357826759 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fbb0c084760 con 0x7fbb14157b90 2023-11-28T18:48:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.636+0000 7fbb18f6d640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbaf80668d0 0x7fbaf8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.636+0000 7fbb18f6d640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbaf80668d0 0x7fbaf8068d90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbb08000f70 tx=0x7fbb08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.640+0000 7fbb117fa640 1 -- 172.21.15.7:0/3357826759 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbb141551b0 con 0x7fbb14157b90 2023-11-28T18:48:44.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.828+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb141551b0 con 0x7fbb14157b90 2023-11-28T18:48:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.828+0000 7fbb117fa640 1 -- 172.21.15.7:0/3357826759 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fbb141551b0 con 0x7fbb14157b90 2023-11-28T18:48:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:44.832 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbaf80668d0 msgr2=0x7fbaf8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbaf80668d0 0x7fbaf8068d90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbb08000f70 tx=0x7fbb08009040 comp rx=0 tx=0).stop 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb14157b90 msgr2=0x7fbb141f45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb14157b90 0x7fbb141f45e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00ca80 tx=0x7fbb0c00cf40 comp rx=0 tx=0).stop 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 shutdown_connections 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbaf80668d0 0x7fbaf8068d90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb14157b90 0x7fbb141f45e0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbb14151cd0 0x7fbb141529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 --2- 172.21.15.7:0/3357826759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb1406e560 0x7fbb14150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 >> 172.21.15.7:0/3357826759 conn(0x7fbb140662e0 msgr2=0x7fbb140ca620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 shutdown_connections 2023-11-28T18:48:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:44.832+0000 7fbb1b1d1640 1 -- 172.21.15.7:0/3357826759 wait complete. 2023-11-28T18:48:44.854 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:46.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:47.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.000+0000 7fadd3e09640 1 Processor -- start 2023-11-28T18:48:47.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.000+0000 7fadd3e09640 1 -- start start 2023-11-28T18:48:47.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.000+0000 7fadd3e09640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fadcc151cd0 0x7fadcc15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:47.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.000+0000 7fadd3e09640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadcc06e560 0x7fadcc157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:47.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.000+0000 7fadd3e09640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc071e60 con 0x7fadcc0701f0 2023-11-28T18:48:47.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.000+0000 7fadd3e09640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadcc070930 con 0x7fadcc06e560 2023-11-28T18:48:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.000+0000 7fadd3e09640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadcc070bf0 con 0x7fadcc151cd0 2023-11-28T18:48:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd13a4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadcc06e560 0x7fadcc157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd13a4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadcc06e560 0x7fadcc157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38356/0 (socket says 172.21.15.7:38356) 2023-11-28T18:48:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd13a4640 1 -- 172.21.15.7:0/2302225225 learned_addr learned my addr 172.21.15.7:0/2302225225 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd1ba5640 1 --2- 172.21.15.7:0/2302225225 >> v2:172.21.15.193:3300/0 conn(0x7fadcc151cd0 0x7fadcc15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd13a4640 1 -- 172.21.15.7:0/2302225225 >> v2:172.21.15.193:3300/0 conn(0x7fadcc151cd0 msgr2=0x7fadcc15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd13a4640 1 --2- 172.21.15.7:0/2302225225 >> v2:172.21.15.193:3300/0 conn(0x7fadcc151cd0 0x7fadcc15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd13a4640 1 -- 172.21.15.7:0/2302225225 >> v1:172.21.15.193:6789/0 conn(0x7fadcc0701f0 legacy=0x7fadcc0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:47.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd13a4640 1 -- 172.21.15.7:0/2302225225 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc070ed0 con 0x7fadcc06e560 2023-11-28T18:48:47.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd13a4640 1 --2- 172.21.15.7:0/2302225225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc157b90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fadcc06fee0 tx=0x7fadbc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2925984521ca0073 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:47.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd0ba3640 1 -- 172.21.15.7:0/2302225225 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc0ad660 con 0x7fadcc06e560 2023-11-28T18:48:47.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd0ba3640 1 -- 172.21.15.7:0/2302225225 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc0ad7c0 con 0x7fadcc06e560 2023-11-28T18:48:47.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.004+0000 7fadd0ba3640 1 -- 172.21.15.7:0/2302225225 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc0adb50 con 0x7fadcc06e560 2023-11-28T18:48:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- 172.21.15.7:0/2302225225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 msgr2=0x7fadcc157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 --2- 172.21.15.7:0/2302225225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc157b90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fadcc06fee0 tx=0x7fadbc01c470 comp rx=0 tx=0).stop 2023-11-28T18:48:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- 172.21.15.7:0/2302225225 shutdown_connections 2023-11-28T18:48:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 --2- 172.21.15.7:0/2302225225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc157b90 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 --2- 172.21.15.7:0/2302225225 >> v2:172.21.15.193:3300/0 conn(0x7fadcc151cd0 0x7fadcc15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- 172.21.15.7:0/2302225225 >> 172.21.15.7:0/2302225225 conn(0x7fadcc0662e0 msgr2=0x7fadcc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:47.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- 172.21.15.7:0/2302225225 shutdown_connections 2023-11-28T18:48:47.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- 172.21.15.7:0/2302225225 wait complete. 2023-11-28T18:48:47.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 Processor -- start 2023-11-28T18:48:47.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- start start 2023-11-28T18:48:47.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:47.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadcc0701f0 0x7fadcc0c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:47.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadcc151cd0 0x7fadcc0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:47.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc147ce0 con 0x7fadcc06e560 2023-11-28T18:48:47.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc1f3250 con 0x7fadcc0701f0 2023-11-28T18:48:47.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd3e09640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc1f2640 con 0x7fadcc151cd0 2023-11-28T18:48:47.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd1ba5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:47.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd1ba5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc150f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38366/0 (socket says 172.21.15.7:38366) 2023-11-28T18:48:47.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd1ba5640 1 -- 172.21.15.7:0/3366348192 learned_addr learned my addr 172.21.15.7:0/3366348192 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd13a4640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadcc0701f0 0x7fadcc0c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd23a6640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadcc151cd0 0x7fadcc0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd1ba5640 1 -- 172.21.15.7:0/3366348192 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadcc151cd0 msgr2=0x7fadcc0c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd1ba5640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadcc151cd0 0x7fadcc0c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd1ba5640 1 -- 172.21.15.7:0/3366348192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadcc0701f0 msgr2=0x7fadcc0c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd1ba5640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadcc0701f0 0x7fadcc0c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.008+0000 7fadd1ba5640 1 -- 172.21.15.7:0/3366348192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc0094d0 con 0x7fadcc06e560 2023-11-28T18:48:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadd1ba5640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc150f30 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fadc800e870 tx=0x7fadc800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadc27fc640 1 -- 172.21.15.7:0/3366348192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8009530 con 0x7fadcc06e560 2023-11-28T18:48:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadc27fc640 1 -- 172.21.15.7:0/3366348192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc800c940 con 0x7fadcc06e560 2023-11-28T18:48:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadc27fc640 1 -- 172.21.15.7:0/3366348192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc800ccd0 con 0x7fadcc06e560 2023-11-28T18:48:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc1f2920 con 0x7fadcc06e560 2023-11-28T18:48:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc1f2ee0 con 0x7fadcc06e560 2023-11-28T18:48:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadcc1f30a0 con 0x7fadcc06e560 2023-11-28T18:48:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadc27fc640 1 -- 172.21.15.7:0/3366348192 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fadc8013470 con 0x7fadcc06e560 2023-11-28T18:48:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadc27fc640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fadb00668d0 0x7fadb0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.012+0000 7fadc27fc640 1 -- 172.21.15.7:0/3366348192 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fadc8084ab0 con 0x7fadcc06e560 2023-11-28T18:48:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.016+0000 7fadd13a4640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fadb00668d0 0x7fadb0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.016+0000 7fadd13a4640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fadb00668d0 0x7fadb0068d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0af000 tx=0x7fadbc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.016+0000 7fadc27fc640 1 -- 172.21.15.7:0/3366348192 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fadc80517f0 con 0x7fadcc06e560 2023-11-28T18:48:47.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.208+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadcc04e890 con 0x7fadcc06e560 2023-11-28T18:48:47.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.208+0000 7fadc27fc640 1 -- 172.21.15.7:0/3366348192 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fadc8017090 con 0x7fadcc06e560 2023-11-28T18:48:47.211 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:47.212 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fadb00668d0 msgr2=0x7fadb0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fadb00668d0 0x7fadb0068d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0af000 tx=0x7fadbc008040 comp rx=0 tx=0).stop 2023-11-28T18:48:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 msgr2=0x7fadcc150f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc150f30 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fadc800e870 tx=0x7fadc800ed30 comp rx=0 tx=0).stop 2023-11-28T18:48:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 shutdown_connections 2023-11-28T18:48:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fadb00668d0 0x7fadb0068d90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fadcc151cd0 0x7fadcc0c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadcc0701f0 0x7fadcc0c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 --2- 172.21.15.7:0/3366348192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadcc06e560 0x7fadcc150f30 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 >> 172.21.15.7:0/3366348192 conn(0x7fadcc0662e0 msgr2=0x7fadcc161c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:47.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 shutdown_connections 2023-11-28T18:48:47.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:47.212+0000 7fadd3e09640 1 -- 172.21.15.7:0/3366348192 wait complete. 2023-11-28T18:48:47.233 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:48:48.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~209s 2023-11-28T18:48:49.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46d3f8640 1 Processor -- start 2023-11-28T18:48:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46d3f8640 1 -- start start 2023-11-28T18:48:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46d3f8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe46815ce10 0x7fe4680ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46d3f8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4680cafe0 0x7fe4680ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46d3f8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe46815a3a0 con 0x7fe46815ca30 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46d3f8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46815a620 con 0x7fe4680cafe0 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46d3f8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46815a900 con 0x7fe46815ce10 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46cbf7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4680cafe0 0x7fe4680ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46cbf7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4680cafe0 0x7fe4680ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38370/0 (socket says 172.21.15.7:38370) 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46cbf7640 1 -- 172.21.15.7:0/2746073960 learned_addr learned my addr 172.21.15.7:0/2746073960 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe4677fe640 1 --2- 172.21.15.7:0/2746073960 >> v2:172.21.15.193:3300/0 conn(0x7fe46815ce10 0x7fe4680ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46cbf7640 1 -- 172.21.15.7:0/2746073960 >> v2:172.21.15.193:3300/0 conn(0x7fe46815ce10 msgr2=0x7fe4680ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46cbf7640 1 --2- 172.21.15.7:0/2746073960 >> v2:172.21.15.193:3300/0 conn(0x7fe46815ce10 0x7fe4680ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46cbf7640 1 -- 172.21.15.7:0/2746073960 >> v1:172.21.15.193:6789/0 conn(0x7fe46815ca30 legacy=0x7fe468159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.388+0000 7fe46cbf7640 1 -- 172.21.15.7:0/2746073960 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46815abe0 con 0x7fe4680cafe0 2023-11-28T18:48:49.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46cbf7640 1 --2- 172.21.15.7:0/2746073960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4680cafe0 0x7fe4680ca710 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe4680d0200 tx=0x7fe46001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ffa4d78f400047d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:49.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe466ffd640 1 -- 172.21.15.7:0/2746073960 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4600a5860 con 0x7fe4680cafe0 2023-11-28T18:48:49.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe466ffd640 1 -- 172.21.15.7:0/2746073960 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4600a59c0 con 0x7fe4680cafe0 2023-11-28T18:48:49.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe466ffd640 1 -- 172.21.15.7:0/2746073960 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4600a5d50 con 0x7fe4680cafe0 2023-11-28T18:48:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46d3f8640 1 -- 172.21.15.7:0/2746073960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4680cafe0 msgr2=0x7fe4680ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/2746073960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4680cafe0 0x7fe4680ca710 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe4680d0200 tx=0x7fe46001c9a0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46d3f8640 1 -- 172.21.15.7:0/2746073960 shutdown_connections 2023-11-28T18:48:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/2746073960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4680cafe0 0x7fe4680ca710 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/2746073960 >> v2:172.21.15.193:3300/0 conn(0x7fe46815ce10 0x7fe4680ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46d3f8640 1 -- 172.21.15.7:0/2746073960 >> 172.21.15.7:0/2746073960 conn(0x7fe46806c060 msgr2=0x7fe4680662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46d3f8640 1 -- 172.21.15.7:0/2746073960 shutdown_connections 2023-11-28T18:48:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.392+0000 7fe46d3f8640 1 -- 172.21.15.7:0/2746073960 wait complete. 2023-11-28T18:48:49.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 Processor -- start 2023-11-28T18:48:49.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 -- start start 2023-11-28T18:48:49.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe4680cafe0 0x7fe468152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:49.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46815ca30 0x7fe4681f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:49.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe46815ce10 0x7fe4681f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:49.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe468154980 con 0x7fe46815ce10 2023-11-28T18:48:49.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe468155d80 con 0x7fe46815ca30 2023-11-28T18:48:49.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681554d0 con 0x7fe4680cafe0 2023-11-28T18:48:49.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe46815ce10 0x7fe4681f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:49.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe46815ce10 0x7fe4681f3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38378/0 (socket says 172.21.15.7:38378) 2023-11-28T18:48:49.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 -- 172.21.15.7:0/11787188 learned_addr learned my addr 172.21.15.7:0/11787188 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:49.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe467fff640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe4680cafe0 0x7fe468152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:49.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe4677fe640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46815ca30 0x7fe4681f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:49.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 -- 172.21.15.7:0/11787188 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe4680cafe0 msgr2=0x7fe468152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:49.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe4680cafe0 0x7fe468152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 -- 172.21.15.7:0/11787188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46815ca30 msgr2=0x7fe4681f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:49.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46815ca30 0x7fe4681f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 -- 172.21.15.7:0/11787188 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460009b30 con 0x7fe46815ce10 2023-11-28T18:48:49.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46cbf7640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe46815ce10 0x7fe4681f3f10 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe4600af040 tx=0x7fe460019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:49.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe464ff9640 1 -- 172.21.15.7:0/11787188 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4600b8670 con 0x7fe46815ce10 2023-11-28T18:48:49.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe464ff9640 1 -- 172.21.15.7:0/11787188 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4600b87d0 con 0x7fe46815ce10 2023-11-28T18:48:49.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe464ff9640 1 -- 172.21.15.7:0/11787188 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4600b8b80 con 0x7fe46815ce10 2023-11-28T18:48:49.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe468150ff0 con 0x7fe46815ce10 2023-11-28T18:48:49.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4681546a0 con 0x7fe46815ce10 2023-11-28T18:48:49.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.396+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46804e890 con 0x7fe46815ce10 2023-11-28T18:48:49.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.400+0000 7fe464ff9640 1 -- 172.21.15.7:0/11787188 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe4680cce00 con 0x7fe46815ce10 2023-11-28T18:48:49.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.400+0000 7fe464ff9640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4540668d0 0x7fe454068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:49.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.400+0000 7fe464ff9640 1 -- 172.21.15.7:0/11787188 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fe46011fad0 con 0x7fe46815ce10 2023-11-28T18:48:49.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.400+0000 7fe467fff640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4540668d0 0x7fe454068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:49.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.400+0000 7fe467fff640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4540668d0 0x7fe454068d90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe458007fd0 tx=0x7fe458009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:49.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.404+0000 7fe464ff9640 1 -- 172.21.15.7:0/11787188 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe4600ec960 con 0x7fe46815ce10 2023-11-28T18:48:49.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.592+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4680cad10 con 0x7fe46815ce10 2023-11-28T18:48:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.592+0000 7fe464ff9640 1 -- 172.21.15.7:0/11787188 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fe4600eff80 con 0x7fe46815ce10 2023-11-28T18:48:49.596 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:49.596 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:49.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4540668d0 msgr2=0x7fe454068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4540668d0 0x7fe454068d90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe458007fd0 tx=0x7fe458009040 comp rx=0 tx=0).stop 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe46815ce10 msgr2=0x7fe4681f3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe46815ce10 0x7fe4681f3f10 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe4600af040 tx=0x7fe460019740 comp rx=0 tx=0).stop 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 shutdown_connections 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe4540668d0 0x7fe454068d90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe46815ce10 0x7fe4681f3f10 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46815ca30 0x7fe4681f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 --2- 172.21.15.7:0/11787188 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe4680cafe0 0x7fe468152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 >> 172.21.15.7:0/11787188 conn(0x7fe46806c060 msgr2=0x7fe468158170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 shutdown_connections 2023-11-28T18:48:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:49.596+0000 7fe46d3f8640 1 -- 172.21.15.7:0/11787188 wait complete. 2023-11-28T18:48:49.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:51.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.771+0000 7f7837d8d640 1 Processor -- start 2023-11-28T18:48:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.771+0000 7f7837d8d640 1 -- start start 2023-11-28T18:48:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7837d8d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783015bd00 0x7f7830157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7837d8d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7830071e60 con 0x7f7830151cd0 2023-11-28T18:48:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7837d8d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7830070930 con 0x7f783006e560 2023-11-28T18:48:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7837d8d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783015c150 con 0x7f783015bd00 2023-11-28T18:48:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7835b29640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7830151cd0 0x7f78300c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35370/0 (socket says 172.21.15.7:35370) 2023-11-28T18:48:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7835b29640 1 -- 172.21.15.7:0/2276468918 learned_addr learned my addr 172.21.15.7:0/2276468918 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f783632a640 1 --2- 172.21.15.7:0/2276468918 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783015bd00 0x7f7830157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7834b27640 1 -- 172.21.15.7:0/2276468918 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194507014 0 0) 0x7f7830071e60 con 0x7f7830151cd0 2023-11-28T18:48:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7834b27640 1 -- 172.21.15.7:0/2276468918 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f781c003530 con 0x7f7830151cd0 2023-11-28T18:48:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7834b27640 1 -- 172.21.15.7:0/2276468918 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207608906 0 0) 0x7f7830070930 con 0x7f783006e560 2023-11-28T18:48:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7834b27640 1 -- 172.21.15.7:0/2276468918 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7830071e60 con 0x7f783006e560 2023-11-28T18:48:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f7834b27640 1 -- 172.21.15.7:0/2276468918 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708675398 0 0) 0x7f781c003530 con 0x7f7830151cd0 2023-11-28T18:48:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f783632a640 1 -- 172.21.15.7:0/2276468918 >> v1:172.21.15.193:6789/0 conn(0x7f783006e560 legacy=0x7f7830160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f783632a640 1 -- 172.21.15.7:0/2276468918 >> v1:172.21.15.7:6789/0 conn(0x7f7830151cd0 legacy=0x7f78300c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.775+0000 7f783632a640 1 -- 172.21.15.7:0/2276468918 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830070bf0 con 0x7f783015bd00 2023-11-28T18:48:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f783632a640 1 --2- 172.21.15.7:0/2276468918 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783015bd00 0x7f7830157b90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7828009dc0 tx=0x7f782801c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d78c95253ef8baee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7834b27640 1 -- 172.21.15.7:0/2276468918 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c001680 con 0x7f783015bd00 2023-11-28T18:48:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7834b27640 1 -- 172.21.15.7:0/2276468918 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78280a55c0 con 0x7f783015bd00 2023-11-28T18:48:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7834b27640 1 -- 172.21.15.7:0/2276468918 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78280a5950 con 0x7f783015bd00 2023-11-28T18:48:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 -- 172.21.15.7:0/2276468918 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783015bd00 msgr2=0x7f7830157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 --2- 172.21.15.7:0/2276468918 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783015bd00 0x7f7830157b90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7828009dc0 tx=0x7f782801c890 comp rx=0 tx=0).stop 2023-11-28T18:48:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 -- 172.21.15.7:0/2276468918 shutdown_connections 2023-11-28T18:48:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 --2- 172.21.15.7:0/2276468918 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783015bd00 0x7f7830157b90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 -- 172.21.15.7:0/2276468918 >> 172.21.15.7:0/2276468918 conn(0x7f78300662e0 msgr2=0x7f78300666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 -- 172.21.15.7:0/2276468918 shutdown_connections 2023-11-28T18:48:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 -- 172.21.15.7:0/2276468918 wait complete. 2023-11-28T18:48:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 Processor -- start 2023-11-28T18:48:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.779+0000 7f7837d8d640 1 -- start start 2023-11-28T18:48:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783006e560 0x7f78300c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7830151cd0 0x7f783015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f783015bd00 0x7f7830155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78301f62f0 con 0x7f783015bd00 2023-11-28T18:48:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78301f6540 con 0x7f7830151cd0 2023-11-28T18:48:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78301f6820 con 0x7f783006e560 2023-11-28T18:48:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f783015bd00 0x7f7830155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f783015bd00 0x7f7830155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38386/0 (socket says 172.21.15.7:38386) 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 -- 172.21.15.7:0/2116458337 learned_addr learned my addr 172.21.15.7:0/2116458337 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7835b29640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783006e560 0x7f78300c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7835328640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7830151cd0 0x7f783015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 -- 172.21.15.7:0/2116458337 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783006e560 msgr2=0x7f78300c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783006e560 0x7f78300c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 -- 172.21.15.7:0/2116458337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7830151cd0 msgr2=0x7f783015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7830151cd0 0x7f783015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 -- 172.21.15.7:0/2116458337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7828009a70 con 0x7f783015bd00 2023-11-28T18:48:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f783632a640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f783015bd00 0x7f7830155bd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f782801cd70 tx=0x7f7828019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f78267fc640 1 -- 172.21.15.7:0/2116458337 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78280ad640 con 0x7f783015bd00 2023-11-28T18:48:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f78267fc640 1 -- 172.21.15.7:0/2116458337 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78280ad7a0 con 0x7f783015bd00 2023-11-28T18:48:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f78267fc640 1 -- 172.21.15.7:0/2116458337 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78280adb50 con 0x7f783015bd00 2023-11-28T18:48:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78301f6b00 con 0x7f783015bd00 2023-11-28T18:48:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78301f70c0 con 0x7f783015bd00 2023-11-28T18:48:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f783004e890 con 0x7f783015bd00 2023-11-28T18:48:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.783+0000 7f78267fc640 1 -- 172.21.15.7:0/2116458337 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f78280b4050 con 0x7f783015bd00 2023-11-28T18:48:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.787+0000 7f78267fc640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f78140668d0 0x7f7814068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.787+0000 7f78267fc640 1 -- 172.21.15.7:0/2116458337 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f782811f440 con 0x7f783015bd00 2023-11-28T18:48:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.787+0000 7f7835b29640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f78140668d0 0x7f7814068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.787+0000 7f7835b29640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f78140668d0 0x7f7814068d90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f782c001ec0 tx=0x7f782c00f5a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.791+0000 7f78267fc640 1 -- 172.21.15.7:0/2116458337 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f78280ec2a0 con 0x7f783015bd00 2023-11-28T18:48:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.979+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78300c26e0 con 0x7f783015bd00 2023-11-28T18:48:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.979+0000 7f78267fc640 1 -- 172.21.15.7:0/2116458337 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f78300c26e0 con 0x7f783015bd00 2023-11-28T18:48:51.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f78140668d0 msgr2=0x7f7814068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f78140668d0 0x7f7814068d90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f782c001ec0 tx=0x7f782c00f5a0 comp rx=0 tx=0).stop 2023-11-28T18:48:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f783015bd00 msgr2=0x7f7830155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f783015bd00 0x7f7830155bd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f782801cd70 tx=0x7f7828019630 comp rx=0 tx=0).stop 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 shutdown_connections 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f78140668d0 0x7f7814068d90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f783015bd00 0x7f7830155bd0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7830151cd0 0x7f783015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 --2- 172.21.15.7:0/2116458337 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f783006e560 0x7f78300c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 >> 172.21.15.7:0/2116458337 conn(0x7f78300662e0 msgr2=0x7f78300c51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 shutdown_connections 2023-11-28T18:48:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:51.983+0000 7f7837d8d640 1 -- 172.21.15.7:0/2116458337 wait complete. 2023-11-28T18:48:52.006 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:54.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:54.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.159+0000 7f5efc0fe640 1 Processor -- start 2023-11-28T18:48:54.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.159+0000 7f5efc0fe640 1 -- start start 2023-11-28T18:48:54.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.159+0000 7f5efc0fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef40c6ca0 0x7f5ef4161f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:54.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.159+0000 7f5efc0fe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef40c2970 con 0x7f5ef415e300 2023-11-28T18:48:54.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.159+0000 7f5efc0fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef415e820 con 0x7f5ef40c68c0 2023-11-28T18:48:54.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.159+0000 7f5efc0fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef415eb00 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5efa69b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5ef415e300 0x7f5ef4157eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57330/0 (socket says 172.21.15.7:57330) 2023-11-28T18:48:54.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5efa69b640 1 -- 172.21.15.7:0/901192057 learned_addr learned my addr 172.21.15.7:0/901192057 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef9699640 1 --2- 172.21.15.7:0/901192057 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef40c6ca0 0x7f5ef4161f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef8e98640 1 -- 172.21.15.7:0/901192057 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586237399 0 0) 0x7f5ef40c2970 con 0x7f5ef415e300 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef8e98640 1 -- 172.21.15.7:0/901192057 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee0003530 con 0x7f5ef415e300 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef8e98640 1 -- 172.21.15.7:0/901192057 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736356446 0 0) 0x7f5ef415e820 con 0x7f5ef40c68c0 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef8e98640 1 -- 172.21.15.7:0/901192057 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ef40c2970 con 0x7f5ef40c68c0 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef8e98640 1 -- 172.21.15.7:0/901192057 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2966121870 0 0) 0x7f5ee0003530 con 0x7f5ef415e300 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef9699640 1 -- 172.21.15.7:0/901192057 >> v1:172.21.15.193:6789/0 conn(0x7f5ef40c68c0 legacy=0x7f5ef40c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef9699640 1 -- 172.21.15.7:0/901192057 >> v1:172.21.15.7:6789/0 conn(0x7f5ef415e300 legacy=0x7f5ef4157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef9699640 1 -- 172.21.15.7:0/901192057 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef415ede0 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.163+0000 7f5ef9699640 1 --2- 172.21.15.7:0/901192057 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef40c6ca0 0x7f5ef4161f90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40097f0 tx=0x7f5ee401ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8759d6dc1923a52 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:54.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5ef8e98640 1 -- 172.21.15.7:0/901192057 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec001980 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5ef8e98640 1 -- 172.21.15.7:0/901192057 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee40a8650 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5ef8e98640 1 -- 172.21.15.7:0/901192057 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40a89e0 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- 172.21.15.7:0/901192057 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef40c6ca0 msgr2=0x7f5ef4161f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 --2- 172.21.15.7:0/901192057 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef40c6ca0 0x7f5ef4161f90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40097f0 tx=0x7f5ee401ebd0 comp rx=0 tx=0).stop 2023-11-28T18:48:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- 172.21.15.7:0/901192057 shutdown_connections 2023-11-28T18:48:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 --2- 172.21.15.7:0/901192057 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef40c6ca0 0x7f5ef4161f90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- 172.21.15.7:0/901192057 >> 172.21.15.7:0/901192057 conn(0x7f5ef40662e0 msgr2=0x7f5ef40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- 172.21.15.7:0/901192057 shutdown_connections 2023-11-28T18:48:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- 172.21.15.7:0/901192057 wait complete. 2023-11-28T18:48:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 Processor -- start 2023-11-28T18:48:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- start start 2023-11-28T18:48:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ef40c68c0 0x7f5ef4157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ef40c6ca0 0x7f5ef40caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef415e300 0x7f5ef41526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef4155710 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef41559b0 con 0x7f5ef40c68c0 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5efc0fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef4155c90 con 0x7f5ef415e300 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5ef9699640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ef40c6ca0 0x7f5ef40caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5ef9699640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ef40c6ca0 0x7f5ef40caa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47944/0 (socket says 172.21.15.7:47944) 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.167+0000 7f5ef9699640 1 -- 172.21.15.7:0/1741744840 learned_addr learned my addr 172.21.15.7:0/1741744840 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5ef9699640 1 -- 172.21.15.7:0/1741744840 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef415e300 msgr2=0x7f5ef41526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5ef9699640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef415e300 0x7f5ef41526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5ef9699640 1 -- 172.21.15.7:0/1741744840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ef40c68c0 msgr2=0x7f5ef4157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5ef9e9a640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ef40c68c0 0x7f5ef4157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5ef9699640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ef40c68c0 0x7f5ef4157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5ef9699640 1 -- 172.21.15.7:0/1741744840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee4009420 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5ef9e9a640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ef40c68c0 0x7f5ef4157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:48:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5ef9699640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ef40c6ca0 0x7f5ef40caa40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40097f0 tx=0x7f5ee401b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5eea7fc640 1 -- 172.21.15.7:0/1741744840 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40bf070 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5eea7fc640 1 -- 172.21.15.7:0/1741744840 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee4006e20 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5eea7fc640 1 -- 172.21.15.7:0/1741744840 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40b0870 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef4155f70 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef4156530 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef4156720 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.171+0000 7f5eea7fc640 1 -- 172.21.15.7:0/1741744840 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5ee40b7040 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.175+0000 7f5eea7fc640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5ed8066900 0x7f5ed8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.175+0000 7f5eea7fc640 1 -- 172.21.15.7:0/1741744840 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f5ee4121580 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.175+0000 7f5ef9e9a640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5ed8066900 0x7f5ed8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:54.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.175+0000 7f5ef9e9a640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5ed8066900 0x7f5ed8068dc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0000f70 tx=0x7f5ef0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:54.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.179+0000 7f5eea7fc640 1 -- 172.21.15.7:0/1741744840 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5ee40ee330 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.367+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ef404e890 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.367+0000 7f5eea7fc640 1 -- 172.21.15.7:0/1741744840 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f5ee40f1950 con 0x7f5ef40c6ca0 2023-11-28T18:48:54.371 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:54.371 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5ed8066900 msgr2=0x7f5ed8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5ed8066900 0x7f5ed8068dc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0000f70 tx=0x7f5ef0009040 comp rx=0 tx=0).stop 2023-11-28T18:48:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ef40c6ca0 msgr2=0x7f5ef40caa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ef40c6ca0 0x7f5ef40caa40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40097f0 tx=0x7f5ee401b970 comp rx=0 tx=0).stop 2023-11-28T18:48:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 shutdown_connections 2023-11-28T18:48:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f5ed8066900 0x7f5ed8068dc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f5ef415e300 0x7f5ef41526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ef40c6ca0 0x7f5ef40caa40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 --2- 172.21.15.7:0/1741744840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ef40c68c0 0x7f5ef4157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 >> 172.21.15.7:0/1741744840 conn(0x7f5ef40662e0 msgr2=0x7f5ef406d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 shutdown_connections 2023-11-28T18:48:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:54.371+0000 7f5efc0fe640 1 -- 172.21.15.7:0/1741744840 wait complete. 2023-11-28T18:48:54.393 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:54.804 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~215s 2023-11-28T18:48:56.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.543+0000 7f89b0f57640 1 Processor -- start 2023-11-28T18:48:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.543+0000 7f89b0f57640 1 -- start start 2023-11-28T18:48:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89b0f57640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 0x7f89ac1575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89b0f57640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89ac0c7980 0x7f89ac0c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89b0f57640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ac0c6140 con 0x7f89ac150b30 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89b0f57640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0c72b0 con 0x7f89ac0c7980 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89b0f57640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac070c00 con 0x7f89ac0c24f0 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89aad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89ac0c7980 0x7f89ac0c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89aa59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f89ac150b30 0x7f89ac0c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57334/0 (socket says 172.21.15.7:57334) 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89aa59c640 1 -- 172.21.15.7:0/2622492048 learned_addr learned my addr 172.21.15.7:0/2622492048 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89a3fff640 1 --2- 172.21.15.7:0/2622492048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 0x7f89ac1575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89aad9d640 1 -- 172.21.15.7:0/2622492048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 msgr2=0x7f89ac1575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:56.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89aad9d640 1 --2- 172.21.15.7:0/2622492048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 0x7f89ac1575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89aad9d640 1 -- 172.21.15.7:0/2622492048 >> v1:172.21.15.7:6789/0 conn(0x7f89ac150b30 legacy=0x7f89ac0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:56.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89aad9d640 1 -- 172.21.15.7:0/2622492048 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac157b00 con 0x7f89ac0c7980 2023-11-28T18:48:56.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.547+0000 7f89a9d9b640 1 -- 172.21.15.7:0/2622492048 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515478578 0 0) 0x7f89ac0c6140 con 0x7f89ac150b30 2023-11-28T18:48:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89aad9d640 1 --2- 172.21.15.7:0/2622492048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac0c7980 0x7f89ac0c7e10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009d90 tx=0x7f89a401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd1ff6d486a9919 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:56.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89a9d9b640 1 -- 172.21.15.7:0/2622492048 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40af600 con 0x7f89ac0c7980 2023-11-28T18:48:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89a9d9b640 1 -- 172.21.15.7:0/2622492048 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a40ab050 con 0x7f89ac0c7980 2023-11-28T18:48:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89a9d9b640 1 -- 172.21.15.7:0/2622492048 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40af980 con 0x7f89ac0c7980 2023-11-28T18:48:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 -- 172.21.15.7:0/2622492048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac0c7980 msgr2=0x7f89ac0c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2622492048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac0c7980 0x7f89ac0c7e10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009d90 tx=0x7f89a401f130 comp rx=0 tx=0).stop 2023-11-28T18:48:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 -- 172.21.15.7:0/2622492048 shutdown_connections 2023-11-28T18:48:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2622492048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac0c7980 0x7f89ac0c7e10 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2622492048 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 0x7f89ac1575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 -- 172.21.15.7:0/2622492048 >> 172.21.15.7:0/2622492048 conn(0x7f89ac0662e0 msgr2=0x7f89ac0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 -- 172.21.15.7:0/2622492048 shutdown_connections 2023-11-28T18:48:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 -- 172.21.15.7:0/2622492048 wait complete. 2023-11-28T18:48:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 Processor -- start 2023-11-28T18:48:56.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.551+0000 7f89b0f57640 1 -- start start 2023-11-28T18:48:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 0x7f89ac1e1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac0c7980 0x7f89ac155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac150b30 0x7f89ac155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac157240 con 0x7f89ac150b30 2023-11-28T18:48:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac143090 con 0x7f89ac0c7980 2023-11-28T18:48:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1f3a70 con 0x7f89ac0c24f0 2023-11-28T18:48:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac150b30 0x7f89ac155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac150b30 0x7f89ac155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47958/0 (socket says 172.21.15.7:47958) 2023-11-28T18:48:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 -- 172.21.15.7:0/2339899385 learned_addr learned my addr 172.21.15.7:0/2339899385 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aa59c640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 0x7f89ac1e1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89a3fff640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac0c7980 0x7f89ac155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 -- 172.21.15.7:0/2339899385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 msgr2=0x7f89ac1e1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 0x7f89ac1e1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 -- 172.21.15.7:0/2339899385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac0c7980 msgr2=0x7f89ac155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:56.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac0c7980 0x7f89ac155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 -- 172.21.15.7:0/2339899385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4009a70 con 0x7f89ac150b30 2023-11-28T18:48:56.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89aad9d640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac150b30 0x7f89ac155c80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f89a40b2040 tx=0x7f89a401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:56.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89a2ffd640 1 -- 172.21.15.7:0/2339899385 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40b9610 con 0x7f89ac150b30 2023-11-28T18:48:56.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89a2ffd640 1 -- 172.21.15.7:0/2339899385 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a40b9770 con 0x7f89ac150b30 2023-11-28T18:48:56.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89a2ffd640 1 -- 172.21.15.7:0/2339899385 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40b9b20 con 0x7f89ac150b30 2023-11-28T18:48:56.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac071c10 con 0x7f89ac150b30 2023-11-28T18:48:56.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac154760 con 0x7f89ac150b30 2023-11-28T18:48:56.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac04e890 con 0x7f89ac150b30 2023-11-28T18:48:56.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.555+0000 7f89a2ffd640 1 -- 172.21.15.7:0/2339899385 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f89a40b6050 con 0x7f89ac150b30 2023-11-28T18:48:56.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.559+0000 7f89a2ffd640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f898c0668d0 0x7f898c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:56.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.559+0000 7f89a2ffd640 1 -- 172.21.15.7:0/2339899385 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f89a4122e60 con 0x7f89ac150b30 2023-11-28T18:48:56.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.559+0000 7f89aa59c640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f898c0668d0 0x7f898c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:56.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.559+0000 7f89aa59c640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f898c0668d0 0x7f898c068d90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f899c007fd0 tx=0x7f899c007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:56.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.563+0000 7f89a2ffd640 1 -- 172.21.15.7:0/2339899385 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f89a40efde0 con 0x7f89ac150b30 2023-11-28T18:48:56.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.751+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89ac0c2d80 con 0x7f89ac150b30 2023-11-28T18:48:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.751+0000 7f89a2ffd640 1 -- 172.21.15.7:0/2339899385 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f89a4009450 con 0x7f89ac150b30 2023-11-28T18:48:56.756 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:56.756 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f898c0668d0 msgr2=0x7f898c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f898c0668d0 0x7f898c068d90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f899c007fd0 tx=0x7f899c007d60 comp rx=0 tx=0).stop 2023-11-28T18:48:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac150b30 msgr2=0x7f89ac155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac150b30 0x7f89ac155c80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f89a40b2040 tx=0x7f89a401bed0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 shutdown_connections 2023-11-28T18:48:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f898c0668d0 0x7f898c068d90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89ac150b30 0x7f89ac155c80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac0c7980 0x7f89ac155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 --2- 172.21.15.7:0/2339899385 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f89ac0c24f0 0x7f89ac1e1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:56.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 >> 172.21.15.7:0/2339899385 conn(0x7f89ac0662e0 msgr2=0x7f89ac06e250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:56.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 shutdown_connections 2023-11-28T18:48:56.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:56.755+0000 7f89b0f57640 1 -- 172.21.15.7:0/2339899385 wait complete. 2023-11-28T18:48:56.780 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:48:58.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:48:58.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5de17640 1 Processor -- start 2023-11-28T18:48:58.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5de17640 1 -- start start 2023-11-28T18:48:58.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5de17640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b58153ee0 0x7f7b5815a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:58.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5de17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 0x7f7b580cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5de17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5815ded0 con 0x7f7b580c9f60 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5de17640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5815ac30 con 0x7f7b58153ee0 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5de17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5815d300 con 0x7f7b580ccad0 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5ce15640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b58153ee0 0x7f7b5815a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b57fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7b580c9f60 0x7f7b58071d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57340/0 (socket says 172.21.15.7:57340) 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b57fff640 1 -- 172.21.15.7:0/3692777555 learned_addr learned my addr 172.21.15.7:0/3692777555 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5d616640 1 --2- 172.21.15.7:0/3692777555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 0x7f7b580cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5ce15640 1 -- 172.21.15.7:0/3692777555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 msgr2=0x7f7b580cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:48:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5ce15640 1 --2- 172.21.15.7:0/3692777555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 0x7f7b580cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5ce15640 1 -- 172.21.15.7:0/3692777555 >> v1:172.21.15.7:6789/0 conn(0x7f7b580c9f60 legacy=0x7f7b58071d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.935+0000 7f7b5ce15640 1 -- 172.21.15.7:0/3692777555 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5815d5a0 con 0x7f7b58153ee0 2023-11-28T18:48:58.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5ce15640 1 --2- 172.21.15.7:0/3692777555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815a6f0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0097c0 tx=0x7f7b4c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf12eb1489fc6ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b577fe640 1 -- 172.21.15.7:0/3692777555 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c0ab650 con 0x7f7b58153ee0 2023-11-28T18:48:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b577fe640 1 -- 172.21.15.7:0/3692777555 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4c0ab7b0 con 0x7f7b58153ee0 2023-11-28T18:48:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b577fe640 1 -- 172.21.15.7:0/3692777555 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c0abb40 con 0x7f7b58153ee0 2023-11-28T18:48:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 -- 172.21.15.7:0/3692777555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 msgr2=0x7f7b5815a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 --2- 172.21.15.7:0/3692777555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815a6f0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0097c0 tx=0x7f7b4c01c450 comp rx=0 tx=0).stop 2023-11-28T18:48:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 -- 172.21.15.7:0/3692777555 shutdown_connections 2023-11-28T18:48:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 --2- 172.21.15.7:0/3692777555 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 0x7f7b580cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 --2- 172.21.15.7:0/3692777555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815a6f0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 -- 172.21.15.7:0/3692777555 >> 172.21.15.7:0/3692777555 conn(0x7f7b5806c260 msgr2=0x7f7b58070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:58.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 -- 172.21.15.7:0/3692777555 shutdown_connections 2023-11-28T18:48:58.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 -- 172.21.15.7:0/3692777555 wait complete. 2023-11-28T18:48:58.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 Processor -- start 2023-11-28T18:48:58.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.939+0000 7f7b5de17640 1 -- start start 2023-11-28T18:48:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b580c9f60 0x7f7b580b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 0x7f7b581ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58141010 con 0x7f7b58153ee0 2023-11-28T18:48:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b580b41f0 con 0x7f7b580c9f60 2023-11-28T18:48:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b581f2360 con 0x7f7b580ccad0 2023-11-28T18:48:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:58.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47982/0 (socket says 172.21.15.7:47982) 2023-11-28T18:48:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 -- 172.21.15.7:0/4093709868 learned_addr learned my addr 172.21.15.7:0/4093709868 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:48:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b57fff640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 0x7f7b581ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5ce15640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b580c9f60 0x7f7b580b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 -- 172.21.15.7:0/4093709868 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 msgr2=0x7f7b581ebb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 0x7f7b581ebb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 -- 172.21.15.7:0/4093709868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b580c9f60 msgr2=0x7f7b580b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b580c9f60 0x7f7b580b2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 -- 172.21.15.7:0/4093709868 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c009420 con 0x7f7b58153ee0 2023-11-28T18:48:58.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5d616640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815c970 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7b50007e40 tx=0x7f7b5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:58.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b557fa640 1 -- 172.21.15.7:0/4093709868 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50014620 con 0x7f7b58153ee0 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b557fa640 1 -- 172.21.15.7:0/4093709868 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b500147a0 con 0x7f7b58153ee0 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b58140820 con 0x7f7b58153ee0 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b557fa640 1 -- 172.21.15.7:0/4093709868 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50014b50 con 0x7f7b58153ee0 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b580b2e80 con 0x7f7b58153ee0 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.943+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b58140ab0 con 0x7f7b58153ee0 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.947+0000 7f7b557fa640 1 -- 172.21.15.7:0/4093709868 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7b50014cb0 con 0x7f7b58153ee0 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.947+0000 7f7b557fa640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b3c0668d0 0x7f7b3c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.947+0000 7f7b557fa640 1 -- 172.21.15.7:0/4093709868 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f7b50085030 con 0x7f7b58153ee0 2023-11-28T18:48:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.947+0000 7f7b5ce15640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b3c0668d0 0x7f7b3c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:48:58.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.947+0000 7f7b5ce15640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b3c0668d0 0x7f7b3c068d90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0af000 tx=0x7f7b4c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:48:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:58.951+0000 7f7b557fa640 1 -- 172.21.15.7:0/4093709868 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7b500b6280 con 0x7f7b58153ee0 2023-11-28T18:48:59.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.143+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b580b2e80 con 0x7f7b58153ee0 2023-11-28T18:48:59.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.143+0000 7f7b557fa640 1 -- 172.21.15.7:0/4093709868 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f7b580b2e80 con 0x7f7b58153ee0 2023-11-28T18:48:59.148 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:48:59.148 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:48:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:48:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b3c0668d0 msgr2=0x7f7b3c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b3c0668d0 0x7f7b3c068d90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0af000 tx=0x7f7b4c0a7000 comp rx=0 tx=0).stop 2023-11-28T18:48:59.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 msgr2=0x7f7b5815c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815c970 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7b50007e40 tx=0x7f7b5000e4b0 comp rx=0 tx=0).stop 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 shutdown_connections 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7b3c0668d0 0x7f7b3c068d90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b58153ee0 0x7f7b5815c970 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7b580ccad0 0x7f7b581ebb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 --2- 172.21.15.7:0/4093709868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b580c9f60 0x7f7b580b2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 >> 172.21.15.7:0/4093709868 conn(0x7f7b5806c260 msgr2=0x7f7b58143710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 shutdown_connections 2023-11-28T18:48:59.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:48:59.147+0000 7f7b5de17640 1 -- 172.21.15.7:0/4093709868 wait complete. 2023-11-28T18:48:59.170 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:49:01.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.319+0000 7f9939e79640 1 Processor -- start 2023-11-28T18:49:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9939e79640 1 -- start start 2023-11-28T18:49:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9939e79640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f99341542e0 0x7f993415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9939e79640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f99340cbb70 0x7f99340cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9939e79640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cc810 0x7f9934072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9939e79640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9934071d50 con 0x7f99341542e0 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9939e79640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9934072030 con 0x7f99340cbb70 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9939e79640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99340ca920 con 0x7f99340cc810 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f99337fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f99341542e0 0x7f993415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f99337fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f99341542e0 0x7f993415e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47994/0 (socket says 172.21.15.7:47994) 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f99337fe640 1 -- 172.21.15.7:0/2500170754 learned_addr learned my addr 172.21.15.7:0/2500170754 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9932ffd640 1 --2- 172.21.15.7:0/2500170754 >> v2:172.21.15.193:3300/0 conn(0x7f99340cbb70 0x7f99340cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f9933fff640 1 --2- 172.21.15.7:0/2500170754 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cc810 0x7f9934072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f99337fe640 1 -- 172.21.15.7:0/2500170754 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cc810 msgr2=0x7f9934072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f99337fe640 1 --2- 172.21.15.7:0/2500170754 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cc810 0x7f9934072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f99337fe640 1 -- 172.21.15.7:0/2500170754 >> v2:172.21.15.193:3300/0 conn(0x7f99340cbb70 msgr2=0x7f99340cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f99337fe640 1 --2- 172.21.15.7:0/2500170754 >> v2:172.21.15.193:3300/0 conn(0x7f99340cbb70 0x7f99340cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.323+0000 7f99337fe640 1 -- 172.21.15.7:0/2500170754 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993415e550 con 0x7f99341542e0 2023-11-28T18:49:01.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f99337fe640 1 --2- 172.21.15.7:0/2500170754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99341542e0 0x7f993415e010 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f99340ca8b0 tx=0x7f992401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac28f66039ed1588 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:01.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f99327fc640 1 -- 172.21.15.7:0/2500170754 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99240ac630 con 0x7f99341542e0 2023-11-28T18:49:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f99327fc640 1 -- 172.21.15.7:0/2500170754 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99240ac790 con 0x7f99341542e0 2023-11-28T18:49:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f99327fc640 1 -- 172.21.15.7:0/2500170754 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99240acb20 con 0x7f99341542e0 2023-11-28T18:49:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 -- 172.21.15.7:0/2500170754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99341542e0 msgr2=0x7f993415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 --2- 172.21.15.7:0/2500170754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99341542e0 0x7f993415e010 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f99340ca8b0 tx=0x7f992401c500 comp rx=0 tx=0).stop 2023-11-28T18:49:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 -- 172.21.15.7:0/2500170754 shutdown_connections 2023-11-28T18:49:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 --2- 172.21.15.7:0/2500170754 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cc810 0x7f9934072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 --2- 172.21.15.7:0/2500170754 >> v2:172.21.15.193:3300/0 conn(0x7f99340cbb70 0x7f99340cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 --2- 172.21.15.7:0/2500170754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99341542e0 0x7f993415e010 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 -- 172.21.15.7:0/2500170754 >> 172.21.15.7:0/2500170754 conn(0x7f99340717e0 msgr2=0x7f99340662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 -- 172.21.15.7:0/2500170754 shutdown_connections 2023-11-28T18:49:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 -- 172.21.15.7:0/2500170754 wait complete. 2023-11-28T18:49:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 Processor -- start 2023-11-28T18:49:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.327+0000 7f9939e79640 1 -- start start 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cbb70 0x7f99340c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99340cc810 0x7f993415d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99341542e0 0x7f993415de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993415e370 con 0x7f99340cc810 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993415d3b0 con 0x7f99341542e0 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99341f6350 con 0x7f99340cbb70 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99340cc810 0x7f993415d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99340cc810 0x7f993415d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48002/0 (socket says 172.21.15.7:48002) 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 -- 172.21.15.7:0/3355906935 learned_addr learned my addr 172.21.15.7:0/3355906935 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f99337fe640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cbb70 0x7f99340c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9933fff640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99341542e0 0x7f993415de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 -- 172.21.15.7:0/3355906935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cbb70 msgr2=0x7f99340c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cbb70 0x7f99340c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 -- 172.21.15.7:0/3355906935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99341542e0 msgr2=0x7f993415de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99341542e0 0x7f993415de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 -- 172.21.15.7:0/3355906935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240094d0 con 0x7f99340cc810 2023-11-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9932ffd640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99340cc810 0x7f993415d8f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f992800d800 tx=0x7f992800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f991bfff640 1 -- 172.21.15.7:0/3355906935 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99280096a0 con 0x7f99340cc810 2023-11-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f991bfff640 1 -- 172.21.15.7:0/3355906935 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9928009800 con 0x7f99340cc810 2023-11-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99341516f0 con 0x7f99340cc810 2023-11-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f991bfff640 1 -- 172.21.15.7:0/3355906935 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928009b70 con 0x7f99340cc810 2023-11-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99340c8000 con 0x7f99340cc810 2023-11-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.331+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993406c670 con 0x7f99340cc810 2023-11-28T18:49:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.335+0000 7f991bfff640 1 -- 172.21.15.7:0/3355906935 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9928009cd0 con 0x7f99340cc810 2023-11-28T18:49:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.335+0000 7f991bfff640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f99200668d0 0x7f9920068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.335+0000 7f991bfff640 1 -- 172.21.15.7:0/3355906935 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f9928084e80 con 0x7f99340cc810 2023-11-28T18:49:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.335+0000 7f99337fe640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f99200668d0 0x7f9920068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.335+0000 7f99337fe640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f99200668d0 0x7f9920068d90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f99240ae000 tx=0x7f9924008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.339+0000 7f991bfff640 1 -- 172.21.15.7:0/3355906935 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9928051cf0 con 0x7f99340cc810 2023-11-28T18:49:01.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~222s 2023-11-28T18:49:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.527+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f993404e890 con 0x7f99340cc810 2023-11-28T18:49:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.527+0000 7f991bfff640 1 -- 172.21.15.7:0/3355906935 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f9928055310 con 0x7f99340cc810 2023-11-28T18:49:01.532 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:01.532 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:01.532 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f99200668d0 msgr2=0x7f9920068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f99200668d0 0x7f9920068d90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f99240ae000 tx=0x7f9924008040 comp rx=0 tx=0).stop 2023-11-28T18:49:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99340cc810 msgr2=0x7f993415d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99340cc810 0x7f993415d8f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f992800d800 tx=0x7f992800dcc0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 shutdown_connections 2023-11-28T18:49:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f99200668d0 0x7f9920068d90 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99341542e0 0x7f993415de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99340cc810 0x7f993415d8f0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 --2- 172.21.15.7:0/3355906935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f99340cbb70 0x7f99340c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:01.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 >> 172.21.15.7:0/3355906935 conn(0x7f99340717e0 msgr2=0x7f99340c5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:01.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 shutdown_connections 2023-11-28T18:49:01.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:01.531+0000 7f9939e79640 1 -- 172.21.15.7:0/3355906935 wait complete. 2023-11-28T18:49:01.555 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:03.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:03.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.707+0000 7f4289d9d640 1 Processor -- start 2023-11-28T18:49:03.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4289d9d640 1 -- start start 2023-11-28T18:49:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4289d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f42840c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4289d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f42841575c0 0x7f42840c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4289d9d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42840c6140 con 0x7f42840c4020 2023-11-28T18:49:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4289d9d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42840c72b0 con 0x7f42841575c0 2023-11-28T18:49:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4289d9d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4284070c00 con 0x7f4284150b30 2023-11-28T18:49:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4283fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f42840c4020 0x7f42840c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57084/0 (socket says 172.21.15.7:57084) 2023-11-28T18:49:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4283fff640 1 -- 172.21.15.7:0/2109130091 learned_addr learned my addr 172.21.15.7:0/2109130091 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f42837fe640 1 --2- 172.21.15.7:0/2109130091 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f42840c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f4282ffd640 1 --2- 172.21.15.7:0/2109130091 >> v2:172.21.15.193:3300/0 conn(0x7f42841575c0 0x7f42840c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f42827fc640 1 -- 172.21.15.7:0/2109130091 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223751370 0 0) 0x7f42840c6140 con 0x7f42840c4020 2023-11-28T18:49:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f42827fc640 1 -- 172.21.15.7:0/2109130091 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f426c003530 con 0x7f42840c4020 2023-11-28T18:49:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f42837fe640 1 -- 172.21.15.7:0/2109130091 >> v2:172.21.15.193:3300/0 conn(0x7f42841575c0 msgr2=0x7f42840c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f42837fe640 1 --2- 172.21.15.7:0/2109130091 >> v2:172.21.15.193:3300/0 conn(0x7f42841575c0 0x7f42840c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f42837fe640 1 -- 172.21.15.7:0/2109130091 >> v1:172.21.15.7:6789/0 conn(0x7f42840c4020 legacy=0x7f42840c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:03.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.711+0000 7f42837fe640 1 -- 172.21.15.7:0/2109130091 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840c6370 con 0x7f4284150b30 2023-11-28T18:49:03.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f42837fe640 1 --2- 172.21.15.7:0/2109130091 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f42840c24f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f42740097c0 tx=0x7f427401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94423df8c0deb899 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:03.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f42827fc640 1 -- 172.21.15.7:0/2109130091 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42740a8970 con 0x7f4284150b30 2023-11-28T18:49:03.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f42827fc640 1 -- 172.21.15.7:0/2109130091 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42740a8ad0 con 0x7f4284150b30 2023-11-28T18:49:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f42827fc640 1 -- 172.21.15.7:0/2109130091 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42740a8e60 con 0x7f4284150b30 2023-11-28T18:49:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f4289d9d640 1 -- 172.21.15.7:0/2109130091 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 msgr2=0x7f42840c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f4289d9d640 1 --2- 172.21.15.7:0/2109130091 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f42840c24f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f42740097c0 tx=0x7f427401ec70 comp rx=0 tx=0).stop 2023-11-28T18:49:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f4289d9d640 1 -- 172.21.15.7:0/2109130091 shutdown_connections 2023-11-28T18:49:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f4289d9d640 1 --2- 172.21.15.7:0/2109130091 >> v2:172.21.15.193:3300/0 conn(0x7f42841575c0 0x7f42840c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f4289d9d640 1 --2- 172.21.15.7:0/2109130091 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f42840c24f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f4289d9d640 1 -- 172.21.15.7:0/2109130091 >> 172.21.15.7:0/2109130091 conn(0x7f42840662e0 msgr2=0x7f42840666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f4289d9d640 1 -- 172.21.15.7:0/2109130091 shutdown_connections 2023-11-28T18:49:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.715+0000 7f4289d9d640 1 -- 172.21.15.7:0/2109130091 wait complete. 2023-11-28T18:49:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 Processor -- start 2023-11-28T18:49:03.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 -- start start 2023-11-28T18:49:03.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42840c4020 0x7f4284151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f428414e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42841575c0 0x7f42840c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42841f0210 con 0x7f42841575c0 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42841ef320 con 0x7f42840c4020 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42841ef600 con 0x7f4284150b30 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42841575c0 0x7f42840c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42841575c0 0x7f42840c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43750/0 (socket says 172.21.15.7:43750) 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 -- 172.21.15.7:0/3542566788 learned_addr learned my addr 172.21.15.7:0/3542566788 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f42837fe640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42840c4020 0x7f4284151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4282ffd640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f428414e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 -- 172.21.15.7:0/3542566788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 msgr2=0x7f428414e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f428414e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 -- 172.21.15.7:0/3542566788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42840c4020 msgr2=0x7f4284151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42840c4020 0x7f4284151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 -- 172.21.15.7:0/3542566788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4274009420 con 0x7f42841575c0 2023-11-28T18:49:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4283fff640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42841575c0 0x7f42840c50c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f427c00cb40 tx=0x7f427c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f426bfff640 1 -- 172.21.15.7:0/3542566788 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c00aca0 con 0x7f42841575c0 2023-11-28T18:49:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f426bfff640 1 -- 172.21.15.7:0/3542566788 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c00ae00 con 0x7f42841575c0 2023-11-28T18:49:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f426bfff640 1 -- 172.21.15.7:0/3542566788 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c012620 con 0x7f42841575c0 2023-11-28T18:49:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42841ef8e0 con 0x7f42841575c0 2023-11-28T18:49:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42841efea0 con 0x7f42841575c0 2023-11-28T18:49:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.719+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428404e890 con 0x7f42841575c0 2023-11-28T18:49:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.723+0000 7f426bfff640 1 -- 172.21.15.7:0/3542566788 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f427c007070 con 0x7f42841575c0 2023-11-28T18:49:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.723+0000 7f426bfff640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42700668d0 0x7f4270068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.723+0000 7f426bfff640 1 -- 172.21.15.7:0/3542566788 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f427c0848f0 con 0x7f42841575c0 2023-11-28T18:49:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.723+0000 7f42837fe640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42700668d0 0x7f4270068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.723+0000 7f42837fe640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42700668d0 0x7f4270068d90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f42740b2000 tx=0x7f4274008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:03.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.727+0000 7f426bfff640 1 -- 172.21.15.7:0/3542566788 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f427c0516c0 con 0x7f42841575c0 2023-11-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.919+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f428414cff0 con 0x7f42841575c0 2023-11-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.919+0000 7f426bfff640 1 -- 172.21.15.7:0/3542566788 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f427c017090 con 0x7f42841575c0 2023-11-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42700668d0 msgr2=0x7f4270068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42700668d0 0x7f4270068d90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f42740b2000 tx=0x7f4274008040 comp rx=0 tx=0).stop 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42841575c0 msgr2=0x7f42840c50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42841575c0 0x7f42840c50c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f427c00cb40 tx=0x7f427c00a060 comp rx=0 tx=0).stop 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 shutdown_connections 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f42700668d0 0x7f4270068d90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42841575c0 0x7f42840c50c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4284150b30 0x7f428414e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 --2- 172.21.15.7:0/3542566788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f42840c4020 0x7f4284151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 >> 172.21.15.7:0/3542566788 conn(0x7f42840662e0 msgr2=0x7f42840c98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 shutdown_connections 2023-11-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:03.923+0000 7f4289d9d640 1 -- 172.21.15.7:0/3542566788 wait complete. 2023-11-28T18:49:03.945 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:05.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:06.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed34fe640 1 Processor -- start 2023-11-28T18:49:06.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed34fe640 1 -- start start 2023-11-28T18:49:06.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed34fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faecc0cbb70 0x7faecc0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:06.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed34fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7faecc0cc810 0x7faecc072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:06.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed34fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0ca920 con 0x7faecc1542e0 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed34fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc071d50 con 0x7faecc0cbb70 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed34fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc072030 con 0x7faecc0cc810 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed0a99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faecc0cbb70 0x7faecc0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed0a99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faecc0cbb70 0x7faecc0cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43766/0 (socket says 172.21.15.7:43766) 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed0a99640 1 -- 172.21.15.7:0/154972153 learned_addr learned my addr 172.21.15.7:0/154972153 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed1a9b640 1 --2- 172.21.15.7:0/154972153 >> v2:172.21.15.193:3300/0 conn(0x7faecc0cc810 0x7faecc072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed0a99640 1 -- 172.21.15.7:0/154972153 >> v2:172.21.15.193:3300/0 conn(0x7faecc0cc810 msgr2=0x7faecc072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed0a99640 1 --2- 172.21.15.7:0/154972153 >> v2:172.21.15.193:3300/0 conn(0x7faecc0cc810 0x7faecc072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed0a99640 1 -- 172.21.15.7:0/154972153 >> v1:172.21.15.193:6789/0 conn(0x7faecc1542e0 legacy=0x7faecc15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.099+0000 7faed0a99640 1 -- 172.21.15.7:0/154972153 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0c4cd0 con 0x7faecc0cbb70 2023-11-28T18:49:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed0a99640 1 --2- 172.21.15.7:0/154972153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc0cc2d0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7faecc066f90 tx=0x7faebc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb64638429f4b146 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faec3fff640 1 -- 172.21.15.7:0/154972153 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc0ab610 con 0x7faecc0cbb70 2023-11-28T18:49:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faec3fff640 1 -- 172.21.15.7:0/154972153 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc0ab770 con 0x7faecc0cbb70 2023-11-28T18:49:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faec3fff640 1 -- 172.21.15.7:0/154972153 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc0abb00 con 0x7faecc0cbb70 2023-11-28T18:49:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed34fe640 1 -- 172.21.15.7:0/154972153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 msgr2=0x7faecc0cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed34fe640 1 --2- 172.21.15.7:0/154972153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc0cc2d0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7faecc066f90 tx=0x7faebc01c500 comp rx=0 tx=0).stop 2023-11-28T18:49:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed34fe640 1 -- 172.21.15.7:0/154972153 shutdown_connections 2023-11-28T18:49:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed34fe640 1 --2- 172.21.15.7:0/154972153 >> v2:172.21.15.193:3300/0 conn(0x7faecc0cc810 0x7faecc072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed34fe640 1 --2- 172.21.15.7:0/154972153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc0cc2d0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed34fe640 1 -- 172.21.15.7:0/154972153 >> 172.21.15.7:0/154972153 conn(0x7faecc0717e0 msgr2=0x7faecc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed34fe640 1 -- 172.21.15.7:0/154972153 shutdown_connections 2023-11-28T18:49:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.103+0000 7faed34fe640 1 -- 172.21.15.7:0/154972153 wait complete. 2023-11-28T18:49:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 Processor -- start 2023-11-28T18:49:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 -- start start 2023-11-28T18:49:06.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:06.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faecc0cc810 0x7faecc14acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:06.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc1542e0 0x7faecc15a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc158750 con 0x7faecc0cbb70 2023-11-28T18:49:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc158f80 con 0x7faecc1542e0 2023-11-28T18:49:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc157ab0 con 0x7faecc0cc810 2023-11-28T18:49:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43774/0 (socket says 172.21.15.7:43774) 2023-11-28T18:49:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 -- 172.21.15.7:0/3813831747 learned_addr learned my addr 172.21.15.7:0/3813831747 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed1a9b640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc1542e0 0x7faecc15a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed0a99640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faecc0cc810 0x7faecc14acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 -- 172.21.15.7:0/3813831747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faecc0cc810 msgr2=0x7faecc14acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faecc0cc810 0x7faecc14acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 -- 172.21.15.7:0/3813831747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc1542e0 msgr2=0x7faecc15a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc1542e0 0x7faecc15a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 -- 172.21.15.7:0/3813831747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc0094d0 con 0x7faecc0cbb70 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed129a640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc157570 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7faec800c3c0 tx=0x7faec800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faec1ffb640 1 -- 172.21.15.7:0/3813831747 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec80052d0 con 0x7faecc0cbb70 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faec1ffb640 1 -- 172.21.15.7:0/3813831747 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec8005430 con 0x7faecc0cbb70 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faec1ffb640 1 -- 172.21.15.7:0/3813831747 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8003840 con 0x7faecc0cbb70 2023-11-28T18:49:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc14b230 con 0x7faecc0cbb70 2023-11-28T18:49:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.107+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc1554e0 con 0x7faecc0cbb70 2023-11-28T18:49:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.111+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc06c670 con 0x7faecc0cbb70 2023-11-28T18:49:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.111+0000 7faec1ffb640 1 -- 172.21.15.7:0/3813831747 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7faec80039a0 con 0x7faecc0cbb70 2023-11-28T18:49:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.111+0000 7faec1ffb640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faeb0066900 0x7faeb0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.111+0000 7faec1ffb640 1 -- 172.21.15.7:0/3813831747 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7faec8083da0 con 0x7faecc0cbb70 2023-11-28T18:49:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.111+0000 7faed0a99640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faeb0066900 0x7faeb0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.111+0000 7faed0a99640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faeb0066900 0x7faeb0068dc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7faebc0af000 tx=0x7faebc0b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.115+0000 7faec1ffb640 1 -- 172.21.15.7:0/3813831747 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faec8050c00 con 0x7faecc0cbb70 2023-11-28T18:49:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.303+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faecc04e890 con 0x7faecc0cbb70 2023-11-28T18:49:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.303+0000 7faec1ffb640 1 -- 172.21.15.7:0/3813831747 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7faec8017090 con 0x7faecc0cbb70 2023-11-28T18:49:06.308 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:06.308 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:06.309 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faeb0066900 msgr2=0x7faeb0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faeb0066900 0x7faeb0068dc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7faebc0af000 tx=0x7faebc0b6040 comp rx=0 tx=0).stop 2023-11-28T18:49:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 msgr2=0x7faecc157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc157570 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7faec800c3c0 tx=0x7faec800c880 comp rx=0 tx=0).stop 2023-11-28T18:49:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 shutdown_connections 2023-11-28T18:49:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7faeb0066900 0x7faeb0068dc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc1542e0 0x7faecc15a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7faecc0cc810 0x7faecc14acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 --2- 172.21.15.7:0/3813831747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc0cbb70 0x7faecc157570 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 >> 172.21.15.7:0/3813831747 conn(0x7faecc0717e0 msgr2=0x7faecc0ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 shutdown_connections 2023-11-28T18:49:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:06.307+0000 7faed34fe640 1 -- 172.21.15.7:0/3813831747 wait complete. 2023-11-28T18:49:06.330 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:08.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~229s 2023-11-28T18:49:08.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:08.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f501830f640 1 Processor -- start 2023-11-28T18:49:08.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f501830f640 1 -- start start 2023-11-28T18:49:08.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f501830f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:08.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f501830f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50100cf6c0 0x7f50100cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:08.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f501830f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501015e380 con 0x7f501015a8a0 2023-11-28T18:49:08.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f501830f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501015d7e0 con 0x7f50100cf6c0 2023-11-28T18:49:08.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f501830f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50100cab80 con 0x7f50101540e0 2023-11-28T18:49:08.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50158aa640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f501015a8a0 0x7f501015dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57096/0 (socket says 172.21.15.7:57096) 2023-11-28T18:49:08.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50158aa640 1 -- 172.21.15.7:0/479344222 learned_addr learned my addr 172.21.15.7:0/479344222 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:08.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50168ac640 1 --2- 172.21.15.7:0/479344222 >> v2:172.21.15.7:3300/0 conn(0x7f50100cf6c0 0x7f50100cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:08.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50160ab640 1 --2- 172.21.15.7:0/479344222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:08.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50150a9640 1 -- 172.21.15.7:0/479344222 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47293031 0 0) 0x7f5000001120 con 0x7f501015a8a0 2023-11-28T18:49:08.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50150a9640 1 -- 172.21.15.7:0/479344222 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc003530 con 0x7f501015a8a0 2023-11-28T18:49:08.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50168ac640 1 -- 172.21.15.7:0/479344222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 msgr2=0x7f50100ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:08.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50168ac640 1 --2- 172.21.15.7:0/479344222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50168ac640 1 -- 172.21.15.7:0/479344222 >> v1:172.21.15.7:6789/0 conn(0x7f501015a8a0 legacy=0x7f501015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:08.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.483+0000 7f50168ac640 1 -- 172.21.15.7:0/479344222 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50100cb850 con 0x7f50100cf6c0 2023-11-28T18:49:08.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f50168ac640 1 --2- 172.21.15.7:0/479344222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50100cfb50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f5008009d90 tx=0x7f500801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc40ce5a148f2ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:08.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f50150a9640 1 -- 172.21.15.7:0/479344222 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50080a5900 con 0x7f50100cf6c0 2023-11-28T18:49:08.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f50150a9640 1 -- 172.21.15.7:0/479344222 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50080a5a60 con 0x7f50100cf6c0 2023-11-28T18:49:08.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f50150a9640 1 -- 172.21.15.7:0/479344222 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50080a5df0 con 0x7f50100cf6c0 2023-11-28T18:49:08.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f501830f640 1 -- 172.21.15.7:0/479344222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 msgr2=0x7f50100cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:08.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f501830f640 1 --2- 172.21.15.7:0/479344222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50100cfb50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f5008009d90 tx=0x7f500801c950 comp rx=0 tx=0).stop 2023-11-28T18:49:08.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f501830f640 1 -- 172.21.15.7:0/479344222 shutdown_connections 2023-11-28T18:49:08.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f501830f640 1 --2- 172.21.15.7:0/479344222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50100cfb50 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f501830f640 1 --2- 172.21.15.7:0/479344222 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f501830f640 1 -- 172.21.15.7:0/479344222 >> 172.21.15.7:0/479344222 conn(0x7f50100662e0 msgr2=0x7f50100666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:08.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f501830f640 1 -- 172.21.15.7:0/479344222 shutdown_connections 2023-11-28T18:49:08.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.487+0000 7f501830f640 1 -- 172.21.15.7:0/479344222 wait complete. 2023-11-28T18:49:08.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 Processor -- start 2023-11-28T18:49:08.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 -- start start 2023-11-28T18:49:08.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50101f6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:08.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:08.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f501015a8a0 0x7f50100c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:08.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50100c72d0 con 0x7f50100cf6c0 2023-11-28T18:49:08.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5010142d00 con 0x7f501015a8a0 2023-11-28T18:49:08.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50101f6a40 con 0x7f50101540e0 2023-11-28T18:49:08.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50101f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:08.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50101f6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43804/0 (socket says 172.21.15.7:43804) 2023-11-28T18:49:08.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 -- 172.21.15.7:0/1856568774 learned_addr learned my addr 172.21.15.7:0/1856568774 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:08.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50168ac640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f501015a8a0 0x7f50100c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:08.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50158aa640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 -- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 msgr2=0x7f50100c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 -- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f501015a8a0 msgr2=0x7f50100c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f501015a8a0 0x7f50100c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 -- 172.21.15.7:0/1856568774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5008009a70 con 0x7f50100cf6c0 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50158aa640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f50160ab640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50101f6170 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f500c00d800 tx=0x7f500c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f5006ffd640 1 -- 172.21.15.7:0/1856568774 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c00b6c0 con 0x7f50100cf6c0 2023-11-28T18:49:08.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f5006ffd640 1 -- 172.21.15.7:0/1856568774 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c00b820 con 0x7f50100cf6c0 2023-11-28T18:49:08.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f5006ffd640 1 -- 172.21.15.7:0/1856568774 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c00bb90 con 0x7f50100cf6c0 2023-11-28T18:49:08.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50101f6d20 con 0x7f50100cf6c0 2023-11-28T18:49:08.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.491+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501015cbd0 con 0x7f50100cf6c0 2023-11-28T18:49:08.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.495+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501004e890 con 0x7f50100cf6c0 2023-11-28T18:49:08.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.495+0000 7f5006ffd640 1 -- 172.21.15.7:0/1856568774 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f500c00bcf0 con 0x7f50100cf6c0 2023-11-28T18:49:08.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.495+0000 7f5006ffd640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ff4066900 0x7f4ff4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:08.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.495+0000 7f5006ffd640 1 -- 172.21.15.7:0/1856568774 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f500c084dd0 con 0x7f50100cf6c0 2023-11-28T18:49:08.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.495+0000 7f50158aa640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ff4066900 0x7f4ff4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:08.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.499+0000 7f50158aa640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ff4066900 0x7f4ff4068dc0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f50100b2300 tx=0x7f5000009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:08.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.499+0000 7f5006ffd640 1 -- 172.21.15.7:0/1856568774 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f500c051af0 con 0x7f50100cf6c0 2023-11-28T18:49:08.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.695+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50100cfd80 con 0x7f50100cf6c0 2023-11-28T18:49:08.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.695+0000 7f5006ffd640 1 -- 172.21.15.7:0/1856568774 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f500c055110 con 0x7f50100cf6c0 2023-11-28T18:49:08.699 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:08.699 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:08.700 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:08.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ff4066900 msgr2=0x7f4ff4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:08.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ff4066900 0x7f4ff4068dc0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f50100b2300 tx=0x7f5000009040 comp rx=0 tx=0).stop 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 msgr2=0x7f50101f6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50101f6170 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f500c00d800 tx=0x7f500c00dcc0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 shutdown_connections 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4ff4066900 0x7f4ff4068dc0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f501015a8a0 0x7f50100c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f50101540e0 0x7f50100c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 --2- 172.21.15.7:0/1856568774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50100cf6c0 0x7f50101f6170 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 >> 172.21.15.7:0/1856568774 conn(0x7f50100662e0 msgr2=0x7f501015dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 shutdown_connections 2023-11-28T18:49:08.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:08.699+0000 7f501830f640 1 -- 172.21.15.7:0/1856568774 wait complete. 2023-11-28T18:49:08.721 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:10.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:10.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f848b3f5640 1 Processor -- start 2023-11-28T18:49:10.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f848b3f5640 1 -- start start 2023-11-28T18:49:10.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f848b3f5640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8484151cd0 0x7f848415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f848b3f5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8484157b90 0x7f84840701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f848b3f5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8484157fe0 con 0x7f848406e560 2023-11-28T18:49:10.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f848b3f5640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8484070730 con 0x7f8484151cd0 2023-11-28T18:49:10.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f848b3f5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8484071e60 con 0x7f8484157b90 2023-11-28T18:49:10.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f8488990640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f848406e560 0x7f84840c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57102/0 (socket says 172.21.15.7:57102) 2023-11-28T18:49:10.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f8488990640 1 -- 172.21.15.7:0/3734211884 learned_addr learned my addr 172.21.15.7:0/3734211884 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:10.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616973534 0 0) 0x7f8484157fe0 con 0x7f848406e560 2023-11-28T18:49:10.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.875+0000 7f8489191640 1 --2- 172.21.15.7:0/3734211884 >> v2:172.21.15.193:3300/0 conn(0x7f8484151cd0 0x7f848415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8470003530 con 0x7f848406e560 2023-11-28T18:49:10.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8489992640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8484157b90 0x7f84840701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2157282413 0 0) 0x7f8470003530 con 0x7f848406e560 2023-11-28T18:49:10.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8484157b90 msgr2=0x7f84840701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:10.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8484157b90 0x7f84840701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:10.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 >> v2:172.21.15.193:3300/0 conn(0x7f8484151cd0 msgr2=0x7f848415bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:10.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 --2- 172.21.15.7:0/3734211884 >> v2:172.21.15.193:3300/0 conn(0x7f8484151cd0 0x7f848415bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:10.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8484070930 con 0x7f848406e560 2023-11-28T18:49:10.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f8474001680 con 0x7f848406e560 2023-11-28T18:49:10.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 >> v1:172.21.15.7:6789/0 conn(0x7f848406e560 legacy=0x7f84840c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:10.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8484157b90 0x7f8470003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8470003f00 0x7f84700047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8470004d10 0x7f8470005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8484070730 con 0x7f8470003f00 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8484071e60 con 0x7f8470004d10 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847000b9e0 con 0x7f8484157b90 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8488990640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8470003f00 0x7f84700047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8489191640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8484157b90 0x7f8470003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8489992640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8470004d10 0x7f8470005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8488990640 1 -- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8484157b90 msgr2=0x7f8470003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8488990640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8484157b90 0x7f8470003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8488990640 1 -- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8470004d10 msgr2=0x7f8470005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8488990640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8470004d10 0x7f8470005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8488990640 1 -- 172.21.15.7:0/3734211884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8470003530 con 0x7f8470003f00 2023-11-28T18:49:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.879+0000 7f8488990640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8470003f00 0x7f84700047d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f847400c270 tx=0x7f847401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13dd2f9fed840f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:10.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84740058e0 con 0x7f8470003f00 2023-11-28T18:49:10.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8474005a40 con 0x7f8470003f00 2023-11-28T18:49:10.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f847bfff640 1 -- 172.21.15.7:0/3734211884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474005e50 con 0x7f8470003f00 2023-11-28T18:49:10.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- 172.21.15.7:0/3734211884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8470003f00 msgr2=0x7f84700047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:10.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 --2- 172.21.15.7:0/3734211884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8470003f00 0x7f84700047d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f847400c270 tx=0x7f847401cc60 comp rx=0 tx=0).stop 2023-11-28T18:49:10.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f8489992640 1 -- 172.21.15.7:0/3734211884 reap_dead start 2023-11-28T18:49:10.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- 172.21.15.7:0/3734211884 shutdown_connections 2023-11-28T18:49:10.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- 172.21.15.7:0/3734211884 >> 172.21.15.7:0/3734211884 conn(0x7f84840662e0 msgr2=0x7f84840666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- 172.21.15.7:0/3734211884 shutdown_connections 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- 172.21.15.7:0/3734211884 wait complete. 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 Processor -- start 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- start start 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8470003f00 0x7f84840c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84840c9c30 0x7f84841f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84840c7920 0x7f848415a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84840c63b0 con 0x7f84840c9c30 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84840c75c0 con 0x7f84840c7920 2023-11-28T18:49:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.883+0000 7f848b3f5640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84841f21c0 con 0x7f8470003f00 2023-11-28T18:49:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84840c9c30 0x7f84841f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84840c9c30 0x7f84841f2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43816/0 (socket says 172.21.15.7:43816) 2023-11-28T18:49:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 -- 172.21.15.7:0/1046954285 learned_addr learned my addr 172.21.15.7:0/1046954285 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8489191640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8470003f00 0x7f84840c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8489992640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84840c7920 0x7f848415a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 -- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8470003f00 msgr2=0x7f84840c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8470003f00 0x7f84840c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 -- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84840c7920 msgr2=0x7f848415a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84840c7920 0x7f848415a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 -- 172.21.15.7:0/1046954285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8484149050 con 0x7f84840c9c30 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8489992640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84840c7920 0x7f848415a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8488990640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84840c9c30 0x7f84841f2650 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f847400c210 tx=0x7f8470005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8479ffb640 1 -- 172.21.15.7:0/1046954285 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474005e30 con 0x7f84840c9c30 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8479ffb640 1 -- 172.21.15.7:0/1046954285 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84740047f0 con 0x7f84840c9c30 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84841f2b90 con 0x7f84840c9c30 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8479ffb640 1 -- 172.21.15.7:0/1046954285 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474004b60 con 0x7f84840c9c30 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84840c60d0 con 0x7f84840c9c30 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848404e890 con 0x7f84840c9c30 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8479ffb640 1 -- 172.21.15.7:0/1046954285 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8474004cc0 con 0x7f84840c9c30 2023-11-28T18:49:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.887+0000 7f8479ffb640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84680668f0 0x7f8468068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:10.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.891+0000 7f8479ffb640 1 -- 172.21.15.7:0/1046954285 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f847411e1e0 con 0x7f84840c9c30 2023-11-28T18:49:10.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.891+0000 7f8489191640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84680668f0 0x7f8468068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:10.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.891+0000 7f8489191640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84680668f0 0x7f8468068db0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8480009be0 tx=0x7f8480009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:10.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:10.895+0000 7f8479ffb640 1 -- 172.21.15.7:0/1046954285 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f84740eaf00 con 0x7f84840c9c30 2023-11-28T18:49:11.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84841f2300 con 0x7f84840c9c30 2023-11-28T18:49:11.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f8479ffb640 1 -- 172.21.15.7:0/1046954285 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f84740ee520 con 0x7f84840c9c30 2023-11-28T18:49:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:11.089 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:11.090 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:11.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84680668f0 msgr2=0x7f8468068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:11.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84680668f0 0x7f8468068db0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8480009be0 tx=0x7f8480009190 comp rx=0 tx=0).stop 2023-11-28T18:49:11.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84840c9c30 msgr2=0x7f84841f2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84840c9c30 0x7f84841f2650 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f847400c210 tx=0x7f8470005800 comp rx=0 tx=0).stop 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 shutdown_connections 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f84680668f0 0x7f8468068db0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84840c7920 0x7f848415a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84840c9c30 0x7f84841f2650 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 --2- 172.21.15.7:0/1046954285 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8470003f00 0x7f84840c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 >> 172.21.15.7:0/1046954285 conn(0x7f84840662e0 msgr2=0x7f84840723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 shutdown_connections 2023-11-28T18:49:11.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:11.087+0000 7f848b3f5640 1 -- 172.21.15.7:0/1046954285 wait complete. 2023-11-28T18:49:11.112 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:49:11.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:49:11.882 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:49:13.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:13.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.263+0000 7f931a617640 1 Processor -- start 2023-11-28T18:49:13.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.263+0000 7f931a617640 1 -- start start 2023-11-28T18:49:13.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f931a617640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f931415bd00 0x7f9314157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:13.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f931a617640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931415c110 con 0x7f9314151cd0 2023-11-28T18:49:13.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f931a617640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9314071e60 con 0x7f931406e560 2023-11-28T18:49:13.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f931a617640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9314070930 con 0x7f931415bd00 2023-11-28T18:49:13.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f9318bb4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f931415bd00 0x7f9314157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:13.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f9313fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9314151cd0 0x7f93140c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57108/0 (socket says 172.21.15.7:57108) 2023-11-28T18:49:13.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f9313fff640 1 -- 172.21.15.7:0/2390756317 learned_addr learned my addr 172.21.15.7:0/2390756317 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:13.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f9318bb4640 1 -- 172.21.15.7:0/2390756317 >> v1:172.21.15.193:6789/0 conn(0x7f931406e560 legacy=0x7f9314160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:13.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f9318bb4640 1 -- 172.21.15.7:0/2390756317 >> v1:172.21.15.7:6789/0 conn(0x7f9314151cd0 legacy=0x7f93140c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:13.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f9318bb4640 1 -- 172.21.15.7:0/2390756317 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314070bf0 con 0x7f931415bd00 2023-11-28T18:49:13.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.267+0000 7f9312ffd640 1 -- 172.21.15.7:0/2390756317 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955312801 0 0) 0x7f931415c110 con 0x7f9314151cd0 2023-11-28T18:49:13.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f9318bb4640 1 --2- 172.21.15.7:0/2390756317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931415bd00 0x7f9314157b90 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f930c009d90 tx=0x7f930c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6811e9393fbbda7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:13.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f9312ffd640 1 -- 172.21.15.7:0/2390756317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c0a5900 con 0x7f931415bd00 2023-11-28T18:49:13.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f9312ffd640 1 -- 172.21.15.7:0/2390756317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c0a5a60 con 0x7f931415bd00 2023-11-28T18:49:13.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f9312ffd640 1 -- 172.21.15.7:0/2390756317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c0a5df0 con 0x7f931415bd00 2023-11-28T18:49:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 -- 172.21.15.7:0/2390756317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931415bd00 msgr2=0x7f9314157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 --2- 172.21.15.7:0/2390756317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931415bd00 0x7f9314157b90 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f930c009d90 tx=0x7f930c01c950 comp rx=0 tx=0).stop 2023-11-28T18:49:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 -- 172.21.15.7:0/2390756317 shutdown_connections 2023-11-28T18:49:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 --2- 172.21.15.7:0/2390756317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931415bd00 0x7f9314157b90 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 -- 172.21.15.7:0/2390756317 >> 172.21.15.7:0/2390756317 conn(0x7f93140662e0 msgr2=0x7f93140666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 -- 172.21.15.7:0/2390756317 shutdown_connections 2023-11-28T18:49:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 -- 172.21.15.7:0/2390756317 wait complete. 2023-11-28T18:49:13.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 Processor -- start 2023-11-28T18:49:13.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 -- start start 2023-11-28T18:49:13.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f931406e560 0x7f9314154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:13.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.271+0000 7f931a617640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9314151cd0 0x7f93140c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:13.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f931a617640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f931415bd00 0x7f9314152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:13.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f931a617640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9314154d50 con 0x7f9314151cd0 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f931a617640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93141f0610 con 0x7f931406e560 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f931a617640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93141f3f30 con 0x7f931415bd00 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9314151cd0 0x7f93140c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f9313fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f931406e560 0x7f9314154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9314151cd0 0x7f93140c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43830/0 (socket says 172.21.15.7:43830) 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 -- 172.21.15.7:0/3571188141 learned_addr learned my addr 172.21.15.7:0/3571188141 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f9318bb4640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f931415bd00 0x7f9314152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 -- 172.21.15.7:0/3571188141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f931415bd00 msgr2=0x7f9314152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f931415bd00 0x7f9314152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 -- 172.21.15.7:0/3571188141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f931406e560 msgr2=0x7f9314154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:13.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f931406e560 0x7f9314154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:13.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 -- 172.21.15.7:0/3571188141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c009a70 con 0x7f9314151cd0 2023-11-28T18:49:13.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f93137fe640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9314151cd0 0x7f93140c85a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f930800c530 tx=0x7f930800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:13.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f9310ff9640 1 -- 172.21.15.7:0/3571188141 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9308005220 con 0x7f9314151cd0 2023-11-28T18:49:13.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f9310ff9640 1 -- 172.21.15.7:0/3571188141 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9308005380 con 0x7f9314151cd0 2023-11-28T18:49:13.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f9310ff9640 1 -- 172.21.15.7:0/3571188141 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9308007270 con 0x7f9314151cd0 2023-11-28T18:49:13.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93141f0800 con 0x7f9314151cd0 2023-11-28T18:49:13.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93141f0d80 con 0x7f9314151cd0 2023-11-28T18:49:13.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93140c26a0 con 0x7f9314151cd0 2023-11-28T18:49:13.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.275+0000 7f9310ff9640 1 -- 172.21.15.7:0/3571188141 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f93080073d0 con 0x7f9314151cd0 2023-11-28T18:49:13.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.279+0000 7f9310ff9640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9300066900 0x7f9300068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:13.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.279+0000 7f9310ff9640 1 -- 172.21.15.7:0/3571188141 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f9308084080 con 0x7f9314151cd0 2023-11-28T18:49:13.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.279+0000 7f9313fff640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9300066900 0x7f9300068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:13.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.279+0000 7f9313fff640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9300066900 0x7f9300068dc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9304000f70 tx=0x7f9304007240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:13.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.283+0000 7f9310ff9640 1 -- 172.21.15.7:0/3571188141 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f93080b5280 con 0x7f9314151cd0 2023-11-28T18:49:13.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.471+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f931404e890 con 0x7f9314151cd0 2023-11-28T18:49:13.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f9310ff9640 1 -- 172.21.15.7:0/3571188141 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f9308050ee0 con 0x7f9314151cd0 2023-11-28T18:49:13.477 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:13.477 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:13.477 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:13.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9300066900 msgr2=0x7f9300068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:13.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9300066900 0x7f9300068dc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9304000f70 tx=0x7f9304007240 comp rx=0 tx=0).stop 2023-11-28T18:49:13.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9314151cd0 msgr2=0x7f93140c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:13.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9314151cd0 0x7f93140c85a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f930800c530 tx=0x7f930800c9f0 comp rx=0 tx=0).stop 2023-11-28T18:49:13.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 shutdown_connections 2023-11-28T18:49:13.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f9300066900 0x7f9300068dc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:13.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f931415bd00 0x7f9314152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:13.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9314151cd0 0x7f93140c85a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:13.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 --2- 172.21.15.7:0/3571188141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f931406e560 0x7f9314154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:13.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 >> 172.21.15.7:0/3571188141 conn(0x7f93140662e0 msgr2=0x7f93140cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:13.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 shutdown_connections 2023-11-28T18:49:13.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:13.475+0000 7f931a617640 1 -- 172.21.15.7:0/3571188141 wait complete. 2023-11-28T18:49:13.499 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:14.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~235s 2023-11-28T18:49:15.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54ffaa640 1 Processor -- start 2023-11-28T18:49:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54ffaa640 1 -- start start 2023-11-28T18:49:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54ffaa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa5481542e0 0x7fa54815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54ffaa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa548072450 0x7fa548072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54ffaa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5480ca920 con 0x7fa5480cbb70 2023-11-28T18:49:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54ffaa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa548071d50 con 0x7fa548072450 2023-11-28T18:49:15.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54ffaa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa548072030 con 0x7fa5481542e0 2023-11-28T18:49:15.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54d545640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa5480cbb70 0x7fa5480cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39784/0 (socket says 172.21.15.7:39784) 2023-11-28T18:49:15.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54d545640 1 -- 172.21.15.7:0/3312704615 learned_addr learned my addr 172.21.15.7:0/3312704615 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:15.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54e547640 1 --2- 172.21.15.7:0/3312704615 >> v2:172.21.15.7:3300/0 conn(0x7fa548072450 0x7fa548072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:15.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54dd46640 1 --2- 172.21.15.7:0/3312704615 >> v2:172.21.15.193:3300/0 conn(0x7fa5481542e0 0x7fa54815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:15.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54e547640 1 -- 172.21.15.7:0/3312704615 >> v2:172.21.15.193:3300/0 conn(0x7fa5481542e0 msgr2=0x7fa54815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:15.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54e547640 1 --2- 172.21.15.7:0/3312704615 >> v2:172.21.15.193:3300/0 conn(0x7fa5481542e0 0x7fa54815e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54e547640 1 -- 172.21.15.7:0/3312704615 >> v1:172.21.15.7:6789/0 conn(0x7fa5480cbb70 legacy=0x7fa5480cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:15.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.655+0000 7fa54e547640 1 -- 172.21.15.7:0/3312704615 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54815e550 con 0x7fa548072450 2023-11-28T18:49:15.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54e547640 1 --2- 172.21.15.7:0/3312704615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa548072450 0x7fa548072870 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa548071d20 tx=0x7fa54001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc163ca83fadda1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:15.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54cd44640 1 -- 172.21.15.7:0/3312704615 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5400a58e0 con 0x7fa548072450 2023-11-28T18:49:15.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54cd44640 1 -- 172.21.15.7:0/3312704615 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5400a5a40 con 0x7fa548072450 2023-11-28T18:49:15.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54cd44640 1 -- 172.21.15.7:0/3312704615 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5400a5dd0 con 0x7fa548072450 2023-11-28T18:49:15.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54ffaa640 1 -- 172.21.15.7:0/3312704615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa548072450 msgr2=0x7fa548072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:15.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/3312704615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa548072450 0x7fa548072870 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa548071d20 tx=0x7fa54001c910 comp rx=0 tx=0).stop 2023-11-28T18:49:15.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54ffaa640 1 -- 172.21.15.7:0/3312704615 shutdown_connections 2023-11-28T18:49:15.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/3312704615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa548072450 0x7fa548072870 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/3312704615 >> v2:172.21.15.193:3300/0 conn(0x7fa5481542e0 0x7fa54815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54ffaa640 1 -- 172.21.15.7:0/3312704615 >> 172.21.15.7:0/3312704615 conn(0x7fa5480717e0 msgr2=0x7fa5480662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:15.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54ffaa640 1 -- 172.21.15.7:0/3312704615 shutdown_connections 2023-11-28T18:49:15.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.659+0000 7fa54ffaa640 1 -- 172.21.15.7:0/3312704615 wait complete. 2023-11-28T18:49:15.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 Processor -- start 2023-11-28T18:49:15.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 -- start start 2023-11-28T18:49:15.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa548072450 0x7fa5481f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:15.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5480cbb70 0x7fa5480cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:15.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa5481542e0 0x7fa548152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:15.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480d0790 con 0x7fa5480cbb70 2023-11-28T18:49:15.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481ec7f0 con 0x7fa548072450 2023-11-28T18:49:15.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480cdb50 con 0x7fa5481542e0 2023-11-28T18:49:15.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5480cbb70 0x7fa5480cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:15.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5480cbb70 0x7fa5480cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51184/0 (socket says 172.21.15.7:51184) 2023-11-28T18:49:15.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 -- 172.21.15.7:0/1650172092 learned_addr learned my addr 172.21.15.7:0/1650172092 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:15.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54dd46640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa548072450 0x7fa5481f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:15.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 -- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa5481542e0 msgr2=0x7fa548152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:15.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54e547640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa5481542e0 0x7fa548152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:15.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa5481542e0 0x7fa548152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 -- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa548072450 msgr2=0x7fa5481f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:15.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa548072450 0x7fa5481f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 -- 172.21.15.7:0/1650172092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540009a70 con 0x7fa5480cbb70 2023-11-28T18:49:15.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54dd46640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa548072450 0x7fa5481f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:49:15.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54e547640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa5481542e0 0x7fa548152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:49:15.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54d545640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5480cbb70 0x7fa5480cdfa0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa53800c420 tx=0x7fa53800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:15.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa53e7fc640 1 -- 172.21.15.7:0/1650172092 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa538005cf0 con 0x7fa5480cbb70 2023-11-28T18:49:15.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa53e7fc640 1 -- 172.21.15.7:0/1650172092 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa538005e50 con 0x7fa5480cbb70 2023-11-28T18:49:15.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa53e7fc640 1 -- 172.21.15.7:0/1650172092 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5380072f0 con 0x7fa5480cbb70 2023-11-28T18:49:15.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5481f30b0 con 0x7fa5480cbb70 2023-11-28T18:49:15.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.663+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5481f3650 con 0x7fa5480cbb70 2023-11-28T18:49:15.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.667+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54806c670 con 0x7fa5480cbb70 2023-11-28T18:49:15.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.667+0000 7fa53e7fc640 1 -- 172.21.15.7:0/1650172092 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa538007450 con 0x7fa5480cbb70 2023-11-28T18:49:15.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.667+0000 7fa53e7fc640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa52c0668d0 0x7fa52c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:15.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.667+0000 7fa53e7fc640 1 -- 172.21.15.7:0/1650172092 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fa538083f50 con 0x7fa5480cbb70 2023-11-28T18:49:15.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.667+0000 7fa54dd46640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa52c0668d0 0x7fa52c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:15.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.667+0000 7fa54dd46640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa52c0668d0 0x7fa52c068d90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa544009a40 tx=0x7fa544008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:15.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.671+0000 7fa53e7fc640 1 -- 172.21.15.7:0/1650172092 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa538050c70 con 0x7fa5480cbb70 2023-11-28T18:49:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.863+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa54804e890 con 0x7fa5480cbb70 2023-11-28T18:49:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.863+0000 7fa53e7fc640 1 -- 172.21.15.7:0/1650172092 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fa538017070 con 0x7fa5480cbb70 2023-11-28T18:49:15.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:15.867 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.863+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa52c0668d0 msgr2=0x7fa52c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.863+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa52c0668d0 0x7fa52c068d90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa544009a40 tx=0x7fa544008040 comp rx=0 tx=0).stop 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5480cbb70 msgr2=0x7fa5480cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5480cbb70 0x7fa5480cdfa0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa53800c420 tx=0x7fa53800c8e0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 shutdown_connections 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa52c0668d0 0x7fa52c068d90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa5481542e0 0x7fa548152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5480cbb70 0x7fa5480cdfa0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 --2- 172.21.15.7:0/1650172092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa548072450 0x7fa5481f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 >> 172.21.15.7:0/1650172092 conn(0x7fa5480717e0 msgr2=0x7fa54815bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 shutdown_connections 2023-11-28T18:49:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:15.867+0000 7fa54ffaa640 1 -- 172.21.15.7:0/1650172092 wait complete. 2023-11-28T18:49:15.889 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:17.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:18.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f391db65640 1 Processor -- start 2023-11-28T18:49:18.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f391db65640 1 -- start start 2023-11-28T18:49:18.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f391db65640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180ca4d0 0x7f391815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f391db65640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f39180cf6c0 0x7f39180cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f391db65640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f391815e380 con 0x7f39181540e0 2023-11-28T18:49:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f391db65640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391815ade0 con 0x7f39180cf6c0 2023-11-28T18:49:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f391db65640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391815d7e0 con 0x7f39180ca4d0 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f390ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180ca4d0 0x7f391815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f3917fff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f39180cf6c0 0x7f39180cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f390ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180ca4d0 0x7f391815a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.7:47538/0 (socket says 172.21.15.7:47538) 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.043+0000 7f390ffff640 1 -- 172.21.15.7:0/708090703 learned_addr learned my addr 172.21.15.7:0/708090703 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f390ffff640 1 -- 172.21.15.7:0/708090703 >> v2:172.21.15.193:3300/0 conn(0x7f39180cf6c0 msgr2=0x7f39180cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f390ffff640 1 --2- 172.21.15.7:0/708090703 >> v2:172.21.15.193:3300/0 conn(0x7f39180cf6c0 0x7f39180cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f390ffff640 1 -- 172.21.15.7:0/708090703 >> v1:172.21.15.193:6789/0 conn(0x7f39181540e0 legacy=0x7f391815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f390ffff640 1 -- 172.21.15.7:0/708090703 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39180ca8f0 con 0x7f39180ca4d0 2023-11-28T18:49:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f3917fff640 1 --2- 172.21.15.7:0/708090703 >> v2:172.21.15.193:3300/0 conn(0x7f39180cf6c0 0x7f39180cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:49:18.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f390ffff640 1 --2- 172.21.15.7:0/708090703 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180ca4d0 0x7f391815a8a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3900009870 tx=0x7f390001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef12123fda93cddd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:18.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f3916ffd640 1 -- 172.21.15.7:0/708090703 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000a7070 con 0x7f39180ca4d0 2023-11-28T18:49:18.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f3916ffd640 1 -- 172.21.15.7:0/708090703 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39000ab430 con 0x7f39180ca4d0 2023-11-28T18:49:18.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f3916ffd640 1 -- 172.21.15.7:0/708090703 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000ab7c0 con 0x7f39180ca4d0 2023-11-28T18:49:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f391db65640 1 -- 172.21.15.7:0/708090703 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180ca4d0 msgr2=0x7f391815a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.047+0000 7f391db65640 1 --2- 172.21.15.7:0/708090703 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180ca4d0 0x7f391815a8a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3900009870 tx=0x7f390001c540 comp rx=0 tx=0).stop 2023-11-28T18:49:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 -- 172.21.15.7:0/708090703 shutdown_connections 2023-11-28T18:49:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 --2- 172.21.15.7:0/708090703 >> v2:172.21.15.193:3300/0 conn(0x7f39180cf6c0 0x7f39180cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 --2- 172.21.15.7:0/708090703 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180ca4d0 0x7f391815a8a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 -- 172.21.15.7:0/708090703 >> 172.21.15.7:0/708090703 conn(0x7f39180662e0 msgr2=0x7f39180666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 -- 172.21.15.7:0/708090703 shutdown_connections 2023-11-28T18:49:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 -- 172.21.15.7:0/708090703 wait complete. 2023-11-28T18:49:18.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 Processor -- start 2023-11-28T18:49:18.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 -- start start 2023-11-28T18:49:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39180ca4d0 0x7f3918144450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180cf6c0 0x7f39181f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39181540e0 0x7f39181f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918145ed0 con 0x7f39181540e0 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918145c80 con 0x7f39180ca4d0 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f391db65640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3918145390 con 0x7f39180cf6c0 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39181540e0 0x7f39181f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39181540e0 0x7f39181f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51188/0 (socket says 172.21.15.7:51188) 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 -- 172.21.15.7:0/2111351556 learned_addr learned my addr 172.21.15.7:0/2111351556 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f390ffff640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180cf6c0 0x7f39181f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f39177fe640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39180ca4d0 0x7f3918144450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 -- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180cf6c0 msgr2=0x7f39181f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180cf6c0 0x7f39181f6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 -- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39180ca4d0 msgr2=0x7f3918144450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39180ca4d0 0x7f3918144450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 -- 172.21.15.7:0/2111351556 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910009a70 con 0x7f39181540e0 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f390ffff640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180cf6c0 0x7f39181f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.051+0000 7f3917fff640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39181540e0 0x7f39181f6a40 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f391000efc0 tx=0x7f3910009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f3914ff9640 1 -- 172.21.15.7:0/2111351556 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3910009d70 con 0x7f39181540e0 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f3914ff9640 1 -- 172.21.15.7:0/2111351556 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391000f040 con 0x7f39181540e0 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39000094d0 con 0x7f39181540e0 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f3914ff9640 1 -- 172.21.15.7:0/2111351556 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39100039a0 con 0x7f39181540e0 2023-11-28T18:49:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3918145630 con 0x7f39181540e0 2023-11-28T18:49:18.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391804e890 con 0x7f39181540e0 2023-11-28T18:49:18.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f3914ff9640 1 -- 172.21.15.7:0/2111351556 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f39100144a0 con 0x7f39181540e0 2023-11-28T18:49:18.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f3914ff9640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38f80668d0 0x7f38f8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:18.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f3914ff9640 1 -- 172.21.15.7:0/2111351556 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f39100855a0 con 0x7f39181540e0 2023-11-28T18:49:18.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f39177fe640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38f80668d0 0x7f38f8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:18.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.055+0000 7f39177fe640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38f80668d0 0x7f38f8068d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3908000f70 tx=0x7f3908009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:18.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.059+0000 7f3914ff9640 1 -- 172.21.15.7:0/2111351556 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f39100522c0 con 0x7f39181540e0 2023-11-28T18:49:18.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.251+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39180ce030 con 0x7f39181540e0 2023-11-28T18:49:18.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.251+0000 7f3914ff9640 1 -- 172.21.15.7:0/2111351556 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f39100558e0 con 0x7f39181540e0 2023-11-28T18:49:18.256 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:18.256 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38f80668d0 msgr2=0x7f38f8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38f80668d0 0x7f38f8068d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3908000f70 tx=0x7f3908009040 comp rx=0 tx=0).stop 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39181540e0 msgr2=0x7f39181f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39181540e0 0x7f39181f6a40 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f391000efc0 tx=0x7f3910009f90 comp rx=0 tx=0).stop 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 shutdown_connections 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38f80668d0 0x7f38f8068d90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39181540e0 0x7f39181f6a40 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f39180cf6c0 0x7f39181f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 --2- 172.21.15.7:0/2111351556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39180ca4d0 0x7f3918144450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 >> 172.21.15.7:0/2111351556 conn(0x7f39180662e0 msgr2=0x7f39180c4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 shutdown_connections 2023-11-28T18:49:18.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:18.255+0000 7f391db65640 1 -- 172.21.15.7:0/2111351556 wait complete. 2023-11-28T18:49:18.278 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:20.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:20.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.431+0000 7f8ce55c4640 1 Processor -- start 2023-11-28T18:49:20.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.431+0000 7f8ce55c4640 1 -- start start 2023-11-28T18:49:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.431+0000 7f8ce55c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce0154140 0x7f8ce015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.431+0000 7f8ce55c4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ce0162840 0x7f8ce0164c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.431+0000 7f8ce55c4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce015a350 con 0x7f8ce00c2f60 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.431+0000 7f8ce55c4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce00c6660 con 0x7f8ce0162840 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.431+0000 7f8ce55c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce00c6be0 con 0x7f8ce0154140 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.431+0000 7f8cdf7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ce0162840 0x7f8ce0164c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cdf7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ce0162840 0x7f8ce0164c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51200/0 (socket says 172.21.15.7:51200) 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cdf7fe640 1 -- 172.21.15.7:0/2445810652 learned_addr learned my addr 172.21.15.7:0/2445810652 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cdeffd640 1 --2- 172.21.15.7:0/2445810652 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce0154140 0x7f8ce015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cdf7fe640 1 -- 172.21.15.7:0/2445810652 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce0154140 msgr2=0x7f8ce015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cdf7fe640 1 --2- 172.21.15.7:0/2445810652 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce0154140 0x7f8ce015a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cdf7fe640 1 -- 172.21.15.7:0/2445810652 >> v1:172.21.15.193:6789/0 conn(0x7f8ce00c2f60 legacy=0x7f8ce00c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cdf7fe640 1 -- 172.21.15.7:0/2445810652 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce015e2f0 con 0x7f8ce0162840 2023-11-28T18:49:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cdf7fe640 1 --2- 172.21.15.7:0/2445810652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0162840 0x7f8ce0164c30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8009e50 tx=0x7f8cd801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c1d9a8f26d2c74 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cddffb640 1 -- 172.21.15.7:0/2445810652 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd80af400 con 0x7f8ce0162840 2023-11-28T18:49:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cddffb640 1 -- 172.21.15.7:0/2445810652 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd80ab050 con 0x7f8ce0162840 2023-11-28T18:49:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.435+0000 7f8cddffb640 1 -- 172.21.15.7:0/2445810652 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd80af780 con 0x7f8ce0162840 2023-11-28T18:49:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- 172.21.15.7:0/2445810652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0162840 msgr2=0x7f8ce0164c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/2445810652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0162840 0x7f8ce0164c30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8009e50 tx=0x7f8cd801f1c0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- 172.21.15.7:0/2445810652 shutdown_connections 2023-11-28T18:49:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/2445810652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0162840 0x7f8ce0164c30 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/2445810652 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce0154140 0x7f8ce015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- 172.21.15.7:0/2445810652 >> 172.21.15.7:0/2445810652 conn(0x7f8ce00662e0 msgr2=0x7f8ce00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- 172.21.15.7:0/2445810652 shutdown_connections 2023-11-28T18:49:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- 172.21.15.7:0/2445810652 wait complete. 2023-11-28T18:49:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 Processor -- start 2023-11-28T18:49:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- start start 2023-11-28T18:49:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce00c2f60 0x7f8ce014f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0154140 0x7f8ce014f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ce0162840 0x7f8ce014a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce014daa0 con 0x7f8ce0154140 2023-11-28T18:49:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0073090 con 0x7f8ce0162840 2023-11-28T18:49:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8ce55c4640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0073330 con 0x7f8ce00c2f60 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cde7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0154140 0x7f8ce014f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cde7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0154140 0x7f8ce014f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51206/0 (socket says 172.21.15.7:51206) 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cde7fc640 1 -- 172.21.15.7:0/3099392355 learned_addr learned my addr 172.21.15.7:0/3099392355 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cdeffd640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce00c2f60 0x7f8ce014f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cdf7fe640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ce0162840 0x7f8ce014a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cde7fc640 1 -- 172.21.15.7:0/3099392355 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce00c2f60 msgr2=0x7f8ce014f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cde7fc640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce00c2f60 0x7f8ce014f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cde7fc640 1 -- 172.21.15.7:0/3099392355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ce0162840 msgr2=0x7f8ce014a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cde7fc640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ce0162840 0x7f8ce014a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.439+0000 7f8cde7fc640 1 -- 172.21.15.7:0/3099392355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd8009b30 con 0x7f8ce0154140 2023-11-28T18:49:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8cde7fc640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0154140 0x7f8ce014f990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400c450 tx=0x7f8cd400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8cc77fe640 1 -- 172.21.15.7:0/3099392355 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4012830 con 0x7f8ce0154140 2023-11-28T18:49:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8cc77fe640 1 -- 172.21.15.7:0/3099392355 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd40129b0 con 0x7f8ce0154140 2023-11-28T18:49:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8cc77fe640 1 -- 172.21.15.7:0/3099392355 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4012d60 con 0x7f8ce0154140 2023-11-28T18:49:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce01f3d90 con 0x7f8ce0154140 2023-11-28T18:49:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce0071bc0 con 0x7f8ce0154140 2023-11-28T18:49:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce01f42b0 con 0x7f8ce0154140 2023-11-28T18:49:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8cc77fe640 1 -- 172.21.15.7:0/3099392355 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8cd401c460 con 0x7f8ce0154140 2023-11-28T18:49:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8cc77fe640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ccc0668d0 0x7f8ccc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8cc77fe640 1 -- 172.21.15.7:0/3099392355 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f8cd4084630 con 0x7f8ce0154140 2023-11-28T18:49:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.443+0000 7f8cdeffd640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ccc0668d0 0x7f8ccc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.447+0000 7f8cdeffd640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ccc0668d0 0x7f8ccc068d90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0007a60 tx=0x7f8cd0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:20.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.447+0000 7f8cc77fe640 1 -- 172.21.15.7:0/3099392355 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8cd4051490 con 0x7f8ce0154140 2023-11-28T18:49:20.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ce004e890 con 0x7f8ce0154140 2023-11-28T18:49:20.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8cc77fe640 1 -- 172.21.15.7:0/3099392355 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f8cd4054ab0 con 0x7f8ce0154140 2023-11-28T18:49:20.646 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:20.646 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:20.647 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:20.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ccc0668d0 msgr2=0x7f8ccc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:20.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ccc0668d0 0x7f8ccc068d90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0007a60 tx=0x7f8cd0008040 comp rx=0 tx=0).stop 2023-11-28T18:49:20.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0154140 msgr2=0x7f8ce014f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:20.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0154140 0x7f8ce014f990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400c450 tx=0x7f8cd400c910 comp rx=0 tx=0).stop 2023-11-28T18:49:20.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 shutdown_connections 2023-11-28T18:49:20.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f8ccc0668d0 0x7f8ccc068d90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ce0162840 0x7f8ce014a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ce0154140 0x7f8ce014f990 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 --2- 172.21.15.7:0/3099392355 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f8ce00c2f60 0x7f8ce014f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:20.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 >> 172.21.15.7:0/3099392355 conn(0x7f8ce00662e0 msgr2=0x7f8ce00c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:20.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 shutdown_connections 2023-11-28T18:49:20.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:20.643+0000 7f8ce55c4640 1 -- 172.21.15.7:0/3099392355 wait complete. 2023-11-28T18:49:20.668 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:21.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~242s 2023-11-28T18:49:22.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.819+0000 7fce70454640 1 Processor -- start 2023-11-28T18:49:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.819+0000 7fce70454640 1 -- start start 2023-11-28T18:49:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce70454640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fce68150b30 0x7fce680c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce70454640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681575c0 0x7fce680c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce70454640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce680c6140 con 0x7fce680c4020 2023-11-28T18:49:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce70454640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce680c72b0 con 0x7fce68150b30 2023-11-28T18:49:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce70454640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce68070c00 con 0x7fce681575c0 2023-11-28T18:49:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6e9f1640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fce680c4020 0x7fce680c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39792/0 (socket says 172.21.15.7:39792) 2023-11-28T18:49:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6e9f1640 1 -- 172.21.15.7:0/3062770950 learned_addr learned my addr 172.21.15.7:0/3062770950 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:22.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876223945 0 0) 0x7fce680c6140 con 0x7fce680c4020 2023-11-28T18:49:22.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce54005740 con 0x7fce680c4020 2023-11-28T18:49:22.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d9ef640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681575c0 0x7fce680c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6e1f0640 1 --2- 172.21.15.7:0/3062770950 >> v2:172.21.15.193:3300/0 conn(0x7fce68150b30 0x7fce680c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756246378 0 0) 0x7fce54005740 con 0x7fce680c4020 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681575c0 msgr2=0x7fce680c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681575c0 0x7fce680c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 >> v2:172.21.15.193:3300/0 conn(0x7fce68150b30 msgr2=0x7fce680c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 --2- 172.21.15.7:0/3062770950 >> v2:172.21.15.193:3300/0 conn(0x7fce68150b30 0x7fce680c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce680c6370 con 0x7fce680c4020 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fce60001a00 con 0x7fce680c4020 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6e1f0640 1 --2- 172.21.15.7:0/3062770950 >> v2:172.21.15.193:3300/0 conn(0x7fce68150b30 0x7fce680c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 >> v1:172.21.15.7:6789/0 conn(0x7fce680c4020 legacy=0x7fce680c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681575c0 0x7fce54005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce54006210 0x7fce540069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006f20 0x7fce540073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce680c72b0 con 0x7fce54006210 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5400dd10 con 0x7fce54006f20 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5400ded0 con 0x7fce681575c0 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d9ef640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce54006210 0x7fce540069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6e1f0640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681575c0 0x7fce54005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6e9f1640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006f20 0x7fce540073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d9ef640 1 -- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681575c0 msgr2=0x7fce54005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d9ef640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681575c0 0x7fce54005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d9ef640 1 -- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006f20 msgr2=0x7fce540073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d9ef640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006f20 0x7fce540073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6d9ef640 1 -- 172.21.15.7:0/3062770950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce54005740 con 0x7fce54006210 2023-11-28T18:49:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.823+0000 7fce6e9f1640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006f20 0x7fce540073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:49:22.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce6d9ef640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce54006210 0x7fce540069e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fce5800ba30 tx=0x7fce5801c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8962c8eb4360ad65 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce580a9660 con 0x7fce54006210 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce580a97c0 con 0x7fce54006210 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce6d1ee640 1 -- 172.21.15.7:0/3062770950 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce580a9b70 con 0x7fce54006210 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce70454640 1 -- 172.21.15.7:0/3062770950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce54006210 msgr2=0x7fce540069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce70454640 1 --2- 172.21.15.7:0/3062770950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce54006210 0x7fce540069e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fce5800ba30 tx=0x7fce5801c680 comp rx=0 tx=0).stop 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce6e9f1640 1 -- 172.21.15.7:0/3062770950 reap_dead start 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce70454640 1 -- 172.21.15.7:0/3062770950 shutdown_connections 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce70454640 1 -- 172.21.15.7:0/3062770950 >> 172.21.15.7:0/3062770950 conn(0x7fce680662e0 msgr2=0x7fce680666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce70454640 1 -- 172.21.15.7:0/3062770950 shutdown_connections 2023-11-28T18:49:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.827+0000 7fce70454640 1 -- 172.21.15.7:0/3062770950 wait complete. 2023-11-28T18:49:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 Processor -- start 2023-11-28T18:49:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 -- start start 2023-11-28T18:49:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006210 0x7fce6814b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681efa40 0x7fce680c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68148a70 0x7fce68148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce681572c0 con 0x7fce68148a70 2023-11-28T18:49:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce681ef5b0 con 0x7fce54006210 2023-11-28T18:49:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6814b9a0 con 0x7fce681efa40 2023-11-28T18:49:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68148a70 0x7fce68148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68148a70 0x7fce68148f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51214/0 (socket says 172.21.15.7:51214) 2023-11-28T18:49:22.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 -- 172.21.15.7:0/370826326 learned_addr learned my addr 172.21.15.7:0/370826326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:22.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6d9ef640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681efa40 0x7fce680c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e1f0640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006210 0x7fce6814b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 -- 172.21.15.7:0/370826326 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681efa40 msgr2=0x7fce680c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:22.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681efa40 0x7fce680c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:22.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 -- 172.21.15.7:0/370826326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006210 msgr2=0x7fce6814b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:22.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006210 0x7fce6814b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:22.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 -- 172.21.15.7:0/370826326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce68156d00 con 0x7fce68148a70 2023-11-28T18:49:22.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce6e9f1640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68148a70 0x7fce68148f20 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fce680c3570 tx=0x7fce6000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:22.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce5effd640 1 -- 172.21.15.7:0/370826326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6000eb90 con 0x7fce68148a70 2023-11-28T18:49:22.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce5effd640 1 -- 172.21.15.7:0/370826326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6000ecf0 con 0x7fce68148a70 2023-11-28T18:49:22.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce5effd640 1 -- 172.21.15.7:0/370826326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60013660 con 0x7fce68148a70 2023-11-28T18:49:22.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce68156fe0 con 0x7fce68148a70 2023-11-28T18:49:22.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.831+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce681f3440 con 0x7fce68148a70 2023-11-28T18:49:22.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.835+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce681ef0a0 con 0x7fce68148a70 2023-11-28T18:49:22.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.835+0000 7fce5effd640 1 -- 172.21.15.7:0/370826326 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fce600137c0 con 0x7fce68148a70 2023-11-28T18:49:22.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.835+0000 7fce5effd640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fce4c066900 0x7fce4c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:22.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.835+0000 7fce5effd640 1 -- 172.21.15.7:0/370826326 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fce60084ca0 con 0x7fce68148a70 2023-11-28T18:49:22.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.835+0000 7fce6e1f0640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fce4c066900 0x7fce4c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:22.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.835+0000 7fce6e1f0640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fce4c066900 0x7fce4c068dc0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fce640099d0 tx=0x7fce64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:22.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:22.839+0000 7fce5effd640 1 -- 172.21.15.7:0/370826326 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fce681ef0a0 con 0x7fce68148a70 2023-11-28T18:49:23.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.031+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce681ef0a0 con 0x7fce68148a70 2023-11-28T18:49:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.031+0000 7fce5effd640 1 -- 172.21.15.7:0/370826326 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fce681ef0a0 con 0x7fce68148a70 2023-11-28T18:49:23.036 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:23.036 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fce4c066900 msgr2=0x7fce4c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fce4c066900 0x7fce4c068dc0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fce640099d0 tx=0x7fce64008040 comp rx=0 tx=0).stop 2023-11-28T18:49:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68148a70 msgr2=0x7fce68148f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68148a70 0x7fce68148f20 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fce680c3570 tx=0x7fce6000e530 comp rx=0 tx=0).stop 2023-11-28T18:49:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 shutdown_connections 2023-11-28T18:49:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fce4c066900 0x7fce4c068dc0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68148a70 0x7fce68148f20 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fce681efa40 0x7fce680c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 --2- 172.21.15.7:0/370826326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce54006210 0x7fce6814b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 >> 172.21.15.7:0/370826326 conn(0x7fce680662e0 msgr2=0x7fce680666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 shutdown_connections 2023-11-28T18:49:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:23.035+0000 7fce70454640 1 -- 172.21.15.7:0/370826326 wait complete. 2023-11-28T18:49:23.059 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:49:25.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:25.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f2479640 1 Processor -- start 2023-11-28T18:49:25.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f2479640 1 -- start start 2023-11-28T18:49:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f2479640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f97ec150b30 0x7f97ec0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f2479640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97ec0c7980 0x7f97ec0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f2479640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec0c6140 con 0x7f97ec1575c0 2023-11-28T18:49:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f2479640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec0c72b0 con 0x7f97ec0c7980 2023-11-28T18:49:25.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f2479640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec070c00 con 0x7f97ec150b30 2023-11-28T18:49:25.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f0a16640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97ec0c7980 0x7f97ec0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:25.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f0a16640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97ec0c7980 0x7f97ec0c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34014/0 (socket says 172.21.15.7:34014) 2023-11-28T18:49:25.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97f0a16640 1 -- 172.21.15.7:0/3160292182 learned_addr learned my addr 172.21.15.7:0/3160292182 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:25.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.211+0000 7f97ebfff640 1 --2- 172.21.15.7:0/3160292182 >> v2:172.21.15.193:3300/0 conn(0x7f97ec150b30 0x7f97ec0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:25.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f0a16640 1 -- 172.21.15.7:0/3160292182 >> v2:172.21.15.193:3300/0 conn(0x7f97ec150b30 msgr2=0x7f97ec0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:25.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f0a16640 1 --2- 172.21.15.7:0/3160292182 >> v2:172.21.15.193:3300/0 conn(0x7f97ec150b30 0x7f97ec0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f0a16640 1 -- 172.21.15.7:0/3160292182 >> v1:172.21.15.193:6789/0 conn(0x7f97ec1575c0 legacy=0x7f97ec0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:25.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f0a16640 1 -- 172.21.15.7:0/3160292182 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0c8310 con 0x7f97ec0c7980 2023-11-28T18:49:25.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f0a16640 1 --2- 172.21.15.7:0/3160292182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec0c7980 0x7f97ec0c7dd0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f97ec06ec80 tx=0x7f97e401f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17038310f315ef2c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97eaffd640 1 -- 172.21.15.7:0/3160292182 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40af400 con 0x7f97ec0c7980 2023-11-28T18:49:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97eaffd640 1 -- 172.21.15.7:0/3160292182 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e40ab050 con 0x7f97ec0c7980 2023-11-28T18:49:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97eaffd640 1 -- 172.21.15.7:0/3160292182 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40af780 con 0x7f97ec0c7980 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f2479640 1 -- 172.21.15.7:0/3160292182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec0c7980 msgr2=0x7f97ec0c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f2479640 1 --2- 172.21.15.7:0/3160292182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec0c7980 0x7f97ec0c7dd0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f97ec06ec80 tx=0x7f97e401f180 comp rx=0 tx=0).stop 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f2479640 1 -- 172.21.15.7:0/3160292182 shutdown_connections 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f2479640 1 --2- 172.21.15.7:0/3160292182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec0c7980 0x7f97ec0c7dd0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f2479640 1 --2- 172.21.15.7:0/3160292182 >> v2:172.21.15.193:3300/0 conn(0x7f97ec150b30 0x7f97ec0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f2479640 1 -- 172.21.15.7:0/3160292182 >> 172.21.15.7:0/3160292182 conn(0x7f97ec0662e0 msgr2=0x7f97ec0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f2479640 1 -- 172.21.15.7:0/3160292182 shutdown_connections 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.215+0000 7f97f2479640 1 -- 172.21.15.7:0/3160292182 wait complete. 2023-11-28T18:49:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f2479640 1 Processor -- start 2023-11-28T18:49:25.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f2479640 1 -- start start 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f2479640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97ec0c7980 0x7f97ec1df2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f2479640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ec150b30 0x7f97ec072530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f2479640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec1575c0 0x7f97ec072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f2479640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec154150 con 0x7f97ec1575c0 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f2479640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec1544c0 con 0x7f97ec150b30 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f2479640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec154910 con 0x7f97ec0c7980 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec1575c0 0x7f97ec072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec1575c0 0x7f97ec072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34022/0 (socket says 172.21.15.7:34022) 2023-11-28T18:49:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 -- 172.21.15.7:0/212377636 learned_addr learned my addr 172.21.15.7:0/212377636 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97ebfff640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97ec0c7980 0x7f97ec1df2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97eb7fe640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ec150b30 0x7f97ec072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 -- 172.21.15.7:0/212377636 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97ec0c7980 msgr2=0x7f97ec1df2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97ec0c7980 0x7f97ec1df2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 -- 172.21.15.7:0/212377636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ec150b30 msgr2=0x7f97ec072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ec150b30 0x7f97ec072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 -- 172.21.15.7:0/212377636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009b30 con 0x7f97ec1575c0 2023-11-28T18:49:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.219+0000 7f97f0a16640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec1575c0 0x7f97ec072a70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f97e401cdb0 tx=0x7f97e401bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97e8ff9640 1 -- 172.21.15.7:0/212377636 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40afe00 con 0x7f97ec1575c0 2023-11-28T18:49:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97e8ff9640 1 -- 172.21.15.7:0/212377636 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e4009510 con 0x7f97ec1575c0 2023-11-28T18:49:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec14a130 con 0x7f97ec1575c0 2023-11-28T18:49:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97e8ff9640 1 -- 172.21.15.7:0/212377636 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40ba660 con 0x7f97ec1575c0 2023-11-28T18:49:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec154e70 con 0x7f97ec1575c0 2023-11-28T18:49:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec04e890 con 0x7f97ec1575c0 2023-11-28T18:49:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97e8ff9640 1 -- 172.21.15.7:0/212377636 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f97e40b6040 con 0x7f97ec1575c0 2023-11-28T18:49:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97e8ff9640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97d80668d0 0x7f97d8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97e8ff9640 1 -- 172.21.15.7:0/212377636 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f97e4122d10 con 0x7f97ec1575c0 2023-11-28T18:49:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97ebfff640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97d80668d0 0x7f97d8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.223+0000 7f97ebfff640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97d80668d0 0x7f97d8068d90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00b9d0 tx=0x7f97dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.227+0000 7f97e8ff9640 1 -- 172.21.15.7:0/212377636 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f97e40ef9a0 con 0x7f97ec1575c0 2023-11-28T18:49:25.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.423+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97ec0c2d80 con 0x7f97ec1575c0 2023-11-28T18:49:25.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.423+0000 7f97e8ff9640 1 -- 172.21.15.7:0/212377636 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f97e40f2fc0 con 0x7f97ec1575c0 2023-11-28T18:49:25.427 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:25.428 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:25.428 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:25.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97d80668d0 msgr2=0x7f97d8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:25.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97d80668d0 0x7f97d8068d90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00b9d0 tx=0x7f97dc008040 comp rx=0 tx=0).stop 2023-11-28T18:49:25.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec1575c0 msgr2=0x7f97ec072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:25.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec1575c0 0x7f97ec072a70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f97e401cdb0 tx=0x7f97e401bf20 comp rx=0 tx=0).stop 2023-11-28T18:49:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 shutdown_connections 2023-11-28T18:49:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f97d80668d0 0x7f97d8068d90 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97ec1575c0 0x7f97ec072a70 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ec150b30 0x7f97ec072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 --2- 172.21.15.7:0/212377636 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f97ec0c7980 0x7f97ec1df2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 >> 172.21.15.7:0/212377636 conn(0x7f97ec0662e0 msgr2=0x7f97ec14f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 shutdown_connections 2023-11-28T18:49:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:25.427+0000 7f97f2479640 1 -- 172.21.15.7:0/212377636 wait complete. 2023-11-28T18:49:25.449 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:27.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:27.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe81930f640 1 Processor -- start 2023-11-28T18:49:27.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe81930f640 1 -- start start 2023-11-28T18:49:27.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe81930f640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe8141503c0 0x7fe8140c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:27.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe81930f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8140c38e0 con 0x7fe8140c3bc0 2023-11-28T18:49:27.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe81930f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8140c2ce0 con 0x7fe8140c3490 2023-11-28T18:49:27.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe81930f640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe814157240 con 0x7fe8141503c0 2023-11-28T18:49:27.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe8127fc640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe8140c3bc0 0x7fe81406d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44498/0 (socket says 172.21.15.7:44498) 2023-11-28T18:49:27.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe8127fc640 1 -- 172.21.15.7:0/2642138178 learned_addr learned my addr 172.21.15.7:0/2642138178 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:27.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.603+0000 7fe812ffd640 1 --2- 172.21.15.7:0/2642138178 >> v2:172.21.15.193:3300/0 conn(0x7fe8141503c0 0x7fe8140c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:27.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe811ffb640 1 -- 172.21.15.7:0/2642138178 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454487841 0 0) 0x7fe8140c38e0 con 0x7fe8140c3bc0 2023-11-28T18:49:27.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe811ffb640 1 -- 172.21.15.7:0/2642138178 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7fc003530 con 0x7fe8140c3bc0 2023-11-28T18:49:27.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe812ffd640 1 -- 172.21.15.7:0/2642138178 >> v1:172.21.15.193:6789/0 conn(0x7fe8140c3490 legacy=0x7fe8140c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:27.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe812ffd640 1 -- 172.21.15.7:0/2642138178 >> v1:172.21.15.7:6789/0 conn(0x7fe8140c3bc0 legacy=0x7fe81406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:27.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe812ffd640 1 -- 172.21.15.7:0/2642138178 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8140c2f80 con 0x7fe8141503c0 2023-11-28T18:49:27.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe812ffd640 1 --2- 172.21.15.7:0/2642138178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8141503c0 0x7fe8140c7c60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe8040097c0 tx=0x7fe80401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4eaeb31a245ab49 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:27.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe811ffb640 1 -- 172.21.15.7:0/2642138178 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8040a8970 con 0x7fe8141503c0 2023-11-28T18:49:27.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe811ffb640 1 -- 172.21.15.7:0/2642138178 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8040a8ad0 con 0x7fe8141503c0 2023-11-28T18:49:27.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.607+0000 7fe811ffb640 1 -- 172.21.15.7:0/2642138178 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8040a8e60 con 0x7fe8141503c0 2023-11-28T18:49:27.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- 172.21.15.7:0/2642138178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8141503c0 msgr2=0x7fe8140c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:27.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 --2- 172.21.15.7:0/2642138178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8141503c0 0x7fe8140c7c60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe8040097c0 tx=0x7fe80401ec70 comp rx=0 tx=0).stop 2023-11-28T18:49:27.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- 172.21.15.7:0/2642138178 shutdown_connections 2023-11-28T18:49:27.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 --2- 172.21.15.7:0/2642138178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8141503c0 0x7fe8140c7c60 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:27.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- 172.21.15.7:0/2642138178 >> 172.21.15.7:0/2642138178 conn(0x7fe8140662e0 msgr2=0x7fe8140666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:27.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- 172.21.15.7:0/2642138178 shutdown_connections 2023-11-28T18:49:27.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- 172.21.15.7:0/2642138178 wait complete. 2023-11-28T18:49:27.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 Processor -- start 2023-11-28T18:49:27.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- start start 2023-11-28T18:49:27.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8140c3490 0x7fe814150b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8140c3bc0 0x7fe814154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8141503c0 0x7fe81414a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8141e3370 con 0x7fe8141503c0 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8141e3970 con 0x7fe8140c3490 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe81930f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe814151050 con 0x7fe8140c3bc0 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe8137fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8141503c0 0x7fe81414a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe8137fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8141503c0 0x7fe81414a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34024/0 (socket says 172.21.15.7:34024) 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe8137fe640 1 -- 172.21.15.7:0/3683868251 learned_addr learned my addr 172.21.15.7:0/3683868251 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe8127fc640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8140c3bc0 0x7fe814154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.611+0000 7fe812ffd640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8140c3490 0x7fe814150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe8137fe640 1 -- 172.21.15.7:0/3683868251 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8140c3bc0 msgr2=0x7fe814154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe8137fe640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8140c3bc0 0x7fe814154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:27.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe8137fe640 1 -- 172.21.15.7:0/3683868251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8140c3490 msgr2=0x7fe814150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:27.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe8137fe640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8140c3490 0x7fe814150b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:27.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe8137fe640 1 -- 172.21.15.7:0/3683868251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe804009420 con 0x7fe8141503c0 2023-11-28T18:49:27.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe8137fe640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8141503c0 0x7fe81414a060 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c770 tx=0x7fe80c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:27.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe7fb7fe640 1 -- 172.21.15.7:0/3683868251 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c003b80 con 0x7fe8141503c0 2023-11-28T18:49:27.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe7fb7fe640 1 -- 172.21.15.7:0/3683868251 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c003ce0 con 0x7fe8141503c0 2023-11-28T18:49:27.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe7fb7fe640 1 -- 172.21.15.7:0/3683868251 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c00aa30 con 0x7fe8141503c0 2023-11-28T18:49:27.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8141556a0 con 0x7fe8141503c0 2023-11-28T18:49:27.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8141f3310 con 0x7fe8141503c0 2023-11-28T18:49:27.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81404e890 con 0x7fe8141503c0 2023-11-28T18:49:27.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.615+0000 7fe7fb7fe640 1 -- 172.21.15.7:0/3683868251 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe80c00ab90 con 0x7fe8141503c0 2023-11-28T18:49:27.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.619+0000 7fe7fb7fe640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe800066900 0x7fe800068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:27.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.619+0000 7fe7fb7fe640 1 -- 172.21.15.7:0/3683868251 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fe80c084550 con 0x7fe8141503c0 2023-11-28T18:49:27.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.619+0000 7fe812ffd640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe800066900 0x7fe800068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:27.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.619+0000 7fe812ffd640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe800066900 0x7fe800068dc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe8040b2000 tx=0x7fe804008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:27.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.623+0000 7fe7fb7fe640 1 -- 172.21.15.7:0/3683868251 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe80c051320 con 0x7fe8141503c0 2023-11-28T18:49:27.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8141f3570 con 0x7fe8141503c0 2023-11-28T18:49:27.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe7fb7fe640 1 -- 172.21.15.7:0/3683868251 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fe80c054940 con 0x7fe8141503c0 2023-11-28T18:49:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:27.822 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:27.822 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:27.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe800066900 msgr2=0x7fe800068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:27.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe800066900 0x7fe800068dc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe8040b2000 tx=0x7fe804008040 comp rx=0 tx=0).stop 2023-11-28T18:49:27.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8141503c0 msgr2=0x7fe81414a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:27.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8141503c0 0x7fe81414a060 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c770 tx=0x7fe80c00cc30 comp rx=0 tx=0).stop 2023-11-28T18:49:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 shutdown_connections 2023-11-28T18:49:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe800066900 0x7fe800068dc0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8141503c0 0x7fe81414a060 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe8140c3bc0 0x7fe814154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 --2- 172.21.15.7:0/3683868251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8140c3490 0x7fe814150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 >> 172.21.15.7:0/3683868251 conn(0x7fe8140662e0 msgr2=0x7fe8140c7280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 shutdown_connections 2023-11-28T18:49:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:27.819+0000 7fe81930f640 1 -- 172.21.15.7:0/3683868251 wait complete. 2023-11-28T18:49:27.843 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:28.374 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~249s 2023-11-28T18:49:29.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.995+0000 7f38eab55640 1 Processor -- start 2023-11-28T18:49:30.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.995+0000 7f38eab55640 1 -- start start 2023-11-28T18:49:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38eab55640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f38e4073070 0x7f38e4074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38eab55640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f38e406c920 0x7f38e406cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38eab55640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e406d310 con 0x7f38e4072950 2023-11-28T18:49:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38eab55640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e40c8490 con 0x7f38e406c920 2023-11-28T18:49:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38eab55640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e40c81f0 con 0x7f38e4073070 2023-11-28T18:49:30.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e90f2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f38e406c920 0x7f38e406cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:30.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e88f1640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f38e4072950 0x7f38e40c19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44504/0 (socket says 172.21.15.7:44504) 2023-11-28T18:49:30.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e88f1640 1 -- 172.21.15.7:0/1101689651 learned_addr learned my addr 172.21.15.7:0/1101689651 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:30.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e3fff640 1 --2- 172.21.15.7:0/1101689651 >> v2:172.21.15.193:3300/0 conn(0x7f38e4073070 0x7f38e4074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:30.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e90f2640 1 -- 172.21.15.7:0/1101689651 >> v2:172.21.15.193:3300/0 conn(0x7f38e4073070 msgr2=0x7f38e4074030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:30.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e90f2640 1 --2- 172.21.15.7:0/1101689651 >> v2:172.21.15.193:3300/0 conn(0x7f38e4073070 0x7f38e4074030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e90f2640 1 -- 172.21.15.7:0/1101689651 >> v1:172.21.15.7:6789/0 conn(0x7f38e4072950 legacy=0x7f38e40c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e90f2640 1 -- 172.21.15.7:0/1101689651 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e415ab00 con 0x7f38e406c920 2023-11-28T18:49:30.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:29.999+0000 7f38e90f2640 1 --2- 172.21.15.7:0/1101689651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e406c920 0x7f38e406cdd0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f38e40c13c0 tx=0x7f38dc01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3197395cb8d2ba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:30.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38e37fe640 1 -- 172.21.15.7:0/1101689651 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc0a66a0 con 0x7f38e406c920 2023-11-28T18:49:30.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38e37fe640 1 -- 172.21.15.7:0/1101689651 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc0a6800 con 0x7f38e406c920 2023-11-28T18:49:30.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38e37fe640 1 -- 172.21.15.7:0/1101689651 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc0a6b90 con 0x7f38e406c920 2023-11-28T18:49:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- 172.21.15.7:0/1101689651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e406c920 msgr2=0x7f38e406cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 --2- 172.21.15.7:0/1101689651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e406c920 0x7f38e406cdd0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f38e40c13c0 tx=0x7f38dc01c7f0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- 172.21.15.7:0/1101689651 shutdown_connections 2023-11-28T18:49:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 --2- 172.21.15.7:0/1101689651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e406c920 0x7f38e406cdd0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 --2- 172.21.15.7:0/1101689651 >> v2:172.21.15.193:3300/0 conn(0x7f38e4073070 0x7f38e4074030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- 172.21.15.7:0/1101689651 >> 172.21.15.7:0/1101689651 conn(0x7f38e40cbf80 msgr2=0x7f38e40cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- 172.21.15.7:0/1101689651 shutdown_connections 2023-11-28T18:49:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- 172.21.15.7:0/1101689651 wait complete. 2023-11-28T18:49:30.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 Processor -- start 2023-11-28T18:49:30.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- start start 2023-11-28T18:49:30.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f38e406c920 0x7f38e41f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:30.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e4072950 0x7f38e4149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e4073070 0x7f38e41f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e414ad90 con 0x7f38e4072950 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e41f6810 con 0x7f38e4073070 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38eab55640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e4143aa0 con 0x7f38e406c920 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.003+0000 7f38e3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e4072950 0x7f38e4149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e4072950 0x7f38e4149900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34050/0 (socket says 172.21.15.7:34050) 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e3fff640 1 -- 172.21.15.7:0/4085245788 learned_addr learned my addr 172.21.15.7:0/4085245788 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e90f2640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e4073070 0x7f38e41f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e88f1640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f38e406c920 0x7f38e41f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e3fff640 1 -- 172.21.15.7:0/4085245788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f38e406c920 msgr2=0x7f38e41f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e3fff640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f38e406c920 0x7f38e41f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e3fff640 1 -- 172.21.15.7:0/4085245788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e4073070 msgr2=0x7f38e41f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e3fff640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e4073070 0x7f38e41f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e3fff640 1 -- 172.21.15.7:0/4085245788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc009a70 con 0x7f38e4072950 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e3fff640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e4072950 0x7f38e4149900 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f38d000e840 tx=0x7f38d000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e17fa640 1 -- 172.21.15.7:0/4085245788 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d0009510 con 0x7f38e4072950 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e17fa640 1 -- 172.21.15.7:0/4085245788 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d000ee90 con 0x7f38e4072950 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e17fa640 1 -- 172.21.15.7:0/4085245788 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d000cb60 con 0x7f38e4072950 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e4143d40 con 0x7f38e4072950 2023-11-28T18:49:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e4144300 con 0x7f38e4072950 2023-11-28T18:49:30.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e4144b00 con 0x7f38e4072950 2023-11-28T18:49:30.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.007+0000 7f38e17fa640 1 -- 172.21.15.7:0/4085245788 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f38d000ccc0 con 0x7f38e4072950 2023-11-28T18:49:30.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.011+0000 7f38e17fa640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38c8066900 0x7f38c8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:30.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.011+0000 7f38e17fa640 1 -- 172.21.15.7:0/4085245788 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f38d0084990 con 0x7f38e4072950 2023-11-28T18:49:30.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.011+0000 7f38e88f1640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38c8066900 0x7f38c8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:30.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.011+0000 7f38e88f1640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38c8066900 0x7f38c8068dc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f38d8005890 tx=0x7f38d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:30.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.011+0000 7f38e17fa640 1 -- 172.21.15.7:0/4085245788 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f38d0051760 con 0x7f38e4072950 2023-11-28T18:49:30.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38e40cca70 con 0x7f38e4072950 2023-11-28T18:49:30.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38e17fa640 1 -- 172.21.15.7:0/4085245788 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f38d0054d80 con 0x7f38e4072950 2023-11-28T18:49:30.210 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:30.210 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38c8066900 msgr2=0x7f38c8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38c8066900 0x7f38c8068dc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f38d8005890 tx=0x7f38d8009040 comp rx=0 tx=0).stop 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e4072950 msgr2=0x7f38e4149900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e4072950 0x7f38e4149900 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f38d000e840 tx=0x7f38d000ec40 comp rx=0 tx=0).stop 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 shutdown_connections 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f38c8066900 0x7f38c8068dc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e4073070 0x7f38e41f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38e4072950 0x7f38e4149900 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 --2- 172.21.15.7:0/4085245788 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f38e406c920 0x7f38e41f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 >> 172.21.15.7:0/4085245788 conn(0x7f38e40cbf80 msgr2=0x7f38e40c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:30.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 shutdown_connections 2023-11-28T18:49:30.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:30.207+0000 7f38eab55640 1 -- 172.21.15.7:0/4085245788 wait complete. 2023-11-28T18:49:30.232 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:32.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:32.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.383+0000 7f86cc157640 1 Processor -- start 2023-11-28T18:49:32.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.383+0000 7f86cc157640 1 -- start start 2023-11-28T18:49:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86cc157640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86c4151cd0 0x7f86c415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86cc157640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f86c406e560 0x7f86c4157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86cc157640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c4071e60 con 0x7f86c40701f0 2023-11-28T18:49:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86cc157640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c4070930 con 0x7f86c406e560 2023-11-28T18:49:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86cc157640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4070660 con 0x7f86c4151cd0 2023-11-28T18:49:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86c96f2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f86c406e560 0x7f86c4157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86ca6f4640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f86c40701f0 0x7f86c40c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44516/0 (socket says 172.21.15.7:44516) 2023-11-28T18:49:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86ca6f4640 1 -- 172.21.15.7:0/160534841 learned_addr learned my addr 172.21.15.7:0/160534841 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86c96f2640 1 -- 172.21.15.7:0/160534841 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86c4151cd0 msgr2=0x7f86c415bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86c96f2640 1 --2- 172.21.15.7:0/160534841 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86c4151cd0 0x7f86c415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86c96f2640 1 -- 172.21.15.7:0/160534841 >> v1:172.21.15.7:6789/0 conn(0x7f86c40701f0 legacy=0x7f86c40c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86c8ef1640 1 -- 172.21.15.7:0/160534841 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217507283 0 0) 0x7f86c4071e60 con 0x7f86c40701f0 2023-11-28T18:49:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.387+0000 7f86c96f2640 1 -- 172.21.15.7:0/160534841 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4070bf0 con 0x7f86c406e560 2023-11-28T18:49:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86c96f2640 1 --2- 172.21.15.7:0/160534841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c4157b90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f86b40097c0 tx=0x7f86b401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8c428a849be363 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86c8ef1640 1 -- 172.21.15.7:0/160534841 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40ab600 con 0x7f86c406e560 2023-11-28T18:49:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86c8ef1640 1 -- 172.21.15.7:0/160534841 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b40a7050 con 0x7f86c406e560 2023-11-28T18:49:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86c8ef1640 1 -- 172.21.15.7:0/160534841 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40ab980 con 0x7f86c406e560 2023-11-28T18:49:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 -- 172.21.15.7:0/160534841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 msgr2=0x7f86c4157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 --2- 172.21.15.7:0/160534841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c4157b90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f86b40097c0 tx=0x7f86b401c3f0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 -- 172.21.15.7:0/160534841 shutdown_connections 2023-11-28T18:49:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 --2- 172.21.15.7:0/160534841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c4157b90 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 --2- 172.21.15.7:0/160534841 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86c4151cd0 0x7f86c415bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 -- 172.21.15.7:0/160534841 >> 172.21.15.7:0/160534841 conn(0x7f86c40662e0 msgr2=0x7f86c40666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:32.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 -- 172.21.15.7:0/160534841 shutdown_connections 2023-11-28T18:49:32.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 -- 172.21.15.7:0/160534841 wait complete. 2023-11-28T18:49:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 Processor -- start 2023-11-28T18:49:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.391+0000 7f86cc157640 1 -- start start 2023-11-28T18:49:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c40c3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86c40701f0 0x7f86c4156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86c4151cd0 0x7f86c4159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c414fee0 con 0x7f86c406e560 2023-11-28T18:49:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40c6440 con 0x7f86c40701f0 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40c43e0 con 0x7f86c4151cd0 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c40c3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c40c3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34070/0 (socket says 172.21.15.7:34070) 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 -- 172.21.15.7:0/1765226730 learned_addr learned my addr 172.21.15.7:0/1765226730 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 -- 172.21.15.7:0/1765226730 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86c4151cd0 msgr2=0x7f86c4159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86c4151cd0 0x7f86c4159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 -- 172.21.15.7:0/1765226730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86c40701f0 msgr2=0x7f86c4156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86c40701f0 0x7f86c4156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 -- 172.21.15.7:0/1765226730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b4009420 con 0x7f86c406e560 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86c9ef3640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c40c3dc0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f86c000b410 tx=0x7f86c000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86ba7fc640 1 -- 172.21.15.7:0/1765226730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0011830 con 0x7f86c406e560 2023-11-28T18:49:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86ba7fc640 1 -- 172.21.15.7:0/1765226730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c0011990 con 0x7f86c406e560 2023-11-28T18:49:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86ba7fc640 1 -- 172.21.15.7:0/1765226730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0011d40 con 0x7f86c406e560 2023-11-28T18:49:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c415a3e0 con 0x7f86c406e560 2023-11-28T18:49:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c415a9a0 con 0x7f86c406e560 2023-11-28T18:49:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c404e890 con 0x7f86c406e560 2023-11-28T18:49:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.395+0000 7f86ba7fc640 1 -- 172.21.15.7:0/1765226730 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f86c001a400 con 0x7f86c406e560 2023-11-28T18:49:32.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.399+0000 7f86ba7fc640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86a80668d0 0x7f86a8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:32.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.399+0000 7f86ba7fc640 1 -- 172.21.15.7:0/1765226730 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f86c0084b70 con 0x7f86c406e560 2023-11-28T18:49:32.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.399+0000 7f86c96f2640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86a80668d0 0x7f86a8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:32.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.399+0000 7f86c96f2640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86a80668d0 0x7f86a8068d90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f86b40ad000 tx=0x7f86b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.403+0000 7f86ba7fc640 1 -- 172.21.15.7:0/1765226730 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f86c0051800 con 0x7f86c406e560 2023-11-28T18:49:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.591+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86c4066ad0 con 0x7f86c406e560 2023-11-28T18:49:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.591+0000 7f86ba7fc640 1 -- 172.21.15.7:0/1765226730 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f86c0015020 con 0x7f86c406e560 2023-11-28T18:49:32.596 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:32.596 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86a80668d0 msgr2=0x7f86a8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86a80668d0 0x7f86a8068d90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f86b40ad000 tx=0x7f86b4008040 comp rx=0 tx=0).stop 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 msgr2=0x7f86c40c3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c40c3dc0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f86c000b410 tx=0x7f86c000b8d0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 shutdown_connections 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f86a80668d0 0x7f86a8068d90 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f86c4151cd0 0x7f86c4159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86c40701f0 0x7f86c4156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 --2- 172.21.15.7:0/1765226730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86c406e560 0x7f86c40c3dc0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 >> 172.21.15.7:0/1765226730 conn(0x7f86c40662e0 msgr2=0x7f86c414b110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 shutdown_connections 2023-11-28T18:49:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:32.595+0000 7f86cc157640 1 -- 172.21.15.7:0/1765226730 wait complete. 2023-11-28T18:49:32.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:34.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.775+0000 7f36bb155640 1 Processor -- start 2023-11-28T18:49:34.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36bb155640 1 -- start start 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36bb155640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b406a8a0 0x7f36b40c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36bb155640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40c8430 con 0x7f36b415a5b0 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36bb155640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b406ae50 con 0x7f36b4154040 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36bb155640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b406b110 con 0x7f36b406a8a0 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f36b415a5b0 0x7f36b40c89b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37992/0 (socket says 172.21.15.7:37992) 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b3fff640 1 -- 172.21.15.7:0/1841420707 learned_addr learned my addr 172.21.15.7:0/1841420707 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b96f2640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b406a8a0 0x7f36b40c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546317176 0 0) 0x7f36b40c8430 con 0x7f36b415a5b0 2023-11-28T18:49:34.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a4003530 con 0x7f36b415a5b0 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 761042847 0 0) 0x7f36a4003530 con 0x7f36b415a5b0 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b406a8a0 msgr2=0x7f36b40c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b406a8a0 0x7f36b40c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 >> v1:172.21.15.193:6789/0 conn(0x7f36b4154040 legacy=0x7f36b4159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40c7820 con 0x7f36b415a5b0 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f36a0001680 con 0x7f36b415a5b0 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 >> v1:172.21.15.7:6789/0 conn(0x7f36b415a5b0 legacy=0x7f36b40c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a40039b0 0x7f36a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b406a8a0 0x7f36a40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a4004d10 0x7f36a4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:34.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b406b110 con 0x7f36a40039b0 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a400b990 con 0x7f36a4004d10 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a400bb50 con 0x7f36b406a8a0 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b8ef1640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a40039b0 0x7f36a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b3fff640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b406a8a0 0x7f36a40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b96f2640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a4004d10 0x7f36a4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b8ef1640 1 -- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b406a8a0 msgr2=0x7f36a40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b8ef1640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b406a8a0 0x7f36a40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b8ef1640 1 -- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a4004d10 msgr2=0x7f36a4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b8ef1640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a4004d10 0x7f36a4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.779+0000 7f36b8ef1640 1 -- 172.21.15.7:0/1841420707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a4003530 con 0x7f36a40039b0 2023-11-28T18:49:34.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36b8ef1640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a40039b0 0x7f36a4003d90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f36a8001860 tx=0x7f36a801ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a221c67627a7fb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:34.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a80a87d0 con 0x7f36a40039b0 2023-11-28T18:49:34.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a80a8930 con 0x7f36a40039b0 2023-11-28T18:49:34.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36b37fe640 1 -- 172.21.15.7:0/1841420707 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a80a8ce0 con 0x7f36a40039b0 2023-11-28T18:49:34.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36bb155640 1 -- 172.21.15.7:0/1841420707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a40039b0 msgr2=0x7f36a4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:34.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36bb155640 1 --2- 172.21.15.7:0/1841420707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a40039b0 0x7f36a4003d90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f36a8001860 tx=0x7f36a801ebb0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36b96f2640 1 -- 172.21.15.7:0/1841420707 reap_dead start 2023-11-28T18:49:34.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36bb155640 1 -- 172.21.15.7:0/1841420707 shutdown_connections 2023-11-28T18:49:34.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36bb155640 1 -- 172.21.15.7:0/1841420707 >> 172.21.15.7:0/1841420707 conn(0x7f36b406d4d0 msgr2=0x7f36b406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:34.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36bb155640 1 -- 172.21.15.7:0/1841420707 shutdown_connections 2023-11-28T18:49:34.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.783+0000 7f36bb155640 1 -- 172.21.15.7:0/1841420707 wait complete. 2023-11-28T18:49:34.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 Processor -- start 2023-11-28T18:49:34.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 -- start start 2023-11-28T18:49:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a40039b0 0x7f36b40be5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b40c3200 0x7f36b4066540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36b4156e00 0x7f36b41547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b41f36f0 con 0x7f36b4156e00 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b4156680 con 0x7f36a40039b0 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b4156250 con 0x7f36b40c3200 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36b4156e00 0x7f36b41547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36b4156e00 0x7f36b41547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45468/0 (socket says 172.21.15.7:45468) 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 -- 172.21.15.7:0/2883050305 learned_addr learned my addr 172.21.15.7:0/2883050305 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b3fff640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b40c3200 0x7f36b4066540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b8ef1640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a40039b0 0x7f36b40be5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 -- 172.21.15.7:0/2883050305 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b40c3200 msgr2=0x7f36b4066540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b40c3200 0x7f36b4066540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 -- 172.21.15.7:0/2883050305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a40039b0 msgr2=0x7f36b40be5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a40039b0 0x7f36b40be5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 -- 172.21.15.7:0/2883050305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40bd4e0 con 0x7f36b4156e00 2023-11-28T18:49:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b96f2640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36b4156e00 0x7f36b41547d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f36b415a990 tx=0x7f36ac00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b17fa640 1 -- 172.21.15.7:0/2883050305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac00e040 con 0x7f36b4156e00 2023-11-28T18:49:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b17fa640 1 -- 172.21.15.7:0/2883050305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac00f040 con 0x7f36b4156e00 2023-11-28T18:49:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36b17fa640 1 -- 172.21.15.7:0/2883050305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac0039a0 con 0x7f36b4156e00 2023-11-28T18:49:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b41f3430 con 0x7f36b4156e00 2023-11-28T18:49:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.787+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b41f3140 con 0x7f36b4156e00 2023-11-28T18:49:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.791+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b404e890 con 0x7f36b4156e00 2023-11-28T18:49:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.791+0000 7f36b17fa640 1 -- 172.21.15.7:0/2883050305 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f36ac003b00 con 0x7f36b4156e00 2023-11-28T18:49:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.791+0000 7f36b17fa640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36980668d0 0x7f3698068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.791+0000 7f36b17fa640 1 -- 172.21.15.7:0/2883050305 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f36ac0850e0 con 0x7f36b4156e00 2023-11-28T18:49:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.791+0000 7f36b8ef1640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36980668d0 0x7f3698068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.791+0000 7f36b8ef1640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36980668d0 0x7f3698068d90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f36a80b2000 tx=0x7f36a80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.795+0000 7f36b17fa640 1 -- 172.21.15.7:0/2883050305 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f36ac051f40 con 0x7f36b4156e00 2023-11-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36b4159c10 con 0x7f36b4156e00 2023-11-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36b17fa640 1 -- 172.21.15.7:0/2883050305 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f36ac055560 con 0x7f36b4156e00 2023-11-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:34.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36980668d0 msgr2=0x7f3698068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:34.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36980668d0 0x7f3698068d90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f36a80b2000 tx=0x7f36a80aa070 comp rx=0 tx=0).stop 2023-11-28T18:49:34.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36b4156e00 msgr2=0x7f36b41547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36b4156e00 0x7f36b41547d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f36b415a990 tx=0x7f36ac00f580 comp rx=0 tx=0).stop 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 shutdown_connections 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f36980668d0 0x7f3698068d90 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36b4156e00 0x7f36b41547d0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36b40c3200 0x7f36b4066540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 --2- 172.21.15.7:0/2883050305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a40039b0 0x7f36b40be5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 >> 172.21.15.7:0/2883050305 conn(0x7f36b406d4d0 msgr2=0x7f36b40ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 shutdown_connections 2023-11-28T18:49:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:34.987+0000 7f36bb155640 1 -- 172.21.15.7:0/2883050305 wait complete. 2023-11-28T18:49:35.009 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:49:35.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~256s 2023-11-28T18:49:37.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:37.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.163+0000 7fbd4bb1d640 1 Processor -- start 2023-11-28T18:49:37.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4bb1d640 1 -- start start 2023-11-28T18:49:37.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4bb1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440c7600 0x7fbd4415a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:37.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4bb1d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd440c1b90 con 0x7fbd440c1700 2023-11-28T18:49:37.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4bb1d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4406ccc0 con 0x7fbd44150840 2023-11-28T18:49:37.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4bb1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4406cf90 con 0x7fbd440c7600 2023-11-28T18:49:37.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4a0ba640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbd440c1700 0x7fbd4406c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38004/0 (socket says 172.21.15.7:38004) 2023-11-28T18:49:37.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4a0ba640 1 -- 172.21.15.7:0/389854263 learned_addr learned my addr 172.21.15.7:0/389854263 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd490b8640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440c7600 0x7fbd4415a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851731175 0 0) 0x7fbd440c1b90 con 0x7fbd440c1700 2023-11-28T18:49:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd30003530 con 0x7fbd440c1700 2023-11-28T18:49:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 844998485 0 0) 0x7fbd30003530 con 0x7fbd440c1700 2023-11-28T18:49:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440c7600 msgr2=0x7fbd4415a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440c7600 0x7fbd4415a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 >> v1:172.21.15.193:6789/0 conn(0x7fbd44150840 legacy=0x7fbd440c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4406d270 con 0x7fbd440c1700 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7fbd3c001a00 con 0x7fbd440c1700 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 >> v1:172.21.15.7:6789/0 conn(0x7fbd440c1700 legacy=0x7fbd4406c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd300039b0 0x7fbd30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440c7600 0x7fbd300047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd30004d10 0x7fbd30005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4406cf90 con 0x7fbd30004d10 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3000ba40 con 0x7fbd300039b0 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3000ddc0 con 0x7fbd440c7600 2023-11-28T18:49:37.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4a0ba640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd30004d10 0x7fbd30005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:37.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd490b8640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440c7600 0x7fbd300047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:37.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd498b9640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd300039b0 0x7fbd30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:37.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4a0ba640 1 -- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440c7600 msgr2=0x7fbd300047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:37.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4a0ba640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440c7600 0x7fbd300047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4a0ba640 1 -- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd300039b0 msgr2=0x7fbd30003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:37.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4a0ba640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd300039b0 0x7fbd30003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.167+0000 7fbd4a0ba640 1 -- 172.21.15.7:0/389854263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd30003530 con 0x7fbd30004d10 2023-11-28T18:49:37.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd4a0ba640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd30004d10 0x7fbd30005190 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00a810 tx=0x7fbd3c01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f2f122559f140f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:37.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c0ad630 con 0x7fbd30004d10 2023-11-28T18:49:37.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3c0a8070 con 0x7fbd30004d10 2023-11-28T18:49:37.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd488b7640 1 -- 172.21.15.7:0/389854263 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c0ad9d0 con 0x7fbd30004d10 2023-11-28T18:49:37.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/389854263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd30004d10 msgr2=0x7fbd30005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:37.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd4bb1d640 1 --2- 172.21.15.7:0/389854263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd30004d10 0x7fbd30005190 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00a810 tx=0x7fbd3c01d300 comp rx=0 tx=0).stop 2023-11-28T18:49:37.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd4a0ba640 1 -- 172.21.15.7:0/389854263 reap_dead start 2023-11-28T18:49:37.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/389854263 shutdown_connections 2023-11-28T18:49:37.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/389854263 >> 172.21.15.7:0/389854263 conn(0x7fbd440cbf80 msgr2=0x7fbd440cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:37.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/389854263 shutdown_connections 2023-11-28T18:49:37.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.171+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/389854263 wait complete. 2023-11-28T18:49:37.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 Processor -- start 2023-11-28T18:49:37.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 -- start start 2023-11-28T18:49:37.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd441e1290 0x7fbd441e3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:37.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440ca840 0x7fbd441e4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:37.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd441e1670 0x7fbd4414d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd441dfec0 con 0x7fbd441e1290 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd441efca0 con 0x7fbd441e1670 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd441f01b0 con 0x7fbd440ca840 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd441e1290 0x7fbd441e3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd441e1290 0x7fbd441e3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45488/0 (socket says 172.21.15.7:45488) 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 -- 172.21.15.7:0/1436972339 learned_addr learned my addr 172.21.15.7:0/1436972339 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd490b8640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440ca840 0x7fbd441e4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4a0ba640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd441e1670 0x7fbd4414d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 -- 172.21.15.7:0/1436972339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440ca840 msgr2=0x7fbd441e4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440ca840 0x7fbd441e4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 -- 172.21.15.7:0/1436972339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd441e1670 msgr2=0x7fbd4414d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:37.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd441e1670 0x7fbd4414d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 -- 172.21.15.7:0/1436972339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd441e3870 con 0x7fbd441e1290 2023-11-28T18:49:37.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd498b9640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd441e1290 0x7fbd441e3d50 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000f740 tx=0x7fbd4000fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:37.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd3a7fc640 1 -- 172.21.15.7:0/1436972339 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd400050d0 con 0x7fbd441e1290 2023-11-28T18:49:37.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd3a7fc640 1 -- 172.21.15.7:0/1436972339 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd40005250 con 0x7fbd441e1290 2023-11-28T18:49:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd3a7fc640 1 -- 172.21.15.7:0/1436972339 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4000d730 con 0x7fbd441e1290 2023-11-28T18:49:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd441efed0 con 0x7fbd441e1290 2023-11-28T18:49:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.175+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd441df6b0 con 0x7fbd441e1290 2023-11-28T18:49:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.179+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4406c430 con 0x7fbd441e1290 2023-11-28T18:49:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.179+0000 7fbd3a7fc640 1 -- 172.21.15.7:0/1436972339 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbd4000d890 con 0x7fbd441e1290 2023-11-28T18:49:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.179+0000 7fbd3a7fc640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbd280668d0 0x7fbd28068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.179+0000 7fbd3a7fc640 1 -- 172.21.15.7:0/1436972339 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fbd40084990 con 0x7fbd441e1290 2023-11-28T18:49:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.179+0000 7fbd490b8640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbd280668d0 0x7fbd28068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.179+0000 7fbd490b8640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbd280668d0 0x7fbd28068d90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbd440ca7d0 tx=0x7fbd34009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:37.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.183+0000 7fbd3a7fc640 1 -- 172.21.15.7:0/1436972339 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbd400516b0 con 0x7fbd441e1290 2023-11-28T18:49:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.371+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd4404e890 con 0x7fbd441e1290 2023-11-28T18:49:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.371+0000 7fbd3a7fc640 1 -- 172.21.15.7:0/1436972339 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fbd40054cd0 con 0x7fbd441e1290 2023-11-28T18:49:37.372 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:37.372 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbd280668d0 msgr2=0x7fbd28068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbd280668d0 0x7fbd28068d90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbd440ca7d0 tx=0x7fbd34009190 comp rx=0 tx=0).stop 2023-11-28T18:49:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd441e1290 msgr2=0x7fbd441e3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd441e1290 0x7fbd441e3d50 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000f740 tx=0x7fbd4000fb40 comp rx=0 tx=0).stop 2023-11-28T18:49:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 shutdown_connections 2023-11-28T18:49:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fbd280668d0 0x7fbd28068d90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd441e1670 0x7fbd4414d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fbd440ca840 0x7fbd441e4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 --2- 172.21.15.7:0/1436972339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd441e1290 0x7fbd441e3d50 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 >> 172.21.15.7:0/1436972339 conn(0x7fbd440cbf80 msgr2=0x7fbd4414fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 shutdown_connections 2023-11-28T18:49:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:37.375+0000 7fbd4bb1d640 1 -- 172.21.15.7:0/1436972339 wait complete. 2023-11-28T18:49:37.394 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:39.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:39.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1ab6780640 1 Processor -- start 2023-11-28T18:49:39.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1ab6780640 1 -- start start 2023-11-28T18:49:39.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1ab6780640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f1ab00c2c40 0x7f1ab00c3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1ab6780640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab0159e90 0x7f1ab015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1ab6780640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab015e0f0 con 0x7f1ab0154000 2023-11-28T18:49:39.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1ab6780640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab015d520 con 0x7f1ab00c2c40 2023-11-28T18:49:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1ab6780640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab00c35d0 con 0x7f1ab0159e90 2023-11-28T18:49:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1ab0154000 0x7f1ab00c6660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38012/0 (socket says 172.21.15.7:38012) 2023-11-28T18:49:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaffff640 1 -- 172.21.15.7:0/2022295060 learned_addr learned my addr 172.21.15.7:0/2022295060 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaf7fe640 1 --2- 172.21.15.7:0/2022295060 >> v2:172.21.15.193:3300/0 conn(0x7f1ab00c2c40 0x7f1ab00c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1ab4d1d640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab0159e90 0x7f1ab015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377498720 0 0) 0x7f1ab015e0f0 con 0x7f1ab0154000 2023-11-28T18:49:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a98003530 con 0x7f1ab0154000 2023-11-28T18:49:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3164876683 0 0) 0x7f1a98003530 con 0x7f1ab0154000 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab0159e90 msgr2=0x7f1ab015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab0159e90 0x7f1ab015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 >> v2:172.21.15.193:3300/0 conn(0x7f1ab00c2c40 msgr2=0x7f1ab00c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 --2- 172.21.15.7:0/2022295060 >> v2:172.21.15.193:3300/0 conn(0x7f1ab00c2c40 0x7f1ab00c3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab00c7c70 con 0x7f1ab0154000 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f1aa0001680 con 0x7f1ab0154000 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaf7fe640 1 --2- 172.21.15.7:0/2022295060 >> v2:172.21.15.193:3300/0 conn(0x7f1ab00c2c40 0x7f1ab00c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 >> v1:172.21.15.7:6789/0 conn(0x7f1ab0154000 legacy=0x7f1ab00c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a98003980 0x7f1a98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980047d0 0x7f1a98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab0159e90 0x7f1a98005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab015d520 con 0x7f1a980047d0 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab00c35d0 con 0x7f1a98003980 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.550+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9800b9c0 con 0x7f1ab0159e90 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaffff640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a98003980 0x7f1a98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaf7fe640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980047d0 0x7f1a98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1ab4d1d640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab0159e90 0x7f1a98005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaffff640 1 -- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab0159e90 msgr2=0x7f1a98005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaffff640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab0159e90 0x7f1a98005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaffff640 1 -- 172.21.15.7:0/2022295060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980047d0 msgr2=0x7f1a98004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaffff640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980047d0 0x7f1a98004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaffff640 1 -- 172.21.15.7:0/2022295060 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a98003530 con 0x7f1a98003980 2023-11-28T18:49:39.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaf7fe640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980047d0 0x7f1a98004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:49:39.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaffff640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a98003980 0x7f1a98003da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000c190 tx=0x7f1aa001cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3674a4f56e22cfa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:39.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.554+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa00058e0 con 0x7f1a98003980 2023-11-28T18:49:39.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa0005a60 con 0x7f1a98003980 2023-11-28T18:49:39.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1aaeffd640 1 -- 172.21.15.7:0/2022295060 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0005e10 con 0x7f1a98003980 2023-11-28T18:49:39.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a98003980 msgr2=0x7f1a98003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:39.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 --2- 172.21.15.7:0/2022295060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a98003980 0x7f1a98003da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000c190 tx=0x7f1aa001cc20 comp rx=0 tx=0).stop 2023-11-28T18:49:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab4d1d640 1 -- 172.21.15.7:0/2022295060 reap_dead start 2023-11-28T18:49:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- 172.21.15.7:0/2022295060 shutdown_connections 2023-11-28T18:49:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- 172.21.15.7:0/2022295060 >> 172.21.15.7:0/2022295060 conn(0x7f1ab00662e0 msgr2=0x7f1ab00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- 172.21.15.7:0/2022295060 shutdown_connections 2023-11-28T18:49:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- 172.21.15.7:0/2022295060 wait complete. 2023-11-28T18:49:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 Processor -- start 2023-11-28T18:49:39.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- start start 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a98003980 0x7f1ab014d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ab0070310 0x7f1ab0152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab01525a0 0x7f1ab0070b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab014afd0 con 0x7f1a98003980 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab0073130 con 0x7f1ab0070310 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1ab6780640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab0150550 con 0x7f1ab01525a0 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.558+0000 7f1aaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a98003980 0x7f1ab014d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a98003980 0x7f1ab014d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45502/0 (socket says 172.21.15.7:45502) 2023-11-28T18:49:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaffff640 1 -- 172.21.15.7:0/3975190058 learned_addr learned my addr 172.21.15.7:0/3975190058 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1ab4d1d640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab01525a0 0x7f1ab0070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaf7fe640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ab0070310 0x7f1ab0152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaffff640 1 -- 172.21.15.7:0/3975190058 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab01525a0 msgr2=0x7f1ab0070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaffff640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab01525a0 0x7f1ab0070b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaffff640 1 -- 172.21.15.7:0/3975190058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ab0070310 msgr2=0x7f1ab0152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaffff640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ab0070310 0x7f1ab0152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaffff640 1 -- 172.21.15.7:0/3975190058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab014ecc0 con 0x7f1a98003980 2023-11-28T18:49:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aaffff640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a98003980 0x7f1ab014d950 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0007040 tx=0x7f1a98005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aacff9640 1 -- 172.21.15.7:0/3975190058 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0004c00 con 0x7f1a98003980 2023-11-28T18:49:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aacff9640 1 -- 172.21.15.7:0/3975190058 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa0004d60 con 0x7f1a98003980 2023-11-28T18:49:39.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab014efa0 con 0x7f1a98003980 2023-11-28T18:49:39.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aacff9640 1 -- 172.21.15.7:0/3975190058 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa00072f0 con 0x7f1a98003980 2023-11-28T18:49:39.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab014f560 con 0x7f1a98003980 2023-11-28T18:49:39.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab004e890 con 0x7f1a98003980 2023-11-28T18:49:39.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.562+0000 7f1aacff9640 1 -- 172.21.15.7:0/3975190058 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1aa0007450 con 0x7f1a98003980 2023-11-28T18:49:39.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.566+0000 7f1aacff9640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1a9c0668d0 0x7f1a9c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:39.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.566+0000 7f1aacff9640 1 -- 172.21.15.7:0/3975190058 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f1aa011e260 con 0x7f1a98003980 2023-11-28T18:49:39.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.566+0000 7f1aaf7fe640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1a9c0668d0 0x7f1a9c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:39.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.566+0000 7f1aaf7fe640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1a9c0668d0 0x7f1a9c068d90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00c4130 tx=0x7f1aa4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:39.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.570+0000 7f1aacff9640 1 -- 172.21.15.7:0/3975190058 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1aa00eaef0 con 0x7f1a98003980 2023-11-28T18:49:39.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ab00c71c0 con 0x7f1a98003980 2023-11-28T18:49:39.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1aacff9640 1 -- 172.21.15.7:0/3975190058 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f1aa00ee510 con 0x7f1a98003980 2023-11-28T18:49:39.763 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:39.763 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:39.763 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:39.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1a9c0668d0 msgr2=0x7f1a9c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:39.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1a9c0668d0 0x7f1a9c068d90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00c4130 tx=0x7f1aa4009190 comp rx=0 tx=0).stop 2023-11-28T18:49:39.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a98003980 msgr2=0x7f1ab014d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a98003980 0x7f1ab014d950 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0007040 tx=0x7f1a98005800 comp rx=0 tx=0).stop 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 shutdown_connections 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1a9c0668d0 0x7f1a9c068d90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1ab01525a0 0x7f1ab0070b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ab0070310 0x7f1ab0152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 --2- 172.21.15.7:0/3975190058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a98003980 0x7f1ab014d950 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 >> 172.21.15.7:0/3975190058 conn(0x7f1ab00662e0 msgr2=0x7f1ab0157c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 shutdown_connections 2023-11-28T18:49:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:39.762+0000 7f1ab6780640 1 -- 172.21.15.7:0/3975190058 wait complete. 2023-11-28T18:49:39.784 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:41.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:41.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~262s 2023-11-28T18:49:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.938+0000 7f7beeff8640 1 Processor -- start 2023-11-28T18:49:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7beeff8640 1 -- start start 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7beeff8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be8151cd0 0x7f7be815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7beeff8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7be8157b90 0x7f7be80701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7beeff8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be8158000 con 0x7f7be806e560 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7beeff8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be8070730 con 0x7f7be8157b90 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7beeff8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be8071e60 con 0x7f7be8151cd0 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7bed595640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7be8157b90 0x7f7be80701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7be7fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7be806e560 0x7f7be80c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38014/0 (socket says 172.21.15.7:38014) 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7be7fff640 1 -- 172.21.15.7:0/1703200269 learned_addr learned my addr 172.21.15.7:0/1703200269 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7becd94640 1 --2- 172.21.15.7:0/1703200269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be8151cd0 0x7f7be815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7bed595640 1 -- 172.21.15.7:0/1703200269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be8151cd0 msgr2=0x7f7be815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7bed595640 1 --2- 172.21.15.7:0/1703200269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be8151cd0 0x7f7be815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7bed595640 1 -- 172.21.15.7:0/1703200269 >> v1:172.21.15.7:6789/0 conn(0x7f7be806e560 legacy=0x7f7be80c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.942+0000 7f7bed595640 1 -- 172.21.15.7:0/1703200269 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be8070930 con 0x7f7be8157b90 2023-11-28T18:49:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7bed595640 1 --2- 172.21.15.7:0/1703200269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be80701f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009d90 tx=0x7f7be001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617494125379af69 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7be77fe640 1 -- 172.21.15.7:0/1703200269 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be00a5870 con 0x7f7be8157b90 2023-11-28T18:49:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7be77fe640 1 -- 172.21.15.7:0/1703200269 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be00a59d0 con 0x7f7be8157b90 2023-11-28T18:49:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7be77fe640 1 -- 172.21.15.7:0/1703200269 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be00a5d60 con 0x7f7be8157b90 2023-11-28T18:49:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 -- 172.21.15.7:0/1703200269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 msgr2=0x7f7be80701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1703200269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be80701f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009d90 tx=0x7f7be001c8c0 comp rx=0 tx=0).stop 2023-11-28T18:49:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 -- 172.21.15.7:0/1703200269 shutdown_connections 2023-11-28T18:49:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1703200269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be80701f0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1703200269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be8151cd0 0x7f7be815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 -- 172.21.15.7:0/1703200269 >> 172.21.15.7:0/1703200269 conn(0x7f7be80662e0 msgr2=0x7f7be80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 -- 172.21.15.7:0/1703200269 shutdown_connections 2023-11-28T18:49:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 -- 172.21.15.7:0/1703200269 wait complete. 2023-11-28T18:49:41.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.946+0000 7f7beeff8640 1 Processor -- start 2023-11-28T18:49:41.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 -- start start 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be806e560 0x7f7be815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7be8151cd0 0x7f7be8146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be81f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80c70c0 con 0x7f7be8157b90 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be81f6890 con 0x7f7be8151cd0 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be815a160 con 0x7f7be806e560 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be81f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be81f2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45514/0 (socket says 172.21.15.7:45514) 2023-11-28T18:49:41.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 -- 172.21.15.7:0/1618335733 learned_addr learned my addr 172.21.15.7:0/1618335733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7becd94640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be806e560 0x7f7be815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7be7fff640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7be8151cd0 0x7f7be8146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 -- 172.21.15.7:0/1618335733 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be806e560 msgr2=0x7f7be815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be806e560 0x7f7be815a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 -- 172.21.15.7:0/1618335733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7be8151cd0 msgr2=0x7f7be8146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7be8151cd0 0x7f7be8146f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 -- 172.21.15.7:0/1618335733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be0009a70 con 0x7f7be8157b90 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7bed595640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be81f2770 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009d90 tx=0x7f7be0019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7be57fa640 1 -- 172.21.15.7:0/1618335733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be00b9860 con 0x7f7be8157b90 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7be57fa640 1 -- 172.21.15.7:0/1618335733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be00b99c0 con 0x7f7be8157b90 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7be57fa640 1 -- 172.21.15.7:0/1618335733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be00b9d70 con 0x7f7be8157b90 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be81f5c80 con 0x7f7be8157b90 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be81f6240 con 0x7f7be8157b90 2023-11-28T18:49:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.950+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be804e890 con 0x7f7be8157b90 2023-11-28T18:49:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.954+0000 7f7be57fa640 1 -- 172.21.15.7:0/1618335733 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7be00ad440 con 0x7f7be8157b90 2023-11-28T18:49:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.954+0000 7f7be57fa640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bcc066900 0x7f7bcc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.954+0000 7f7be57fa640 1 -- 172.21.15.7:0/1618335733 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f7be011f670 con 0x7f7be8157b90 2023-11-28T18:49:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.954+0000 7f7becd94640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bcc066900 0x7f7bcc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.954+0000 7f7becd94640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bcc066900 0x7f7bcc068dc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc007a60 tx=0x7f7bdc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:41.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:41.958+0000 7f7be57fa640 1 -- 172.21.15.7:0/1618335733 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7be00ec3b0 con 0x7f7be8157b90 2023-11-28T18:49:42.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:49:42.040 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:49:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7be81555b0 con 0x7f7be8157b90 2023-11-28T18:49:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7be57fa640 1 -- 172.21.15.7:0/1618335733 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f7be00ef9d0 con 0x7f7be8157b90 2023-11-28T18:49:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:42.151 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:42.151 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bcc066900 msgr2=0x7f7bcc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bcc066900 0x7f7bcc068dc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc007a60 tx=0x7f7bdc008040 comp rx=0 tx=0).stop 2023-11-28T18:49:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 msgr2=0x7f7be81f2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be81f2770 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009d90 tx=0x7f7be0019660 comp rx=0 tx=0).stop 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 shutdown_connections 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f7bcc066900 0x7f7bcc068dc0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7be8157b90 0x7f7be81f2770 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7be8151cd0 0x7f7be8146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 --2- 172.21.15.7:0/1618335733 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7be806e560 0x7f7be815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 >> 172.21.15.7:0/1618335733 conn(0x7f7be80662e0 msgr2=0x7f7be8157200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 shutdown_connections 2023-11-28T18:49:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:42.150+0000 7f7beeff8640 1 -- 172.21.15.7:0/1618335733 wait complete. 2023-11-28T18:49:42.172 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:44.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f98a5a9a640 1 Processor -- start 2023-11-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f98a5a9a640 1 -- start start 2023-11-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f98a5a9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a0072450 0x7f98a00728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f98a5a9a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00ca920 con 0x7f98a01542e0 2023-11-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f98a5a9a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00c4cd0 con 0x7f98a00cbb70 2023-11-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f98a5a9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0071d50 con 0x7f98a0072450 2023-11-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f98a01542e0 0x7f98a015e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51822/0 (socket says 172.21.15.7:51822) 2023-11-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989f7fe640 1 -- 172.21.15.7:0/513340483 learned_addr learned my addr 172.21.15.7:0/513340483 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989ffff640 1 --2- 172.21.15.7:0/513340483 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a0072450 0x7f98a00728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989e7fc640 1 -- 172.21.15.7:0/513340483 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663496758 0 0) 0x7f98a00ca920 con 0x7f98a01542e0 2023-11-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989e7fc640 1 -- 172.21.15.7:0/513340483 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9888003530 con 0x7f98a01542e0 2023-11-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989e7fc640 1 -- 172.21.15.7:0/513340483 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715520109 0 0) 0x7f98a00c4cd0 con 0x7f98a00cbb70 2023-11-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989ffff640 1 -- 172.21.15.7:0/513340483 >> v1:172.21.15.193:6789/0 conn(0x7f98a00cbb70 legacy=0x7f98a00cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989ffff640 1 -- 172.21.15.7:0/513340483 >> v1:172.21.15.7:6789/0 conn(0x7f98a01542e0 legacy=0x7f98a015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.330+0000 7f989ffff640 1 -- 172.21.15.7:0/513340483 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00cab70 con 0x7f98a0072450 2023-11-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f989ffff640 1 --2- 172.21.15.7:0/513340483 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a0072450 0x7f98a00728c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9898009dc0 tx=0x7f989801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e32c5281591cc5f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f989e7fc640 1 -- 172.21.15.7:0/513340483 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98980a5940 con 0x7f98a0072450 2023-11-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f989e7fc640 1 -- 172.21.15.7:0/513340483 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98980a5aa0 con 0x7f98a0072450 2023-11-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f989e7fc640 1 -- 172.21.15.7:0/513340483 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98980a5e30 con 0x7f98a0072450 2023-11-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f98a5a9a640 1 -- 172.21.15.7:0/513340483 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a0072450 msgr2=0x7f98a00728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f98a5a9a640 1 --2- 172.21.15.7:0/513340483 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a0072450 0x7f98a00728c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9898009dc0 tx=0x7f989801c970 comp rx=0 tx=0).stop 2023-11-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f98a5a9a640 1 -- 172.21.15.7:0/513340483 shutdown_connections 2023-11-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f98a5a9a640 1 --2- 172.21.15.7:0/513340483 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a0072450 0x7f98a00728c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f98a5a9a640 1 -- 172.21.15.7:0/513340483 >> 172.21.15.7:0/513340483 conn(0x7f98a00717e0 msgr2=0x7f98a00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f98a5a9a640 1 -- 172.21.15.7:0/513340483 shutdown_connections 2023-11-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.334+0000 7f98a5a9a640 1 -- 172.21.15.7:0/513340483 wait complete. 2023-11-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 Processor -- start 2023-11-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 -- start start 2023-11-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a0072450 0x7f98a00ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a00cbb70 0x7f98a00c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98a01542e0 0x7f98a0143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0072030 con 0x7f98a0072450 2023-11-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c8a60 con 0x7f98a01542e0 2023-11-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0150af0 con 0x7f98a00cbb70 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a0072450 0x7f98a00ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a0072450 0x7f98a00ce6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50100/0 (socket says 172.21.15.7:50100) 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 -- 172.21.15.7:0/542284889 learned_addr learned my addr 172.21.15.7:0/542284889 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989effd640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a00cbb70 0x7f98a00c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989ffff640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98a01542e0 0x7f98a0143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 -- 172.21.15.7:0/542284889 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a00cbb70 msgr2=0x7f98a00c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a00cbb70 0x7f98a00c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 -- 172.21.15.7:0/542284889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98a01542e0 msgr2=0x7f98a0143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98a01542e0 0x7f98a0143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 -- 172.21.15.7:0/542284889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9898009a70 con 0x7f98a0072450 2023-11-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f989f7fe640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a0072450 0x7f98a00ce6f0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f989000c3f0 tx=0x7f989000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f9887fff640 1 -- 172.21.15.7:0/542284889 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890005cf0 con 0x7f98a0072450 2023-11-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f9887fff640 1 -- 172.21.15.7:0/542284889 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9890005e50 con 0x7f98a0072450 2023-11-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a0150db0 con 0x7f98a0072450 2023-11-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f9887fff640 1 -- 172.21.15.7:0/542284889 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98900072f0 con 0x7f98a0072450 2023-11-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.338+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a0151370 con 0x7f98a0072450 2023-11-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.342+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a006c670 con 0x7f98a0072450 2023-11-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.342+0000 7f9887fff640 1 -- 172.21.15.7:0/542284889 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9890007450 con 0x7f98a0072450 2023-11-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.342+0000 7f9887fff640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f988c0668d0 0x7f988c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.342+0000 7f9887fff640 1 -- 172.21.15.7:0/542284889 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f9890083f00 con 0x7f98a0072450 2023-11-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.342+0000 7f989effd640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f988c0668d0 0x7f988c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.342+0000 7f989effd640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f988c0668d0 0x7f988c068d90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f98a00b4110 tx=0x7f9894009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.346+0000 7f9887fff640 1 -- 172.21.15.7:0/542284889 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9890050c40 con 0x7f98a0072450 2023-11-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.534+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98a004e890 con 0x7f98a0072450 2023-11-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.534+0000 7f9887fff640 1 -- 172.21.15.7:0/542284889 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f9890017070 con 0x7f98a0072450 2023-11-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f988c0668d0 msgr2=0x7f988c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f988c0668d0 0x7f988c068d90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f98a00b4110 tx=0x7f9894009040 comp rx=0 tx=0).stop 2023-11-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a0072450 msgr2=0x7f98a00ce6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a0072450 0x7f98a00ce6f0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f989000c3f0 tx=0x7f989000c8b0 comp rx=0 tx=0).stop 2023-11-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 shutdown_connections 2023-11-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f988c0668d0 0x7f988c068d90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98a01542e0 0x7f98a0143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f98a00cbb70 0x7f98a00c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 --2- 172.21.15.7:0/542284889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a0072450 0x7f98a00ce6f0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 >> 172.21.15.7:0/542284889 conn(0x7f98a00717e0 msgr2=0x7f98a01518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 shutdown_connections 2023-11-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:44.538+0000 7f98a5a9a640 1 -- 172.21.15.7:0/542284889 wait complete. 2023-11-28T18:49:44.559 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:46.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:46.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.714+0000 7fe6d5980640 1 Processor -- start 2023-11-28T18:49:46.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.714+0000 7fe6d5980640 1 -- start start 2023-11-28T18:49:46.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6d5980640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6d0150b30 0x7fe6d00c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6d5980640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 0x7fe6d00c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6d5980640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fe6d00c4020 0x7fe6d00c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6d5980640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d00c72b0 con 0x7fe6d0150b30 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6d5980640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d0070c00 con 0x7fe6d00c4020 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6d5980640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00c6140 con 0x7fe6d01575c0 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ceffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6d0150b30 0x7fe6d00c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ceffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6d0150b30 0x7fe6d00c24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50104/0 (socket says 172.21.15.7:50104) 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ceffd640 1 -- 172.21.15.7:0/1552919706 learned_addr learned my addr 172.21.15.7:0/1552919706 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6cf7fe640 1 --2- 172.21.15.7:0/1552919706 >> v2:172.21.15.193:3300/0 conn(0x7fe6d00c4020 0x7fe6d00c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:46.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ce7fc640 1 --2- 172.21.15.7:0/1552919706 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 0x7fe6d00c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:46.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ceffd640 1 -- 172.21.15.7:0/1552919706 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 msgr2=0x7fe6d00c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:46.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ceffd640 1 --2- 172.21.15.7:0/1552919706 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 0x7fe6d00c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ceffd640 1 -- 172.21.15.7:0/1552919706 >> v2:172.21.15.193:3300/0 conn(0x7fe6d00c4020 msgr2=0x7fe6d00c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:46.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ceffd640 1 --2- 172.21.15.7:0/1552919706 >> v2:172.21.15.193:3300/0 conn(0x7fe6d00c4020 0x7fe6d00c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.718+0000 7fe6ceffd640 1 -- 172.21.15.7:0/1552919706 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d00c6370 con 0x7fe6d0150b30 2023-11-28T18:49:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6ceffd640 1 --2- 172.21.15.7:0/1552919706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d0150b30 0x7fe6d00c24f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00c95f0 tx=0x7fe6c001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6ecb326a01f2ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6cdffb640 1 -- 172.21.15.7:0/1552919706 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00a89b0 con 0x7fe6d0150b30 2023-11-28T18:49:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6cdffb640 1 -- 172.21.15.7:0/1552919706 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c00a8b10 con 0x7fe6d0150b30 2023-11-28T18:49:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6cdffb640 1 -- 172.21.15.7:0/1552919706 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00a8ea0 con 0x7fe6d0150b30 2023-11-28T18:49:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 -- 172.21.15.7:0/1552919706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d0150b30 msgr2=0x7fe6d00c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 --2- 172.21.15.7:0/1552919706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d0150b30 0x7fe6d00c24f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00c95f0 tx=0x7fe6c001ecf0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 -- 172.21.15.7:0/1552919706 shutdown_connections 2023-11-28T18:49:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 --2- 172.21.15.7:0/1552919706 >> v2:172.21.15.193:3300/0 conn(0x7fe6d00c4020 0x7fe6d00c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 --2- 172.21.15.7:0/1552919706 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 0x7fe6d00c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 --2- 172.21.15.7:0/1552919706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d0150b30 0x7fe6d00c24f0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 -- 172.21.15.7:0/1552919706 >> 172.21.15.7:0/1552919706 conn(0x7fe6d00662e0 msgr2=0x7fe6d00666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 -- 172.21.15.7:0/1552919706 shutdown_connections 2023-11-28T18:49:46.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 -- 172.21.15.7:0/1552919706 wait complete. 2023-11-28T18:49:46.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 Processor -- start 2023-11-28T18:49:46.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.722+0000 7fe6d5980640 1 -- start start 2023-11-28T18:49:46.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d00c4020 0x7fe6d014eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:46.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0150b30 0x7fe6d0144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:46.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 0x7fe6d0145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01ee6b0 con 0x7fe6d00c4020 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01ed7c0 con 0x7fe6d0150b30 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01edaa0 con 0x7fe6d01575c0 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d00c4020 0x7fe6d014eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d00c4020 0x7fe6d014eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50112/0 (socket says 172.21.15.7:50112) 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 -- 172.21.15.7:0/2612782718 learned_addr learned my addr 172.21.15.7:0/2612782718 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6cf7fe640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 0x7fe6d0145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ce7fc640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0150b30 0x7fe6d0144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 -- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 msgr2=0x7fe6d0145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 0x7fe6d0145080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 -- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0150b30 msgr2=0x7fe6d0144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:46.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0150b30 0x7fe6d0144b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 -- 172.21.15.7:0/2612782718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00094d0 con 0x7fe6d00c4020 2023-11-28T18:49:46.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ce7fc640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0150b30 0x7fe6d0144b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:49:46.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6ceffd640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d00c4020 0x7fe6d014eaa0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00b3040 tx=0x7fe6c001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:46.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6b7fff640 1 -- 172.21.15.7:0/2612782718 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00b0ca0 con 0x7fe6d00c4020 2023-11-28T18:49:46.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6b7fff640 1 -- 172.21.15.7:0/2612782718 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c00b0e20 con 0x7fe6d00c4020 2023-11-28T18:49:46.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6b7fff640 1 -- 172.21.15.7:0/2612782718 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00bb6b0 con 0x7fe6d00c4020 2023-11-28T18:49:46.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d01edd80 con 0x7fe6d00c4020 2023-11-28T18:49:46.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d01ee340 con 0x7fe6d00c4020 2023-11-28T18:49:46.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d004e890 con 0x7fe6d00c4020 2023-11-28T18:49:46.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.726+0000 7fe6b7fff640 1 -- 172.21.15.7:0/2612782718 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe6c00b7070 con 0x7fe6d00c4020 2023-11-28T18:49:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.730+0000 7fe6b7fff640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe6bc066900 0x7fe6bc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.730+0000 7fe6b7fff640 1 -- 172.21.15.7:0/2612782718 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fe6c0122bb0 con 0x7fe6d00c4020 2023-11-28T18:49:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.730+0000 7fe6ce7fc640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe6bc066900 0x7fe6bc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.730+0000 7fe6ce7fc640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe6bc066900 0x7fe6bc068dc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe6d01611c0 tx=0x7fe6c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.734+0000 7fe6b7fff640 1 -- 172.21.15.7:0/2612782718 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe6c00ef8d0 con 0x7fe6d00c4020 2023-11-28T18:49:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.926+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6d00c2d80 con 0x7fe6d00c4020 2023-11-28T18:49:46.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6b7fff640 1 -- 172.21.15.7:0/2612782718 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fe6c00b5090 con 0x7fe6d00c4020 2023-11-28T18:49:46.929 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:46.929 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe6bc066900 msgr2=0x7fe6bc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe6bc066900 0x7fe6bc068dc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe6d01611c0 tx=0x7fe6c4008040 comp rx=0 tx=0).stop 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d00c4020 msgr2=0x7fe6d014eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d00c4020 0x7fe6d014eaa0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00b3040 tx=0x7fe6c001ba90 comp rx=0 tx=0).stop 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 shutdown_connections 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fe6bc066900 0x7fe6bc068dc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fe6d01575c0 0x7fe6d0145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0150b30 0x7fe6d0144b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 --2- 172.21.15.7:0/2612782718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6d00c4020 0x7fe6d014eaa0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 >> 172.21.15.7:0/2612782718 conn(0x7fe6d00662e0 msgr2=0x7fe6d00ca920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 shutdown_connections 2023-11-28T18:49:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:46.930+0000 7fe6d5980640 1 -- 172.21.15.7:0/2612782718 wait complete. 2023-11-28T18:49:46.952 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:49:48.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~269s 2023-11-28T18:49:48.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:49.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.106+0000 7fca4e09c640 1 Processor -- start 2023-11-28T18:49:49.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.106+0000 7fca4e09c640 1 -- start start 2023-11-28T18:49:49.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.106+0000 7fca4e09c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca480c24f0 0x7fca481575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:49.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.106+0000 7fca4e09c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca480c6140 con 0x7fca48150b30 2023-11-28T18:49:49.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.106+0000 7fca4e09c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca48157b00 con 0x7fca480c7980 2023-11-28T18:49:49.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.106+0000 7fca4e09c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca480c72b0 con 0x7fca480c24f0 2023-11-28T18:49:49.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca477fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fca48150b30 0x7fca480c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51832/0 (socket says 172.21.15.7:51832) 2023-11-28T18:49:49.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca477fe640 1 -- 172.21.15.7:0/1979540316 learned_addr learned my addr 172.21.15.7:0/1979540316 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:49.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca46ffd640 1 --2- 172.21.15.7:0/1979540316 >> v2:172.21.15.7:3300/0 conn(0x7fca480c24f0 0x7fca481575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:49.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca467fc640 1 -- 172.21.15.7:0/1979540316 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555848176 0 0) 0x7fca480c6140 con 0x7fca48150b30 2023-11-28T18:49:49.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca467fc640 1 -- 172.21.15.7:0/1979540316 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca30003530 con 0x7fca48150b30 2023-11-28T18:49:49.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca46ffd640 1 -- 172.21.15.7:0/1979540316 >> v1:172.21.15.193:6789/0 conn(0x7fca480c7980 legacy=0x7fca480c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:49.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca46ffd640 1 -- 172.21.15.7:0/1979540316 >> v1:172.21.15.7:6789/0 conn(0x7fca48150b30 legacy=0x7fca480c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:49.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca46ffd640 1 -- 172.21.15.7:0/1979540316 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca480c2960 con 0x7fca480c24f0 2023-11-28T18:49:49.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca46ffd640 1 --2- 172.21.15.7:0/1979540316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca481575c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0097c0 tx=0x7fca3c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f69a4074379f49a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:49.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca467fc640 1 -- 172.21.15.7:0/1979540316 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c0a8970 con 0x7fca480c24f0 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca467fc640 1 -- 172.21.15.7:0/1979540316 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c0a8ad0 con 0x7fca480c24f0 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.110+0000 7fca467fc640 1 -- 172.21.15.7:0/1979540316 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c0a8e60 con 0x7fca480c24f0 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- 172.21.15.7:0/1979540316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 msgr2=0x7fca481575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 --2- 172.21.15.7:0/1979540316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca481575c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0097c0 tx=0x7fca3c01ec70 comp rx=0 tx=0).stop 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- 172.21.15.7:0/1979540316 shutdown_connections 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 --2- 172.21.15.7:0/1979540316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca481575c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- 172.21.15.7:0/1979540316 >> 172.21.15.7:0/1979540316 conn(0x7fca480662e0 msgr2=0x7fca480666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- 172.21.15.7:0/1979540316 shutdown_connections 2023-11-28T18:49:49.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- 172.21.15.7:0/1979540316 wait complete. 2023-11-28T18:49:49.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 Processor -- start 2023-11-28T18:49:49.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- start start 2023-11-28T18:49:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca48145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca480c7980 0x7fca480c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca48150b30 0x7fca480c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca48145970 con 0x7fca480c24f0 2023-11-28T18:49:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca48148930 con 0x7fca48150b30 2023-11-28T18:49:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca4e09c640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca481512d0 con 0x7fca480c7980 2023-11-28T18:49:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca477fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca48145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:49.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca477fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca48145430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50136/0 (socket says 172.21.15.7:50136) 2023-11-28T18:49:49.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca477fe640 1 -- 172.21.15.7:0/3083352741 learned_addr learned my addr 172.21.15.7:0/3083352741 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:49.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca477fe640 1 -- 172.21.15.7:0/3083352741 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca480c7980 msgr2=0x7fca480c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:49.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca477fe640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca480c7980 0x7fca480c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:49.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca47fff640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca48150b30 0x7fca480c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:49.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.114+0000 7fca477fe640 1 -- 172.21.15.7:0/3083352741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca48150b30 msgr2=0x7fca480c51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:49.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca477fe640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca48150b30 0x7fca480c51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:49.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca477fe640 1 -- 172.21.15.7:0/3083352741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c009420 con 0x7fca480c24f0 2023-11-28T18:49:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca47fff640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca48150b30 0x7fca480c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:49:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca477fe640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca48145430 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fca3800c420 tx=0x7fca3800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca2ffff640 1 -- 172.21.15.7:0/3083352741 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38005cf0 con 0x7fca480c24f0 2023-11-28T18:49:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca2ffff640 1 -- 172.21.15.7:0/3083352741 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca38005e50 con 0x7fca480c24f0 2023-11-28T18:49:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca48151590 con 0x7fca480c24f0 2023-11-28T18:49:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca2ffff640 1 -- 172.21.15.7:0/3083352741 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca380072d0 con 0x7fca480c24f0 2023-11-28T18:49:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca481482e0 con 0x7fca480c24f0 2023-11-28T18:49:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4804e890 con 0x7fca480c24f0 2023-11-28T18:49:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca2ffff640 1 -- 172.21.15.7:0/3083352741 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fca38007430 con 0x7fca480c24f0 2023-11-28T18:49:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca2ffff640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fca340668d0 0x7fca34068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca2ffff640 1 -- 172.21.15.7:0/3083352741 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fca38083f00 con 0x7fca480c24f0 2023-11-28T18:49:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.118+0000 7fca46ffd640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fca340668d0 0x7fca34068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.122+0000 7fca46ffd640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fca340668d0 0x7fca34068d90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0b2000 tx=0x7fca3c004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.126+0000 7fca2ffff640 1 -- 172.21.15.7:0/3083352741 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fca38050c40 con 0x7fca480c24f0 2023-11-28T18:49:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.314+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca480c2d80 con 0x7fca480c24f0 2023-11-28T18:49:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.314+0000 7fca2ffff640 1 -- 172.21.15.7:0/3083352741 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fca38017070 con 0x7fca480c24f0 2023-11-28T18:49:49.316 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:49.316 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fca340668d0 msgr2=0x7fca34068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fca340668d0 0x7fca34068d90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0b2000 tx=0x7fca3c004740 comp rx=0 tx=0).stop 2023-11-28T18:49:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 msgr2=0x7fca48145430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca48145430 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fca3800c420 tx=0x7fca3800c8e0 comp rx=0 tx=0).stop 2023-11-28T18:49:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 shutdown_connections 2023-11-28T18:49:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fca340668d0 0x7fca34068d90 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca48150b30 0x7fca480c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca480c7980 0x7fca480c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 --2- 172.21.15.7:0/3083352741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca480c24f0 0x7fca48145430 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 >> 172.21.15.7:0/3083352741 conn(0x7fca480662e0 msgr2=0x7fca4814e890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 shutdown_connections 2023-11-28T18:49:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:49.318+0000 7fca4e09c640 1 -- 172.21.15.7:0/3083352741 wait complete. 2023-11-28T18:49:49.338 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:51.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693ab80640 1 Processor -- start 2023-11-28T18:49:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693ab80640 1 -- start start 2023-11-28T18:49:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693ab80640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6934150840 0x7f69340c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693ab80640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f693415a3a0 0x7f69340c1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693ab80640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693406ccc0 con 0x7f69340c7d60 2023-11-28T18:49:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693ab80640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693415a7c0 con 0x7f6934150840 2023-11-28T18:49:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693ab80640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693406cf90 con 0x7f693415a3a0 2023-11-28T18:49:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693911d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f69340c7d60 0x7f693406c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51834/0 (socket says 172.21.15.7:51834) 2023-11-28T18:49:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693911d640 1 -- 172.21.15.7:0/2599068274 learned_addr learned my addr 172.21.15.7:0/2599068274 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.494+0000 7f693891c640 1 --2- 172.21.15.7:0/2599068274 >> v2:172.21.15.7:3300/0 conn(0x7f6934150840 0x7f69340c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f6933fff640 1 --2- 172.21.15.7:0/2599068274 >> v2:172.21.15.193:3300/0 conn(0x7f693415a3a0 0x7f69340c1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f69337fe640 1 -- 172.21.15.7:0/2599068274 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831089000 0 0) 0x7f693406ccc0 con 0x7f69340c7d60 2023-11-28T18:49:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f69337fe640 1 -- 172.21.15.7:0/2599068274 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6924003530 con 0x7f69340c7d60 2023-11-28T18:49:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f693891c640 1 -- 172.21.15.7:0/2599068274 >> v2:172.21.15.193:3300/0 conn(0x7f693415a3a0 msgr2=0x7f69340c1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f693891c640 1 --2- 172.21.15.7:0/2599068274 >> v2:172.21.15.193:3300/0 conn(0x7f693415a3a0 0x7f69340c1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f693891c640 1 -- 172.21.15.7:0/2599068274 >> v1:172.21.15.7:6789/0 conn(0x7f69340c7d60 legacy=0x7f693406c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:51.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f693891c640 1 -- 172.21.15.7:0/2599068274 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693406d270 con 0x7f6934150840 2023-11-28T18:49:51.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f6933fff640 1 --2- 172.21.15.7:0/2599068274 >> v2:172.21.15.193:3300/0 conn(0x7f693415a3a0 0x7f69340c1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:49:51.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f693891c640 1 --2- 172.21.15.7:0/2599068274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f69340c7600 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f6934067640 tx=0x7f692801fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16760e05674d36c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f69337fe640 1 -- 172.21.15.7:0/2599068274 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280a9a60 con 0x7f6934150840 2023-11-28T18:49:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f69337fe640 1 -- 172.21.15.7:0/2599068274 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69280a9bc0 con 0x7f6934150840 2023-11-28T18:49:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f69337fe640 1 -- 172.21.15.7:0/2599068274 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280b1440 con 0x7f6934150840 2023-11-28T18:49:51.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f693ab80640 1 -- 172.21.15.7:0/2599068274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 msgr2=0x7f69340c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:51.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.498+0000 7f693ab80640 1 --2- 172.21.15.7:0/2599068274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f69340c7600 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f6934067640 tx=0x7f692801fd60 comp rx=0 tx=0).stop 2023-11-28T18:49:51.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 -- 172.21.15.7:0/2599068274 shutdown_connections 2023-11-28T18:49:51.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 --2- 172.21.15.7:0/2599068274 >> v2:172.21.15.193:3300/0 conn(0x7f693415a3a0 0x7f69340c1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 --2- 172.21.15.7:0/2599068274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f69340c7600 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 -- 172.21.15.7:0/2599068274 >> 172.21.15.7:0/2599068274 conn(0x7f69340cbf80 msgr2=0x7f69340cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:51.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 -- 172.21.15.7:0/2599068274 shutdown_connections 2023-11-28T18:49:51.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 -- 172.21.15.7:0/2599068274 wait complete. 2023-11-28T18:49:51.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 Processor -- start 2023-11-28T18:49:51.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 -- start start 2023-11-28T18:49:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69340c7d60 0x7f6934159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f693414abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f693415a3a0 0x7f69340c1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69340af120 con 0x7f6934150840 2023-11-28T18:49:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934158f50 con 0x7f693415a3a0 2023-11-28T18:49:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693ab80640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69340afcd0 con 0x7f69340c7d60 2023-11-28T18:49:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f6933fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f693414abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f6933fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f693414abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50152/0 (socket says 172.21.15.7:50152) 2023-11-28T18:49:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f6933fff640 1 -- 172.21.15.7:0/533819498 learned_addr learned my addr 172.21.15.7:0/533819498 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693891c640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69340c7d60 0x7f6934159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f693911d640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f693415a3a0 0x7f69340c1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f6933fff640 1 -- 172.21.15.7:0/533819498 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69340c7d60 msgr2=0x7f6934159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:51.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f6933fff640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69340c7d60 0x7f6934159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f6933fff640 1 -- 172.21.15.7:0/533819498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f693415a3a0 msgr2=0x7f69340c1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:51.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f6933fff640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f693415a3a0 0x7f69340c1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.502+0000 7f6933fff640 1 -- 172.21.15.7:0/533819498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6928009420 con 0x7f6934150840 2023-11-28T18:49:51.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f6933fff640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f693414abd0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f692000c7f0 tx=0x7f692000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:51.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f69317fa640 1 -- 172.21.15.7:0/533819498 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920007820 con 0x7f6934150840 2023-11-28T18:49:51.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f69317fa640 1 -- 172.21.15.7:0/533819498 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6920007980 con 0x7f6934150840 2023-11-28T18:49:51.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f69317fa640 1 -- 172.21.15.7:0/533819498 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920007d30 con 0x7f6934150840 2023-11-28T18:49:51.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69340c8d00 con 0x7f6934150840 2023-11-28T18:49:51.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6934158890 con 0x7f6934150840 2023-11-28T18:49:51.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693404e890 con 0x7f6934150840 2023-11-28T18:49:51.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f69317fa640 1 -- 172.21.15.7:0/533819498 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f69200113f0 con 0x7f6934150840 2023-11-28T18:49:51.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f69317fa640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69180668d0 0x7f6918068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:51.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f69317fa640 1 -- 172.21.15.7:0/533819498 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f6920084a60 con 0x7f6934150840 2023-11-28T18:49:51.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.506+0000 7f693891c640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69180668d0 0x7f6918068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:51.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.510+0000 7f693891c640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69180668d0 0x7f6918068d90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f69280b3000 tx=0x7f69280ab070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:51.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.510+0000 7f69317fa640 1 -- 172.21.15.7:0/533819498 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6920051830 con 0x7f6934150840 2023-11-28T18:49:51.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.702+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69340c7b40 con 0x7f6934150840 2023-11-28T18:49:51.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.702+0000 7f69317fa640 1 -- 172.21.15.7:0/533819498 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f6920054e50 con 0x7f6934150840 2023-11-28T18:49:51.704 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:51.704 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:51.704 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69180668d0 msgr2=0x7f6918068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69180668d0 0x7f6918068d90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f69280b3000 tx=0x7f69280ab070 comp rx=0 tx=0).stop 2023-11-28T18:49:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 msgr2=0x7f693414abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f693414abd0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f692000c7f0 tx=0x7f692000ccb0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 shutdown_connections 2023-11-28T18:49:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69180668d0 0x7f6918068d90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f693415a3a0 0x7f69340c1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6934150840 0x7f693414abd0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 --2- 172.21.15.7:0/533819498 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69340c7d60 0x7f6934159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 >> 172.21.15.7:0/533819498 conn(0x7f69340cbf80 msgr2=0x7f693406c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 shutdown_connections 2023-11-28T18:49:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:51.706+0000 7f693ab80640 1 -- 172.21.15.7:0/533819498 wait complete. 2023-11-28T18:49:51.725 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:53.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c14f57640 1 Processor -- start 2023-11-28T18:49:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c14f57640 1 -- start start 2023-11-28T18:49:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c14f57640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0c10150b30 0x7f0c100c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c14f57640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c100c6140 con 0x7f0c100c7980 2023-11-28T18:49:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c14f57640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c100c6370 con 0x7f0c101575c0 2023-11-28T18:49:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c14f57640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c100c72b0 con 0x7f0c10150b30 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0ed9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0c100c7980 0x7f0c100c7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42106/0 (socket says 172.21.15.7:42106) 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0ed9d640 1 -- 172.21.15.7:0/741238152 learned_addr learned my addr 172.21.15.7:0/741238152 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0e59c640 1 --2- 172.21.15.7:0/741238152 >> v2:172.21.15.193:3300/0 conn(0x7f0c10150b30 0x7f0c100c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417383665 0 0) 0x7f0c100c6140 con 0x7f0c100c7980 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bfc003530 con 0x7f0c100c7980 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864957267 0 0) 0x7f0bfc003530 con 0x7f0c100c7980 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 >> v2:172.21.15.193:3300/0 conn(0x7f0c10150b30 msgr2=0x7f0c100c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 --2- 172.21.15.7:0/741238152 >> v2:172.21.15.193:3300/0 conn(0x7f0c10150b30 0x7f0c100c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 >> v1:172.21.15.193:6789/0 conn(0x7f0c101575c0 legacy=0x7f0c100c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c100c6630 con 0x7f0c100c7980 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f0c08001a00 con 0x7f0c100c7980 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.878+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 >> v1:172.21.15.7:6789/0 conn(0x7f0c100c7980 legacy=0x7f0c100c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0039d0 0x7f0bfc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bfc0047d0 0x7f0bfc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bfc005190 0x7f0bfc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c100c72b0 con 0x7f0bfc0047d0 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc00c070 con 0x7f0bfc0039d0 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc00c230 con 0x7f0bfc005190 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c07fff640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bfc0047d0 0x7f0bfc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0e59c640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0039d0 0x7f0bfc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0ed9d640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bfc005190 0x7f0bfc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:53.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c07fff640 1 -- 172.21.15.7:0/741238152 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bfc005190 msgr2=0x7f0bfc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:53.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c07fff640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0bfc005190 0x7f0bfc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:53.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c07fff640 1 -- 172.21.15.7:0/741238152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0039d0 msgr2=0x7f0bfc003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:53.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c07fff640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0039d0 0x7f0bfc003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:53.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c07fff640 1 -- 172.21.15.7:0/741238152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc003530 con 0x7f0bfc0047d0 2023-11-28T18:49:53.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0ed9d640 1 -- 172.21.15.7:0/741238152 reap_dead start 2023-11-28T18:49:53.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c07fff640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bfc0047d0 0x7f0bfc004c50 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8001780 tx=0x7f0bf801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597836b260be8978 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:53.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80052d0 con 0x7f0bfc0047d0 2023-11-28T18:49:53.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf80a7050 con 0x7f0bfc0047d0 2023-11-28T18:49:53.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.882+0000 7f0c0dd9b640 1 -- 172.21.15.7:0/741238152 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8003600 con 0x7f0bfc0047d0 2023-11-28T18:49:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- 172.21.15.7:0/741238152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bfc0047d0 msgr2=0x7f0bfc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bfc0047d0 0x7f0bfc004c50 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8001780 tx=0x7f0bf801c400 comp rx=0 tx=0).stop 2023-11-28T18:49:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- 172.21.15.7:0/741238152 shutdown_connections 2023-11-28T18:49:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 --2- 172.21.15.7:0/741238152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bfc0047d0 0x7f0bfc004c50 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- 172.21.15.7:0/741238152 >> 172.21.15.7:0/741238152 conn(0x7f0c100662e0 msgr2=0x7f0c100666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- 172.21.15.7:0/741238152 shutdown_connections 2023-11-28T18:49:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- 172.21.15.7:0/741238152 wait complete. 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 Processor -- start 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- start start 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0047d0 0x7f0c10154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0c101549b0 0x7f0c1006dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c10152fe0 0x7f0c10153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c10152d20 con 0x7f0c10152fe0 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c101539a0 con 0x7f0bfc0047d0 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c14f57640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c101555b0 con 0x7f0c101549b0 2023-11-28T18:49:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c0ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c10152fe0 0x7f0c10153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:53.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c0e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0047d0 0x7f0c10154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:53.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c0ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c10152fe0 0x7f0c10153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51272/0 (socket says 172.21.15.7:51272) 2023-11-28T18:49:53.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c0ed9d640 1 -- 172.21.15.7:0/1807548563 learned_addr learned my addr 172.21.15.7:0/1807548563 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:53.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.886+0000 7f0c07fff640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0c101549b0 0x7f0c1006dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:53.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c0ed9d640 1 -- 172.21.15.7:0/1807548563 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0c101549b0 msgr2=0x7f0c1006dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:53.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c0ed9d640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0c101549b0 0x7f0c1006dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c0ed9d640 1 -- 172.21.15.7:0/1807548563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0047d0 msgr2=0x7f0c10154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c0ed9d640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0047d0 0x7f0c10154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c0ed9d640 1 -- 172.21.15.7:0/1807548563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c101f2e60 con 0x7f0c10152fe0 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c0ed9d640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c10152fe0 0x7f0c10153460 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0c101579a0 tx=0x7f0c0800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c06ffd640 1 -- 172.21.15.7:0/1807548563 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800ec10 con 0x7f0c10152fe0 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c06ffd640 1 -- 172.21.15.7:0/1807548563 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0800ed70 con 0x7f0c10152fe0 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c06ffd640 1 -- 172.21.15.7:0/1807548563 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800eee0 con 0x7f0c10152fe0 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c101f3140 con 0x7f0c10152fe0 2023-11-28T18:49:53.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c101f3700 con 0x7f0c10152fe0 2023-11-28T18:49:53.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c101f3c20 con 0x7f0c10152fe0 2023-11-28T18:49:53.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c06ffd640 1 -- 172.21.15.7:0/1807548563 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0c08013660 con 0x7f0c10152fe0 2023-11-28T18:49:53.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c06ffd640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0bf0066900 0x7f0bf0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:53.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.890+0000 7f0c06ffd640 1 -- 172.21.15.7:0/1807548563 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f0c08084b20 con 0x7f0c10152fe0 2023-11-28T18:49:53.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.894+0000 7f0c0e59c640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0bf0066900 0x7f0bf0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:53.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.894+0000 7f0c0e59c640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0bf0066900 0x7f0bf0068dc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0c10074f10 tx=0x7f0c00009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:53.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:53.894+0000 7f0c06ffd640 1 -- 172.21.15.7:0/1807548563 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0c101f3c20 con 0x7f0c10152fe0 2023-11-28T18:49:54.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.090+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c101f3c20 con 0x7f0c10152fe0 2023-11-28T18:49:54.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.090+0000 7f0c06ffd640 1 -- 172.21.15.7:0/1807548563 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f0c101f3c20 con 0x7f0c10152fe0 2023-11-28T18:49:54.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:54.092 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:54.093 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:54.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0bf0066900 msgr2=0x7f0bf0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0bf0066900 0x7f0bf0068dc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0c10074f10 tx=0x7f0c00009190 comp rx=0 tx=0).stop 2023-11-28T18:49:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c10152fe0 msgr2=0x7f0c10153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c10152fe0 0x7f0c10153460 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0c101579a0 tx=0x7f0c0800e530 comp rx=0 tx=0).stop 2023-11-28T18:49:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 shutdown_connections 2023-11-28T18:49:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f0bf0066900 0x7f0bf0068dc0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c10152fe0 0x7f0c10153460 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:54.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0c101549b0 0x7f0c1006dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:54.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 --2- 172.21.15.7:0/1807548563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bfc0047d0 0x7f0c10154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:54.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 >> 172.21.15.7:0/1807548563 conn(0x7f0c100662e0 msgr2=0x7f0c100ca180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:54.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 shutdown_connections 2023-11-28T18:49:54.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:54.094+0000 7f0c14f57640 1 -- 172.21.15.7:0/1807548563 wait complete. 2023-11-28T18:49:54.114 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:55.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~276s 2023-11-28T18:49:56.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:56.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68acac8640 1 Processor -- start 2023-11-28T18:49:56.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68acac8640 1 -- start start 2023-11-28T18:49:56.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68acac8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f68a81540a0 0x7f68a806c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:56.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68acac8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68a806d590 0x7f68a815d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:56.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68acac8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80c7a90 con 0x7f68a806c9b0 2023-11-28T18:49:56.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68acac8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a8154480 con 0x7f68a806d590 2023-11-28T18:49:56.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68acac8640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a806cf20 con 0x7f68a81540a0 2023-11-28T18:49:56.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68a7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68a806d590 0x7f68a815d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:56.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68a7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68a806d590 0x7f68a815d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51274/0 (socket says 172.21.15.7:51274) 2023-11-28T18:49:56.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68a7fff640 1 -- 172.21.15.7:0/4083868969 learned_addr learned my addr 172.21.15.7:0/4083868969 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:56.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68a77fe640 1 --2- 172.21.15.7:0/4083868969 >> v2:172.21.15.193:3300/0 conn(0x7f68a81540a0 0x7f68a806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:56.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68a7fff640 1 -- 172.21.15.7:0/4083868969 >> v2:172.21.15.193:3300/0 conn(0x7f68a81540a0 msgr2=0x7f68a806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:56.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68a7fff640 1 --2- 172.21.15.7:0/4083868969 >> v2:172.21.15.193:3300/0 conn(0x7f68a81540a0 0x7f68a806c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68a7fff640 1 -- 172.21.15.7:0/4083868969 >> v1:172.21.15.193:6789/0 conn(0x7f68a806c9b0 legacy=0x7f68a80ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:56.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.274+0000 7f68a7fff640 1 -- 172.21.15.7:0/4083868969 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80c6f10 con 0x7f68a806d590 2023-11-28T18:49:56.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68a7fff640 1 --2- 172.21.15.7:0/4083868969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806d590 0x7f68a815d610 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009e50 tx=0x7f68a001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2538b879c8cc2919 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:56.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68a67fc640 1 -- 172.21.15.7:0/4083868969 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a00af400 con 0x7f68a806d590 2023-11-28T18:49:56.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68a67fc640 1 -- 172.21.15.7:0/4083868969 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a00ab050 con 0x7f68a806d590 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68a67fc640 1 -- 172.21.15.7:0/4083868969 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a00af780 con 0x7f68a806d590 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 -- 172.21.15.7:0/4083868969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806d590 msgr2=0x7f68a815d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 --2- 172.21.15.7:0/4083868969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806d590 0x7f68a815d610 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009e50 tx=0x7f68a001f1c0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 -- 172.21.15.7:0/4083868969 shutdown_connections 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 --2- 172.21.15.7:0/4083868969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806d590 0x7f68a815d610 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 --2- 172.21.15.7:0/4083868969 >> v2:172.21.15.193:3300/0 conn(0x7f68a81540a0 0x7f68a806c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 -- 172.21.15.7:0/4083868969 >> 172.21.15.7:0/4083868969 conn(0x7f68a80cbcc0 msgr2=0x7f68a80cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 -- 172.21.15.7:0/4083868969 shutdown_connections 2023-11-28T18:49:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 -- 172.21.15.7:0/4083868969 wait complete. 2023-11-28T18:49:56.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.278+0000 7f68acac8640 1 Processor -- start 2023-11-28T18:49:56.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 -- start start 2023-11-28T18:49:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806c9b0 0x7f68a81f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a806d590 0x7f68a8152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f68a81540a0 0x7f68a815c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8148940 con 0x7f68a806c9b0 2023-11-28T18:49:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8157920 con 0x7f68a806d590 2023-11-28T18:49:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8157ee0 con 0x7f68a81540a0 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806c9b0 0x7f68a81f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806c9b0 0x7f68a81f2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51288/0 (socket says 172.21.15.7:51288) 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 -- 172.21.15.7:0/2316491876 learned_addr learned my addr 172.21.15.7:0/2316491876 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a6ffd640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a806d590 0x7f68a8152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a7fff640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f68a81540a0 0x7f68a815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 -- 172.21.15.7:0/2316491876 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f68a81540a0 msgr2=0x7f68a815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f68a81540a0 0x7f68a815c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 -- 172.21.15.7:0/2316491876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a806d590 msgr2=0x7f68a8152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a806d590 0x7f68a8152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 -- 172.21.15.7:0/2316491876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a0009b30 con 0x7f68a806c9b0 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68a77fe640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806c9b0 0x7f68a81f2df0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f689800e8a0 tx=0x7f689800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f688ffff640 1 -- 172.21.15.7:0/2316491876 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898009510 con 0x7f68a806c9b0 2023-11-28T18:49:56.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f688ffff640 1 -- 172.21.15.7:0/2316491876 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689800c940 con 0x7f68a806c9b0 2023-11-28T18:49:56.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a81f3330 con 0x7f68a806c9b0 2023-11-28T18:49:56.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f688ffff640 1 -- 172.21.15.7:0/2316491876 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689800ccb0 con 0x7f68a806c9b0 2023-11-28T18:49:56.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a80af860 con 0x7f68a806c9b0 2023-11-28T18:49:56.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.282+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a804e890 con 0x7f68a806c9b0 2023-11-28T18:49:56.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.286+0000 7f688ffff640 1 -- 172.21.15.7:0/2316491876 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6898013470 con 0x7f68a806c9b0 2023-11-28T18:49:56.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.286+0000 7f688ffff640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f68940668d0 0x7f6894068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:56.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.286+0000 7f688ffff640 1 -- 172.21.15.7:0/2316491876 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f6898084d80 con 0x7f68a806c9b0 2023-11-28T18:49:56.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.286+0000 7f68a6ffd640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f68940668d0 0x7f6894068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:56.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.286+0000 7f68a6ffd640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f68940668d0 0x7f6894068d90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f68a80c9460 tx=0x7f689c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.290+0000 7f688ffff640 1 -- 172.21.15.7:0/2316491876 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6898051ac0 con 0x7f68a806c9b0 2023-11-28T18:49:56.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68a80ca1a0 con 0x7f68a806c9b0 2023-11-28T18:49:56.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f688ffff640 1 -- 172.21.15.7:0/2316491876 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f6898021090 con 0x7f68a806c9b0 2023-11-28T18:49:56.478 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:56.478 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:56.478 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f68940668d0 msgr2=0x7f6894068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f68940668d0 0x7f6894068d90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f68a80c9460 tx=0x7f689c009040 comp rx=0 tx=0).stop 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806c9b0 msgr2=0x7f68a81f2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806c9b0 0x7f68a81f2df0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f689800e8a0 tx=0x7f689800eca0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 shutdown_connections 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f68940668d0 0x7f6894068d90 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f68a81540a0 0x7f68a815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a806d590 0x7f68a8152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 --2- 172.21.15.7:0/2316491876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a806c9b0 0x7f68a81f2df0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:56.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 >> 172.21.15.7:0/2316491876 conn(0x7f68a80cbcc0 msgr2=0x7f68a80ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:56.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 shutdown_connections 2023-11-28T18:49:56.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:56.478+0000 7f68acac8640 1 -- 172.21.15.7:0/2316491876 wait complete. 2023-11-28T18:49:56.500 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:49:58.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:49:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.654+0000 7f300759e640 1 Processor -- start 2023-11-28T18:49:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.654+0000 7f300759e640 1 -- start start 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f300759e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806da80 0x7f30080c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f300759e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30080c2810 0x7f3008150070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f300759e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3008156dd0 con 0x7f300806de60 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f300759e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30080c8d30 con 0x7f30080c2810 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f300759e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30080c33a0 con 0x7f300806da80 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f3006d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30080c2810 0x7f3008150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f3006d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30080c2810 0x7f3008150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51296/0 (socket says 172.21.15.7:51296) 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f3006d9d640 1 -- 172.21.15.7:0/4146013966 learned_addr learned my addr 172.21.15.7:0/4146013966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f300659c640 1 --2- 172.21.15.7:0/4146013966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806da80 0x7f30080c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f3006d9d640 1 -- 172.21.15.7:0/4146013966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806da80 msgr2=0x7f30080c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:49:58.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f3006d9d640 1 --2- 172.21.15.7:0/4146013966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806da80 0x7f30080c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f3006d9d640 1 -- 172.21.15.7:0/4146013966 >> v1:172.21.15.193:6789/0 conn(0x7f300806de60 legacy=0x7f30080c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:58.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.658+0000 7f3006d9d640 1 -- 172.21.15.7:0/4146013966 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3008157050 con 0x7f30080c2810 2023-11-28T18:49:58.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f3006d9d640 1 --2- 172.21.15.7:0/4146013966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008150070 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3000009e50 tx=0x7f300001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf67594e96cad29 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:58.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f3005d9b640 1 -- 172.21.15.7:0/4146013966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000af400 con 0x7f30080c2810 2023-11-28T18:49:58.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f3005d9b640 1 -- 172.21.15.7:0/4146013966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30000ab050 con 0x7f30080c2810 2023-11-28T18:49:58.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f3005d9b640 1 -- 172.21.15.7:0/4146013966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000af780 con 0x7f30080c2810 2023-11-28T18:49:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- 172.21.15.7:0/4146013966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 msgr2=0x7f3008150070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 --2- 172.21.15.7:0/4146013966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008150070 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3000009e50 tx=0x7f300001f1f0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- 172.21.15.7:0/4146013966 shutdown_connections 2023-11-28T18:49:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 --2- 172.21.15.7:0/4146013966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008150070 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 --2- 172.21.15.7:0/4146013966 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806da80 0x7f30080c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- 172.21.15.7:0/4146013966 >> 172.21.15.7:0/4146013966 conn(0x7f300806c370 msgr2=0x7f30080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- 172.21.15.7:0/4146013966 shutdown_connections 2023-11-28T18:49:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- 172.21.15.7:0/4146013966 wait complete. 2023-11-28T18:49:58.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 Processor -- start 2023-11-28T18:49:58.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- start start 2023-11-28T18:49:58.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f300806da80 0x7f300813caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:58.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806de60 0x7f30081f1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:58.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:58.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3008157ef0 con 0x7f30080c2810 2023-11-28T18:49:58.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30081591c0 con 0x7f300806da80 2023-11-28T18:49:58.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.662+0000 7f300759e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30081f1180 con 0x7f300806de60 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f300659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f300806da80 0x7f300813caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f2ffffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806de60 0x7f30081f1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51306/0 (socket says 172.21.15.7:51306) 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 -- 172.21.15.7:0/2776436500 learned_addr learned my addr 172.21.15.7:0/2776436500 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 -- 172.21.15.7:0/2776436500 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806de60 msgr2=0x7f30081f1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806de60 0x7f30081f1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 -- 172.21.15.7:0/2776436500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f300806da80 msgr2=0x7f300813caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f300806da80 0x7f300813caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 -- 172.21.15.7:0/2776436500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000009b30 con 0x7f30080c2810 2023-11-28T18:49:58.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f3006d9d640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008159700 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f300001ce20 tx=0x7f300001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:58.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f2ffeffd640 1 -- 172.21.15.7:0/2776436500 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000c0070 con 0x7f30080c2810 2023-11-28T18:49:58.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f2ffeffd640 1 -- 172.21.15.7:0/2776436500 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3000009510 con 0x7f30080c2810 2023-11-28T18:49:58.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f2ffeffd640 1 -- 172.21.15.7:0/2776436500 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000afda0 con 0x7f30080c2810 2023-11-28T18:49:58.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30081f1b50 con 0x7f30080c2810 2023-11-28T18:49:58.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3008157650 con 0x7f30080c2810 2023-11-28T18:49:58.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30081f1330 con 0x7f30080c2810 2023-11-28T18:49:58.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.666+0000 7f2ffeffd640 1 -- 172.21.15.7:0/2776436500 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f30000b6040 con 0x7f30080c2810 2023-11-28T18:49:58.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.670+0000 7f2ffeffd640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2fe80668d0 0x7f2fe8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:49:58.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.670+0000 7f2ffeffd640 1 -- 172.21.15.7:0/2776436500 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f3000122220 con 0x7f30080c2810 2023-11-28T18:49:58.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.670+0000 7f300659c640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2fe80668d0 0x7f2fe8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:49:58.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.670+0000 7f300659c640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2fe80668d0 0x7f2fe8068d90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80045f0 tx=0x7f2ff8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:49:58.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.674+0000 7f2ffeffd640 1 -- 172.21.15.7:0/2776436500 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f30081f1330 con 0x7f30080c2810 2023-11-28T18:49:58.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.858+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30081f1330 con 0x7f30080c2810 2023-11-28T18:49:58.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f2ffeffd640 1 -- 172.21.15.7:0/2776436500 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f30081f1330 con 0x7f30080c2810 2023-11-28T18:49:58.861 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:49:58.861 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:49:58.861 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2fe80668d0 msgr2=0x7f2fe8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2fe80668d0 0x7f2fe8068d90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80045f0 tx=0x7f2ff8008040 comp rx=0 tx=0).stop 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 msgr2=0x7f3008159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008159700 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f300001ce20 tx=0x7f300001bf90 comp rx=0 tx=0).stop 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 shutdown_connections 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2fe80668d0 0x7f2fe8068d90 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30080c2810 0x7f3008159700 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f300806de60 0x7f30081f1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 --2- 172.21.15.7:0/2776436500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f300806da80 0x7f300813caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:49:58.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 >> 172.21.15.7:0/2776436500 conn(0x7f300806c370 msgr2=0x7f300814bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:49:58.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 shutdown_connections 2023-11-28T18:49:58.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:49:58.862+0000 7f300759e640 1 -- 172.21.15.7:0/2776436500 wait complete. 2023-11-28T18:49:58.883 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:50:00.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:01.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.038+0000 7fc171f72640 1 Processor -- start 2023-11-28T18:50:01.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.038+0000 7fc171f72640 1 -- start start 2023-11-28T18:50:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc171f72640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc16c1503c0 0x7fc16c0c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc171f72640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 0x7fc16c0c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc171f72640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16c0c3920 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc171f72640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c157240 con 0x7fc16c1503c0 2023-11-28T18:50:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc171f72640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c0c2ce0 con 0x7fc16c0c3490 2023-11-28T18:50:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16affd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc16c0c3bc0 0x7fc16c06d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42112/0 (socket says 172.21.15.7:42112) 2023-11-28T18:50:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16affd640 1 -- 172.21.15.7:0/686379427 learned_addr learned my addr 172.21.15.7:0/686379427 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16b7fe640 1 --2- 172.21.15.7:0/686379427 >> v2:172.21.15.193:3300/0 conn(0x7fc16c1503c0 0x7fc16c0c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16bfff640 1 --2- 172.21.15.7:0/686379427 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 0x7fc16c0c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16a7fc640 1 -- 172.21.15.7:0/686379427 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495392158 0 0) 0x7fc160001120 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16a7fc640 1 -- 172.21.15.7:0/686379427 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc154003530 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16b7fe640 1 -- 172.21.15.7:0/686379427 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 msgr2=0x7fc16c0c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16b7fe640 1 --2- 172.21.15.7:0/686379427 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 0x7fc16c0c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16b7fe640 1 -- 172.21.15.7:0/686379427 >> v1:172.21.15.7:6789/0 conn(0x7fc16c0c3bc0 legacy=0x7fc16c06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.042+0000 7fc16b7fe640 1 -- 172.21.15.7:0/686379427 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c0c2f80 con 0x7fc16c1503c0 2023-11-28T18:50:01.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc16b7fe640 1 --2- 172.21.15.7:0/686379427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc16c1503c0 0x7fc16c0c7c60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0097c0 tx=0x7fc15c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d0a06fe067a5edc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc16a7fc640 1 -- 172.21.15.7:0/686379427 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0a8970 con 0x7fc16c1503c0 2023-11-28T18:50:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc16a7fc640 1 -- 172.21.15.7:0/686379427 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c0a8ad0 con 0x7fc16c1503c0 2023-11-28T18:50:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc16a7fc640 1 -- 172.21.15.7:0/686379427 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0a8e60 con 0x7fc16c1503c0 2023-11-28T18:50:01.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc171f72640 1 -- 172.21.15.7:0/686379427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc16c1503c0 msgr2=0x7fc16c0c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:01.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc171f72640 1 --2- 172.21.15.7:0/686379427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc16c1503c0 0x7fc16c0c7c60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0097c0 tx=0x7fc15c01ec70 comp rx=0 tx=0).stop 2023-11-28T18:50:01.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc171f72640 1 -- 172.21.15.7:0/686379427 shutdown_connections 2023-11-28T18:50:01.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc171f72640 1 --2- 172.21.15.7:0/686379427 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 0x7fc16c0c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc171f72640 1 --2- 172.21.15.7:0/686379427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc16c1503c0 0x7fc16c0c7c60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc171f72640 1 -- 172.21.15.7:0/686379427 >> 172.21.15.7:0/686379427 conn(0x7fc16c0662e0 msgr2=0x7fc16c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:01.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc171f72640 1 -- 172.21.15.7:0/686379427 shutdown_connections 2023-11-28T18:50:01.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.046+0000 7fc171f72640 1 -- 172.21.15.7:0/686379427 wait complete. 2023-11-28T18:50:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 Processor -- start 2023-11-28T18:50:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 -- start start 2023-11-28T18:50:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 0x7fc16c153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c0c3bc0 0x7fc16c0c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc16c1503c0 0x7fc16c1f0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c154250 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c1e05a0 con 0x7fc16c1503c0 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c1dfb50 con 0x7fc16c0c3490 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c0c3bc0 0x7fc16c0c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c0c3bc0 0x7fc16c0c6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51308/0 (socket says 172.21.15.7:51308) 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 -- 172.21.15.7:0/2809661695 learned_addr learned my addr 172.21.15.7:0/2809661695 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 -- 172.21.15.7:0/2809661695 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 msgr2=0x7fc16c153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 0x7fc16c153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 -- 172.21.15.7:0/2809661695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc16c1503c0 msgr2=0x7fc16c1f0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc16c1503c0 0x7fc16c1f0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 -- 172.21.15.7:0/2809661695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c009420 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc16affd640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c0c3bc0 0x7fc16c0c6090 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc16000c700 tx=0x7fc16000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc153fff640 1 -- 172.21.15.7:0/2809661695 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc160005ac0 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc153fff640 1 -- 172.21.15.7:0/2809661695 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc160005c60 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc153fff640 1 -- 172.21.15.7:0/2809661695 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc160001d60 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c154430 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.050+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c1df220 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.054+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c04e890 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.054+0000 7fc153fff640 1 -- 172.21.15.7:0/2809661695 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc160007070 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.054+0000 7fc153fff640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1580668d0 0x7fc158068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:01.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.054+0000 7fc153fff640 1 -- 172.21.15.7:0/2809661695 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fc160083af0 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.054+0000 7fc16b7fe640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1580668d0 0x7fc158068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:01.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.054+0000 7fc16b7fe640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1580668d0 0x7fc158068d90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0b2000 tx=0x7fc15c004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:01.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.058+0000 7fc153fff640 1 -- 172.21.15.7:0/2809661695 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc160050950 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.246+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc16c0c3920 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc153fff640 1 -- 172.21.15.7:0/2809661695 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7fc16c0c3920 con 0x7fc16c0c3bc0 2023-11-28T18:50:01.249 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:01.250 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:01.250 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:01.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1580668d0 msgr2=0x7fc158068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:01.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1580668d0 0x7fc158068d90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0b2000 tx=0x7fc15c004740 comp rx=0 tx=0).stop 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c0c3bc0 msgr2=0x7fc16c0c6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c0c3bc0 0x7fc16c0c6090 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc16000c700 tx=0x7fc16000cbc0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 shutdown_connections 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc1580668d0 0x7fc158068d90 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc16c1503c0 0x7fc16c1f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c0c3bc0 0x7fc16c0c6090 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 --2- 172.21.15.7:0/2809661695 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc16c0c3490 0x7fc16c153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 >> 172.21.15.7:0/2809661695 conn(0x7fc16c0662e0 msgr2=0x7fc16c1472d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:01.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 shutdown_connections 2023-11-28T18:50:01.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:01.250+0000 7fc171f72640 1 -- 172.21.15.7:0/2809661695 wait complete. 2023-11-28T18:50:01.271 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:01.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~282s 2023-11-28T18:50:03.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:03.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.426+0000 7f770d71d640 1 Processor -- start 2023-11-28T18:50:03.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.426+0000 7f770d71d640 1 -- start start 2023-11-28T18:50:03.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f770d71d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f77081541a0 0x7f77080c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f770d71d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77080c7db0 0x7f77080c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f770d71d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f77080c7200 0x7f77080c7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f770d71d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770815e190 con 0x7f77081541a0 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f770d71d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770815d330 con 0x7f77080c7200 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f770d71d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770815d580 con 0x7f77080c7db0 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f7706ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f77081541a0 0x7f77080c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f7706ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f77081541a0 0x7f77080c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51310/0 (socket says 172.21.15.7:51310) 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f7706ffd640 1 -- 172.21.15.7:0/1547903160 learned_addr learned my addr 172.21.15.7:0/1547903160 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f76fffff640 1 --2- 172.21.15.7:0/1547903160 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77080c7db0 0x7f77080c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:03.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f77077fe640 1 --2- 172.21.15.7:0/1547903160 >> v2:172.21.15.193:3300/0 conn(0x7f77080c7200 0x7f77080c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:03.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f7706ffd640 1 -- 172.21.15.7:0/1547903160 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77080c7db0 msgr2=0x7f77080c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:03.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f7706ffd640 1 --2- 172.21.15.7:0/1547903160 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77080c7db0 0x7f77080c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f7706ffd640 1 -- 172.21.15.7:0/1547903160 >> v2:172.21.15.193:3300/0 conn(0x7f77080c7200 msgr2=0x7f77080c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:03.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f7706ffd640 1 --2- 172.21.15.7:0/1547903160 >> v2:172.21.15.193:3300/0 conn(0x7f77080c7200 0x7f77080c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.430+0000 7f7706ffd640 1 -- 172.21.15.7:0/1547903160 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770815d860 con 0x7f77081541a0 2023-11-28T18:50:03.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f7706ffd640 1 --2- 172.21.15.7:0/1547903160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77081541a0 0x7f77080c8c80 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f770806ed30 tx=0x7f76f801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0cfa6d6be9382f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f77067fc640 1 -- 172.21.15.7:0/1547903160 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80a89b0 con 0x7f77081541a0 2023-11-28T18:50:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f77067fc640 1 -- 172.21.15.7:0/1547903160 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f80a8b10 con 0x7f77081541a0 2023-11-28T18:50:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f77067fc640 1 -- 172.21.15.7:0/1547903160 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80a8ea0 con 0x7f77081541a0 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 -- 172.21.15.7:0/1547903160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77081541a0 msgr2=0x7f77080c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 --2- 172.21.15.7:0/1547903160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77081541a0 0x7f77080c8c80 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f770806ed30 tx=0x7f76f801ecf0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 -- 172.21.15.7:0/1547903160 shutdown_connections 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 --2- 172.21.15.7:0/1547903160 >> v2:172.21.15.193:3300/0 conn(0x7f77080c7200 0x7f77080c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 --2- 172.21.15.7:0/1547903160 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77080c7db0 0x7f77080c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 --2- 172.21.15.7:0/1547903160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77081541a0 0x7f77080c8c80 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 -- 172.21.15.7:0/1547903160 >> 172.21.15.7:0/1547903160 conn(0x7f77080662e0 msgr2=0x7f77080666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 -- 172.21.15.7:0/1547903160 shutdown_connections 2023-11-28T18:50:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 -- 172.21.15.7:0/1547903160 wait complete. 2023-11-28T18:50:03.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 Processor -- start 2023-11-28T18:50:03.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.434+0000 7f770d71d640 1 -- start start 2023-11-28T18:50:03.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77080c7200 0x7f770814f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:03.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77080c7db0 0x7f7708148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:03.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77081541a0 0x7f7708146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:03.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77081501c0 con 0x7f77080c7200 2023-11-28T18:50:03.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770815b320 con 0x7f77080c7db0 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7708147ea0 con 0x7f77081541a0 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77080c7200 0x7f770814f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77080c7200 0x7f770814f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51326/0 (socket says 172.21.15.7:51326) 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 -- 172.21.15.7:0/2992993214 learned_addr learned my addr 172.21.15.7:0/2992993214 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f76fffff640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77080c7db0 0x7f7708148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f77077fe640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77081541a0 0x7f7708146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 -- 172.21.15.7:0/2992993214 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77081541a0 msgr2=0x7f7708146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77081541a0 0x7f7708146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 -- 172.21.15.7:0/2992993214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77080c7db0 msgr2=0x7f7708148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77080c7db0 0x7f7708148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 -- 172.21.15.7:0/2992993214 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f80094d0 con 0x7f77080c7200 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f7706ffd640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77080c7200 0x7f770814f230 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f76f80b3040 tx=0x7f76f801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f76ff7fe640 1 -- 172.21.15.7:0/2992993214 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80b0e00 con 0x7f77080c7200 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f76ff7fe640 1 -- 172.21.15.7:0/2992993214 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f80bb470 con 0x7f77080c7200 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77080c1ea0 con 0x7f77080c7200 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f76ff7fe640 1 -- 172.21.15.7:0/2992993214 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80bb820 con 0x7f77080c7200 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770814fee0 con 0x7f77080c7200 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770804e890 con 0x7f77080c7200 2023-11-28T18:50:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.438+0000 7f76ff7fe640 1 -- 172.21.15.7:0/2992993214 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f76f80b7070 con 0x7f77080c7200 2023-11-28T18:50:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.442+0000 7f76ff7fe640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f76e80668d0 0x7f76e8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.442+0000 7f76ff7fe640 1 -- 172.21.15.7:0/2992993214 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f76f8122ba0 con 0x7f77080c7200 2023-11-28T18:50:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.442+0000 7f76fffff640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f76e80668d0 0x7f76e8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.442+0000 7f76fffff640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f76e80668d0 0x7f76e8068d90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f77081f4080 tx=0x7f76f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:03.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.446+0000 7f76ff7fe640 1 -- 172.21.15.7:0/2992993214 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f76f80ef8e0 con 0x7f77080c7200 2023-11-28T18:50:03.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77080b1c30 con 0x7f77080c7200 2023-11-28T18:50:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f76ff7fe640 1 -- 172.21.15.7:0/2992993214 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f76f80f2f00 con 0x7f77080c7200 2023-11-28T18:50:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:03.638 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f76e80668d0 msgr2=0x7f76e8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f76e80668d0 0x7f76e8068d90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f77081f4080 tx=0x7f76f0008040 comp rx=0 tx=0).stop 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77080c7200 msgr2=0x7f770814f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77080c7200 0x7f770814f230 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f76f80b3040 tx=0x7f76f801ba90 comp rx=0 tx=0).stop 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 shutdown_connections 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f76e80668d0 0x7f76e8068d90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f77081541a0 0x7f7708146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77080c7db0 0x7f7708148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 --2- 172.21.15.7:0/2992993214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77080c7200 0x7f770814f230 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 >> 172.21.15.7:0/2992993214 conn(0x7f77080662e0 msgr2=0x7f770815a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 shutdown_connections 2023-11-28T18:50:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:03.638+0000 7f770d71d640 1 -- 172.21.15.7:0/2992993214 wait complete. 2023-11-28T18:50:03.660 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:05.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.814+0000 7ff7ff380640 1 Processor -- start 2023-11-28T18:50:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.814+0000 7ff7ff380640 1 -- start start 2023-11-28T18:50:05.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7ff380640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7f80c4740 0x7ff7f80cab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:05.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7ff380640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f81522e0 0x7ff7f8162660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:05.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7ff380640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f80cb0d0 con 0x7ff7f8151f00 2023-11-28T18:50:05.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7ff380640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f80c9f00 con 0x7ff7f80c4740 2023-11-28T18:50:05.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7ff380640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f8072950 con 0x7ff7f81522e0 2023-11-28T18:50:05.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fd11c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff7f8151f00 0x7ff7f80cd6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60136/0 (socket says 172.21.15.7:60136) 2023-11-28T18:50:05.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fd11c640 1 -- 172.21.15.7:0/2090717115 learned_addr learned my addr 172.21.15.7:0/2090717115 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:05.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fc91b640 1 --2- 172.21.15.7:0/2090717115 >> v2:172.21.15.7:3300/0 conn(0x7ff7f80c4740 0x7ff7f80cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:05.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fd91d640 1 --2- 172.21.15.7:0/2090717115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f81522e0 0x7ff7f8162660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:05.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fc91b640 1 -- 172.21.15.7:0/2090717115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f81522e0 msgr2=0x7ff7f8162660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:05.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fc91b640 1 --2- 172.21.15.7:0/2090717115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f81522e0 0x7ff7f8162660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:05.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fc91b640 1 -- 172.21.15.7:0/2090717115 >> v1:172.21.15.7:6789/0 conn(0x7ff7f8151f00 legacy=0x7ff7f80cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:05.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fc91b640 1 -- 172.21.15.7:0/2090717115 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8072c30 con 0x7ff7f80c4740 2023-11-28T18:50:05.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7effff640 1 -- 172.21.15.7:0/2090717115 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768901617 0 0) 0x7ff7f4001120 con 0x7ff7f8151f00 2023-11-28T18:50:05.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.818+0000 7ff7fc91b640 1 --2- 172.21.15.7:0/2090717115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f80cab90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80097c0 tx=0x7ff7e801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6912aa34ecb6a58 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7effff640 1 -- 172.21.15.7:0/2090717115 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80ab600 con 0x7ff7f80c4740 2023-11-28T18:50:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7effff640 1 -- 172.21.15.7:0/2090717115 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e80a7050 con 0x7ff7f80c4740 2023-11-28T18:50:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7effff640 1 -- 172.21.15.7:0/2090717115 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80ab980 con 0x7ff7f80c4740 2023-11-28T18:50:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- 172.21.15.7:0/2090717115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 msgr2=0x7ff7f80cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2090717115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f80cab90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80097c0 tx=0x7ff7e801c3f0 comp rx=0 tx=0).stop 2023-11-28T18:50:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- 172.21.15.7:0/2090717115 shutdown_connections 2023-11-28T18:50:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2090717115 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f81522e0 0x7ff7f8162660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2090717115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f80cab90 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- 172.21.15.7:0/2090717115 >> 172.21.15.7:0/2090717115 conn(0x7ff7f8073cd0 msgr2=0x7ff7f80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- 172.21.15.7:0/2090717115 shutdown_connections 2023-11-28T18:50:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- 172.21.15.7:0/2090717115 wait complete. 2023-11-28T18:50:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 Processor -- start 2023-11-28T18:50:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- start start 2023-11-28T18:50:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f806b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f8151f00 0x7ff7f8157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7f81f6e80 0x7ff7f8140db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f81503c0 con 0x7ff7f80c4740 2023-11-28T18:50:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f815abf0 con 0x7ff7f81f6e80 2023-11-28T18:50:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.822+0000 7ff7ff380640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f81f69f0 con 0x7ff7f8151f00 2023-11-28T18:50:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f806b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f806b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43676/0 (socket says 172.21.15.7:43676) 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 -- 172.21.15.7:0/2214977718 learned_addr learned my addr 172.21.15.7:0/2214977718 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fc91b640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f8151f00 0x7ff7f8157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd91d640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7f81f6e80 0x7ff7f8140db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 -- 172.21.15.7:0/2214977718 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f8151f00 msgr2=0x7ff7f8157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f8151f00 0x7ff7f8157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 -- 172.21.15.7:0/2214977718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7f81f6e80 msgr2=0x7ff7f8140db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7f81f6e80 0x7ff7f8140db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 -- 172.21.15.7:0/2214977718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e8009420 con 0x7ff7f80c4740 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7fd11c640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f806b230 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400c780 tx=0x7ff7f400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:05.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7edffb640 1 -- 172.21.15.7:0/2214977718 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40072d0 con 0x7ff7f80c4740 2023-11-28T18:50:05.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7edffb640 1 -- 172.21.15.7:0/2214977718 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f4007450 con 0x7ff7f80c4740 2023-11-28T18:50:05.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f806b770 con 0x7ff7f80c4740 2023-11-28T18:50:05.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7edffb640 1 -- 172.21.15.7:0/2214977718 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40047f0 con 0x7ff7f80c4740 2023-11-28T18:50:05.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f8159970 con 0x7ff7f80c4740 2023-11-28T18:50:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f804e890 con 0x7ff7f80c4740 2023-11-28T18:50:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.826+0000 7ff7edffb640 1 -- 172.21.15.7:0/2214977718 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff7f4004950 con 0x7ff7f80c4740 2023-11-28T18:50:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.830+0000 7ff7edffb640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff7dc066920 0x7ff7dc068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.830+0000 7ff7edffb640 1 -- 172.21.15.7:0/2214977718 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff7f4083d30 con 0x7ff7f80c4740 2023-11-28T18:50:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.830+0000 7ff7fc91b640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff7dc066920 0x7ff7dc068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.830+0000 7ff7fc91b640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff7dc066920 0x7ff7dc068de0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80ad000 tx=0x7ff7e80a4f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:05.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:05.834+0000 7ff7edffb640 1 -- 172.21.15.7:0/2214977718 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff7f4050b80 con 0x7ff7f80c4740 2023-11-28T18:50:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.022+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7f80cb0d0 con 0x7ff7f80c4740 2023-11-28T18:50:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.022+0000 7ff7edffb640 1 -- 172.21.15.7:0/2214977718 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7ff7f80cb0d0 con 0x7ff7f80c4740 2023-11-28T18:50:06.023 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:06.023 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:06.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.022+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff7dc066920 msgr2=0x7ff7dc068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:06.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.022+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff7dc066920 0x7ff7dc068de0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80ad000 tx=0x7ff7e80a4f90 comp rx=0 tx=0).stop 2023-11-28T18:50:06.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.022+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 msgr2=0x7ff7f806b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:06.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.022+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f806b230 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400c780 tx=0x7ff7f400cc40 comp rx=0 tx=0).stop 2023-11-28T18:50:06.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.026+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 shutdown_connections 2023-11-28T18:50:06.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.026+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff7dc066920 0x7ff7dc068de0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:06.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.026+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7f81f6e80 0x7ff7f8140db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:06.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.026+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff7f8151f00 0x7ff7f8157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:06.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.026+0000 7ff7ff380640 1 --2- 172.21.15.7:0/2214977718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80c4740 0x7ff7f806b230 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:06.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.026+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 >> 172.21.15.7:0/2214977718 conn(0x7ff7f8073cd0 msgr2=0x7ff7f814b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:06.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.026+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 shutdown_connections 2023-11-28T18:50:06.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:06.026+0000 7ff7ff380640 1 -- 172.21.15.7:0/2214977718 wait complete. 2023-11-28T18:50:06.045 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:08.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:08.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.198+0000 7f22d54e3640 1 Processor -- start 2023-11-28T18:50:08.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22d54e3640 1 -- start start 2023-11-28T18:50:08.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22d54e3640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f22d0154040 0x7f22d0159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:08.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22d54e3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22d015a3e0 0x7f22d00669f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:08.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22d54e3640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d00c8430 con 0x7f22d006a8a0 2023-11-28T18:50:08.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22d54e3640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d015a830 con 0x7f22d015a3e0 2023-11-28T18:50:08.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22d54e3640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d006ae50 con 0x7f22d0154040 2023-11-28T18:50:08.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22c7fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f22d006a8a0 0x7f22d00662e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60152/0 (socket says 172.21.15.7:60152) 2023-11-28T18:50:08.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22c7fff640 1 -- 172.21.15.7:0/3251445073 learned_addr learned my addr 172.21.15.7:0/3251445073 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:08.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22cf7fe640 1 --2- 172.21.15.7:0/3251445073 >> v2:172.21.15.7:3300/0 conn(0x7f22d015a3e0 0x7f22d00669f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:08.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22ceffd640 1 --2- 172.21.15.7:0/3251445073 >> v2:172.21.15.193:3300/0 conn(0x7f22d0154040 0x7f22d0159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22ce7fc640 1 -- 172.21.15.7:0/3251445073 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767007615 0 0) 0x7f22d00c8430 con 0x7f22d006a8a0 2023-11-28T18:50:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22ce7fc640 1 -- 172.21.15.7:0/3251445073 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22bc003530 con 0x7f22d006a8a0 2023-11-28T18:50:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22cf7fe640 1 -- 172.21.15.7:0/3251445073 >> v2:172.21.15.193:3300/0 conn(0x7f22d0154040 msgr2=0x7f22d0159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22cf7fe640 1 --2- 172.21.15.7:0/3251445073 >> v2:172.21.15.193:3300/0 conn(0x7f22d0154040 0x7f22d0159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22cf7fe640 1 -- 172.21.15.7:0/3251445073 >> v1:172.21.15.7:6789/0 conn(0x7f22d006a8a0 legacy=0x7f22d00662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.202+0000 7f22cf7fe640 1 -- 172.21.15.7:0/3251445073 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d006b110 con 0x7f22d015a3e0 2023-11-28T18:50:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22cf7fe640 1 --2- 172.21.15.7:0/3251445073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d015a3e0 0x7f22d00669f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009d90 tx=0x7f22c801f100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed58b1a4feb85b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22ce7fc640 1 -- 172.21.15.7:0/3251445073 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c80af600 con 0x7f22d015a3e0 2023-11-28T18:50:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22ce7fc640 1 -- 172.21.15.7:0/3251445073 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c80ab050 con 0x7f22d015a3e0 2023-11-28T18:50:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22ce7fc640 1 -- 172.21.15.7:0/3251445073 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c80af980 con 0x7f22d015a3e0 2023-11-28T18:50:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 -- 172.21.15.7:0/3251445073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d015a3e0 msgr2=0x7f22d00669f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 --2- 172.21.15.7:0/3251445073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d015a3e0 0x7f22d00669f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009d90 tx=0x7f22c801f100 comp rx=0 tx=0).stop 2023-11-28T18:50:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 -- 172.21.15.7:0/3251445073 shutdown_connections 2023-11-28T18:50:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 --2- 172.21.15.7:0/3251445073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d015a3e0 0x7f22d00669f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 --2- 172.21.15.7:0/3251445073 >> v2:172.21.15.193:3300/0 conn(0x7f22d0154040 0x7f22d0159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 -- 172.21.15.7:0/3251445073 >> 172.21.15.7:0/3251445073 conn(0x7f22d006d4d0 msgr2=0x7f22d006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 -- 172.21.15.7:0/3251445073 shutdown_connections 2023-11-28T18:50:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 -- 172.21.15.7:0/3251445073 wait complete. 2023-11-28T18:50:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 Processor -- start 2023-11-28T18:50:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.206+0000 7f22d54e3640 1 -- start start 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d006a8a0 0x7f22d00cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d0154040 0x7f22d00cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d015a3e0 0x7f22d015b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d01effe0 con 0x7f22d006a8a0 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d00ccc30 con 0x7f22d0154040 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d014eb70 con 0x7f22d015a3e0 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d006a8a0 0x7f22d00cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d006a8a0 0x7f22d00cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43690/0 (socket says 172.21.15.7:43690) 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 -- 172.21.15.7:0/1394873318 learned_addr learned my addr 172.21.15.7:0/1394873318 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 -- 172.21.15.7:0/1394873318 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d015a3e0 msgr2=0x7f22d015b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d015a3e0 0x7f22d015b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 -- 172.21.15.7:0/1394873318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d0154040 msgr2=0x7f22d00cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22c7fff640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d0154040 0x7f22d00cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d0154040 0x7f22d00cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 -- 172.21.15.7:0/1394873318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c8009a70 con 0x7f22d006a8a0 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22c7fff640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d0154040 0x7f22d00cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22ceffd640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d006a8a0 0x7f22d00cae80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f22c000e930 tx=0x7f22c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22c77fe640 1 -- 172.21.15.7:0/1394873318 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0009690 con 0x7f22d006a8a0 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22c77fe640 1 -- 172.21.15.7:0/1394873318 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c000ca00 con 0x7f22d006a8a0 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d01ef380 con 0x7f22d006a8a0 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22c77fe640 1 -- 172.21.15.7:0/1394873318 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c000cd70 con 0x7f22d006a8a0 2023-11-28T18:50:08.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d01ef640 con 0x7f22d006a8a0 2023-11-28T18:50:08.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.210+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d004e890 con 0x7f22d006a8a0 2023-11-28T18:50:08.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.214+0000 7f22c77fe640 1 -- 172.21.15.7:0/1394873318 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f22c0013470 con 0x7f22d006a8a0 2023-11-28T18:50:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.214+0000 7f22c77fe640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b00668d0 0x7f22b0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.214+0000 7f22c77fe640 1 -- 172.21.15.7:0/1394873318 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f22c0084c20 con 0x7f22d006a8a0 2023-11-28T18:50:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.214+0000 7f22c7fff640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b00668d0 0x7f22b0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.214+0000 7f22c7fff640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b00668d0 0x7f22b0068d90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f22d014e710 tx=0x7f22b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.218+0000 7f22c77fe640 1 -- 172.21.15.7:0/1394873318 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f22c0051970 con 0x7f22d006a8a0 2023-11-28T18:50:08.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.410+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22d0153ac0 con 0x7f22d006a8a0 2023-11-28T18:50:08.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.410+0000 7f22c77fe640 1 -- 172.21.15.7:0/1394873318 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f22c0021090 con 0x7f22d006a8a0 2023-11-28T18:50:08.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:08.412 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:08.412 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:08.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b00668d0 msgr2=0x7f22b0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:08.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b00668d0 0x7f22b0068d90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f22d014e710 tx=0x7f22b8009040 comp rx=0 tx=0).stop 2023-11-28T18:50:08.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d006a8a0 msgr2=0x7f22d00cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:08.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d006a8a0 0x7f22d00cae80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f22c000e930 tx=0x7f22c000edf0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 shutdown_connections 2023-11-28T18:50:08.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f22b00668d0 0x7f22b0068d90 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f22d015a3e0 0x7f22d015b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22d0154040 0x7f22d00cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 --2- 172.21.15.7:0/1394873318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22d006a8a0 0x7f22d00cae80 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:08.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 >> 172.21.15.7:0/1394873318 conn(0x7f22d006d4d0 msgr2=0x7f22d0157d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:08.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 shutdown_connections 2023-11-28T18:50:08.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:08.414+0000 7f22d54e3640 1 -- 172.21.15.7:0/1394873318 wait complete. 2023-11-28T18:50:08.434 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:08.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~289s 2023-11-28T18:50:10.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:10.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.590+0000 7ff69faaa640 1 Processor -- start 2023-11-28T18:50:10.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.590+0000 7ff69faaa640 1 -- start start 2023-11-28T18:50:10.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69faaa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff6980c6ca0 0x7ff698161f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:10.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69faaa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff69815e300 0x7ff6981624d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:10.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69faaa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6980c2970 con 0x7ff6980c68c0 2023-11-28T18:50:10.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69faaa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69815e820 con 0x7ff6980c6ca0 2023-11-28T18:50:10.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69faaa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69815eb00 con 0x7ff69815e300 2023-11-28T18:50:10.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69d045640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff6980c6ca0 0x7ff698161f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:10.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69e047640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff69815e300 0x7ff6981624d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:10.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69d846640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff6980c68c0 0x7ff6980c2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:44678/0 (socket says 172.21.15.7:44678) 2023-11-28T18:50:10.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69d846640 1 -- 172.21.15.7:0/1559507925 learned_addr learned my addr 172.21.15.7:0/1559507925 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:10.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69d045640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7ff6980c6ca0 0x7ff698161f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:41792/0 (socket says 172.21.15.7:41792) 2023-11-28T18:50:10.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69d045640 1 -- 172.21.15.7:0/1559507925 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff69815e300 msgr2=0x7ff6981624d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:10.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69d045640 1 --2- 172.21.15.7:0/1559507925 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff69815e300 0x7ff6981624d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69d045640 1 -- 172.21.15.7:0/1559507925 >> v1:172.21.15.193:6789/0 conn(0x7ff6980c68c0 legacy=0x7ff6980c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:10.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69d045640 1 -- 172.21.15.7:0/1559507925 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69815ede0 con 0x7ff6980c6ca0 2023-11-28T18:50:10.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69c844640 1 -- 172.21.15.7:0/1559507925 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713294763 0 0) 0x7ff6980c2970 con 0x7ff6980c68c0 2023-11-28T18:50:10.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.594+0000 7ff69e047640 1 --2- 172.21.15.7:0/1559507925 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff69815e300 0x7ff6981624d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:50:10.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69d045640 1 --2- 172.21.15.7:0/1559507925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6980c6ca0 0x7ff698161f90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff6880097c0 tx=0x7ff68801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2f2625dc26a5483 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69c844640 1 -- 172.21.15.7:0/1559507925 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6880a8970 con 0x7ff6980c6ca0 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69c844640 1 -- 172.21.15.7:0/1559507925 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6880a8ad0 con 0x7ff6980c6ca0 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69c844640 1 -- 172.21.15.7:0/1559507925 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6880a8e60 con 0x7ff6980c6ca0 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 -- 172.21.15.7:0/1559507925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6980c6ca0 msgr2=0x7ff698161f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1559507925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6980c6ca0 0x7ff698161f90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff6880097c0 tx=0x7ff68801ec70 comp rx=0 tx=0).stop 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 -- 172.21.15.7:0/1559507925 shutdown_connections 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1559507925 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff69815e300 0x7ff6981624d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1559507925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6980c6ca0 0x7ff698161f90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 -- 172.21.15.7:0/1559507925 >> 172.21.15.7:0/1559507925 conn(0x7ff6980662e0 msgr2=0x7ff6980666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:10.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 -- 172.21.15.7:0/1559507925 shutdown_connections 2023-11-28T18:50:10.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 -- 172.21.15.7:0/1559507925 wait complete. 2023-11-28T18:50:10.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 Processor -- start 2023-11-28T18:50:10.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.598+0000 7ff69faaa640 1 -- start start 2023-11-28T18:50:10.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6980c68c0 0x7ff6980cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6980c6ca0 0x7ff6980c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff69815e300 0x7ff6981e8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69815ff60 con 0x7ff6980c6ca0 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980ca180 con 0x7ff69815e300 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980c95e0 con 0x7ff6980c68c0 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6980c6ca0 0x7ff6980c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6980c6ca0 0x7ff6980c3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43702/0 (socket says 172.21.15.7:43702) 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 -- 172.21.15.7:0/1816976162 learned_addr learned my addr 172.21.15.7:0/1816976162 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d846640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6980c68c0 0x7ff6980cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69e047640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff69815e300 0x7ff6981e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 -- 172.21.15.7:0/1816976162 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6980c68c0 msgr2=0x7ff6980cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:10.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6980c68c0 0x7ff6980cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 -- 172.21.15.7:0/1816976162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff69815e300 msgr2=0x7ff6981e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:10.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff69815e300 0x7ff6981e8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 -- 172.21.15.7:0/1816976162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff690009a70 con 0x7ff6980c6ca0 2023-11-28T18:50:10.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69d045640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6980c6ca0 0x7ff6980c3dd0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7ff68801c8a0 tx=0x7ff68801d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:10.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff68e7fc640 1 -- 172.21.15.7:0/1816976162 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6880b1e00 con 0x7ff6980c6ca0 2023-11-28T18:50:10.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff68e7fc640 1 -- 172.21.15.7:0/1816976162 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff688006e20 con 0x7ff6980c6ca0 2023-11-28T18:50:10.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff68e7fc640 1 -- 172.21.15.7:0/1816976162 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff688004230 con 0x7ff6980c6ca0 2023-11-28T18:50:10.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff688009420 con 0x7ff6980c6ca0 2023-11-28T18:50:10.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6980c9b30 con 0x7ff6980c6ca0 2023-11-28T18:50:10.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69804e890 con 0x7ff6980c6ca0 2023-11-28T18:50:10.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.602+0000 7ff68e7fc640 1 -- 172.21.15.7:0/1816976162 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff688004390 con 0x7ff6980c6ca0 2023-11-28T18:50:10.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.606+0000 7ff68e7fc640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff67c0668d0 0x7ff67c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:10.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.606+0000 7ff68e7fc640 1 -- 172.21.15.7:0/1816976162 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7ff688121f40 con 0x7ff6980c6ca0 2023-11-28T18:50:10.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.606+0000 7ff69d846640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff67c0668d0 0x7ff67c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:10.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.606+0000 7ff69d846640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff67c0668d0 0x7ff67c068d90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff6940011f0 tx=0x7ff694001140 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:10.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.610+0000 7ff68e7fc640 1 -- 172.21.15.7:0/1816976162 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff6880eeed0 con 0x7ff6980c6ca0 2023-11-28T18:50:10.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.810+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6980667c0 con 0x7ff6980c6ca0 2023-11-28T18:50:10.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.810+0000 7ff68e7fc640 1 -- 172.21.15.7:0/1816976162 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7ff6880b5090 con 0x7ff6980c6ca0 2023-11-28T18:50:10.813 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:10.813 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:10.813 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:10.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff67c0668d0 msgr2=0x7ff67c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff67c0668d0 0x7ff67c068d90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff6940011f0 tx=0x7ff694001140 comp rx=0 tx=0).stop 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6980c6ca0 msgr2=0x7ff6980c3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6980c6ca0 0x7ff6980c3dd0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7ff68801c8a0 tx=0x7ff68801d4b0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 shutdown_connections 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7ff67c0668d0 0x7ff67c068d90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff69815e300 0x7ff6981e8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6980c6ca0 0x7ff6980c3dd0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 --2- 172.21.15.7:0/1816976162 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff6980c68c0 0x7ff6980cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 >> 172.21.15.7:0/1816976162 conn(0x7ff6980662e0 msgr2=0x7ff6981e97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 shutdown_connections 2023-11-28T18:50:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:10.814+0000 7ff69faaa640 1 -- 172.21.15.7:0/1816976162 wait complete. 2023-11-28T18:50:10.835 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:50:12.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:50:12.095 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:50:12.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:12.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.990+0000 7f26528ea640 1 Processor -- start 2023-11-28T18:50:12.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.990+0000 7f26528ea640 1 -- start start 2023-11-28T18:50:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f26528ea640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f264c150b30 0x7f264c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f26528ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c0c7980 0x7f264c0c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f26528ea640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c0c6140 con 0x7f264c1575c0 2023-11-28T18:50:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f26528ea640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264c0c72b0 con 0x7f264c150b30 2023-11-28T18:50:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f26528ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264c070c00 con 0x7f264c0c7980 2023-11-28T18:50:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f264c150b30 0x7f264c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f2650e87640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c0c7980 0x7f264c0c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264b7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f264c1575c0 0x7f264c0c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:44684/0 (socket says 172.21.15.7:44684) 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264b7fe640 1 -- 172.21.15.7:0/2825802563 learned_addr learned my addr 172.21.15.7:0/2825802563 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264affd640 1 -- 172.21.15.7:0/2825802563 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519466874 0 0) 0x7f264c0c6140 con 0x7f264c1575c0 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264affd640 1 -- 172.21.15.7:0/2825802563 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2634003530 con 0x7f264c1575c0 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264bfff640 1 -- 172.21.15.7:0/2825802563 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c0c7980 msgr2=0x7f264c0c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264bfff640 1 --2- 172.21.15.7:0/2825802563 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c0c7980 0x7f264c0c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264bfff640 1 -- 172.21.15.7:0/2825802563 >> v1:172.21.15.193:6789/0 conn(0x7f264c1575c0 legacy=0x7f264c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f264bfff640 1 -- 172.21.15.7:0/2825802563 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c0c8350 con 0x7f264c150b30 2023-11-28T18:50:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.994+0000 7f2650e87640 1 --2- 172.21.15.7:0/2825802563 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c0c7980 0x7f264c0c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:50:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f264bfff640 1 --2- 172.21.15.7:0/2825802563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c150b30 0x7f264c0c24f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f263c0097c0 tx=0x7f263c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa046735861a088e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f264affd640 1 -- 172.21.15.7:0/2825802563 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c0a8970 con 0x7f264c150b30 2023-11-28T18:50:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f264affd640 1 -- 172.21.15.7:0/2825802563 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c0a8ad0 con 0x7f264c150b30 2023-11-28T18:50:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f264affd640 1 -- 172.21.15.7:0/2825802563 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c0a8e60 con 0x7f264c150b30 2023-11-28T18:50:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 -- 172.21.15.7:0/2825802563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c150b30 msgr2=0x7f264c0c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 --2- 172.21.15.7:0/2825802563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c150b30 0x7f264c0c24f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f263c0097c0 tx=0x7f263c01ec70 comp rx=0 tx=0).stop 2023-11-28T18:50:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 -- 172.21.15.7:0/2825802563 shutdown_connections 2023-11-28T18:50:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 --2- 172.21.15.7:0/2825802563 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c0c7980 0x7f264c0c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 --2- 172.21.15.7:0/2825802563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c150b30 0x7f264c0c24f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 -- 172.21.15.7:0/2825802563 >> 172.21.15.7:0/2825802563 conn(0x7f264c0662e0 msgr2=0x7f264c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 -- 172.21.15.7:0/2825802563 shutdown_connections 2023-11-28T18:50:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 -- 172.21.15.7:0/2825802563 wait complete. 2023-11-28T18:50:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 Processor -- start 2023-11-28T18:50:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:12.998+0000 7f26528ea640 1 -- start start 2023-11-28T18:50:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0c7980 0x7f264c1f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c150b30 0x7f264c1e1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c1575c0 0x7f264c147940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c1f34b0 con 0x7f264c0c7980 2023-11-28T18:50:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c14a030 con 0x7f264c1575c0 2023-11-28T18:50:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264c1474b0 con 0x7f264c150b30 2023-11-28T18:50:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0c7980 0x7f264c1f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0c7980 0x7f264c1f25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43716/0 (socket says 172.21.15.7:43716) 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 -- 172.21.15.7:0/3217424375 learned_addr learned my addr 172.21.15.7:0/3217424375 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264b7fe640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c150b30 0x7f264c1e1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f2650e87640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c1575c0 0x7f264c147940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 -- 172.21.15.7:0/3217424375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c150b30 msgr2=0x7f264c1e1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c150b30 0x7f264c1e1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 -- 172.21.15.7:0/3217424375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c1575c0 msgr2=0x7f264c147940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c1575c0 0x7f264c147940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 -- 172.21.15.7:0/3217424375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264400bc80 con 0x7f264c0c7980 2023-11-28T18:50:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f264bfff640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0c7980 0x7f264c1f25a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f263c0b3040 tx=0x7f263c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f2648ff9640 1 -- 172.21.15.7:0/3217424375 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c0b0d60 con 0x7f264c0c7980 2023-11-28T18:50:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f2648ff9640 1 -- 172.21.15.7:0/3217424375 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c0b9430 con 0x7f264c0c7980 2023-11-28T18:50:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f2648ff9640 1 -- 172.21.15.7:0/3217424375 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c0b97e0 con 0x7f264c0c7980 2023-11-28T18:50:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c009420 con 0x7f264c0c7980 2023-11-28T18:50:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264c1f2ef0 con 0x7f264c0c7980 2023-11-28T18:50:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.002+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264c04e890 con 0x7f264c0c7980 2023-11-28T18:50:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.006+0000 7f2648ff9640 1 -- 172.21.15.7:0/3217424375 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f263c0b7040 con 0x7f264c0c7980 2023-11-28T18:50:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.006+0000 7f2648ff9640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2638066900 0x7f2638068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.006+0000 7f2648ff9640 1 -- 172.21.15.7:0/3217424375 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f263c122230 con 0x7f264c0c7980 2023-11-28T18:50:13.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.006+0000 7f264b7fe640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2638066900 0x7f2638068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:13.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.006+0000 7f264b7fe640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2638066900 0x7f2638068dc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f264c1f0160 tx=0x7f2640007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:13.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.010+0000 7f2648ff9640 1 -- 172.21.15.7:0/3217424375 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f263c0eef70 con 0x7f264c0c7980 2023-11-28T18:50:13.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.202+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f264c0c2d80 con 0x7f264c0c7980 2023-11-28T18:50:13.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.202+0000 7f2648ff9640 1 -- 172.21.15.7:0/3217424375 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f263c0f2590 con 0x7f264c0c7980 2023-11-28T18:50:13.203 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:13.203 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:13.203 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:13.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2638066900 msgr2=0x7f2638068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:13.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2638066900 0x7f2638068dc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f264c1f0160 tx=0x7f2640007d40 comp rx=0 tx=0).stop 2023-11-28T18:50:13.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0c7980 msgr2=0x7f264c1f25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0c7980 0x7f264c1f25a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f263c0b3040 tx=0x7f263c01ba10 comp rx=0 tx=0).stop 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 shutdown_connections 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f2638066900 0x7f2638068dc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f264c1575c0 0x7f264c147940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f264c150b30 0x7f264c1e1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 --2- 172.21.15.7:0/3217424375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0c7980 0x7f264c1f25a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 >> 172.21.15.7:0/3217424375 conn(0x7f264c0662e0 msgr2=0x7f264c0c9620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 shutdown_connections 2023-11-28T18:50:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:13.206+0000 7f26528ea640 1 -- 172.21.15.7:0/3217424375 wait complete. 2023-11-28T18:50:13.225 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:15.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:15.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~296s 2023-11-28T18:50:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71ed101640 1 Processor -- start 2023-11-28T18:50:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71ed101640 1 -- start start 2023-11-28T18:50:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71ed101640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f71e8150b30 0x7f71e80c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71ed101640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e81575c0 0x7f71e80c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71ed101640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f71e80c4020 0x7f71e80c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71ed101640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e80c72b0 con 0x7f71e8150b30 2023-11-28T18:50:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71ed101640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e8070c00 con 0x7f71e80c4020 2023-11-28T18:50:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71ed101640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80c6140 con 0x7f71e81575c0 2023-11-28T18:50:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e6d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f71e8150b30 0x7f71e80c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e6d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f71e8150b30 0x7f71e80c24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41180/0 (socket says 172.21.15.7:41180) 2023-11-28T18:50:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e6d9d640 1 -- 172.21.15.7:0/3164000523 learned_addr learned my addr 172.21.15.7:0/3164000523 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e759e640 1 --2- 172.21.15.7:0/3164000523 >> v2:172.21.15.193:3300/0 conn(0x7f71e80c4020 0x7f71e80c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71dffff640 1 --2- 172.21.15.7:0/3164000523 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e81575c0 0x7f71e80c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e6d9d640 1 -- 172.21.15.7:0/3164000523 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e81575c0 msgr2=0x7f71e80c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e6d9d640 1 --2- 172.21.15.7:0/3164000523 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e81575c0 0x7f71e80c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e6d9d640 1 -- 172.21.15.7:0/3164000523 >> v2:172.21.15.193:3300/0 conn(0x7f71e80c4020 msgr2=0x7f71e80c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e6d9d640 1 --2- 172.21.15.7:0/3164000523 >> v2:172.21.15.193:3300/0 conn(0x7f71e80c4020 0x7f71e80c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e6d9d640 1 -- 172.21.15.7:0/3164000523 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e80c6370 con 0x7f71e8150b30 2023-11-28T18:50:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71e759e640 1 --2- 172.21.15.7:0/3164000523 >> v2:172.21.15.193:3300/0 conn(0x7f71e80c4020 0x7f71e80c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:50:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.382+0000 7f71dffff640 1 --2- 172.21.15.7:0/3164000523 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e81575c0 0x7f71e80c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:50:15.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71e6d9d640 1 --2- 172.21.15.7:0/3164000523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e8150b30 0x7f71e80c24f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f71e80c95f0 tx=0x7f71d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f70221777fd164 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:15.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71e659c640 1 -- 172.21.15.7:0/3164000523 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d80a89b0 con 0x7f71e8150b30 2023-11-28T18:50:15.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71e659c640 1 -- 172.21.15.7:0/3164000523 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d80a8b10 con 0x7f71e8150b30 2023-11-28T18:50:15.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71e659c640 1 -- 172.21.15.7:0/3164000523 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d80a8ea0 con 0x7f71e8150b30 2023-11-28T18:50:15.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71ed101640 1 -- 172.21.15.7:0/3164000523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e8150b30 msgr2=0x7f71e80c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:15.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71ed101640 1 --2- 172.21.15.7:0/3164000523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e8150b30 0x7f71e80c24f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f71e80c95f0 tx=0x7f71d801ecf0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71ed101640 1 -- 172.21.15.7:0/3164000523 shutdown_connections 2023-11-28T18:50:15.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71ed101640 1 --2- 172.21.15.7:0/3164000523 >> v2:172.21.15.193:3300/0 conn(0x7f71e80c4020 0x7f71e80c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71ed101640 1 --2- 172.21.15.7:0/3164000523 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e81575c0 0x7f71e80c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.386+0000 7f71ed101640 1 --2- 172.21.15.7:0/3164000523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e8150b30 0x7f71e80c24f0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 -- 172.21.15.7:0/3164000523 >> 172.21.15.7:0/3164000523 conn(0x7f71e80662e0 msgr2=0x7f71e80666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:15.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 -- 172.21.15.7:0/3164000523 shutdown_connections 2023-11-28T18:50:15.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 -- 172.21.15.7:0/3164000523 wait complete. 2023-11-28T18:50:15.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 Processor -- start 2023-11-28T18:50:15.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 -- start start 2023-11-28T18:50:15.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71e80c4020 0x7f71e81ef340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e8150b30 0x7f71e8071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e81575c0 0x7f71e806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e8071880 con 0x7f71e81575c0 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e81f0290 con 0x7f71e80c4020 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71ed101640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e81ef9f0 con 0x7f71e8150b30 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e81575c0 0x7f71e806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e81575c0 0x7f71e806dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41196/0 (socket says 172.21.15.7:41196) 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e759e640 1 -- 172.21.15.7:0/3305846577 learned_addr learned my addr 172.21.15.7:0/3305846577 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71dffff640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e8150b30 0x7f71e8071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e6d9d640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71e80c4020 0x7f71e81ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e759e640 1 -- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e8150b30 msgr2=0x7f71e8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:15.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e759e640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e8150b30 0x7f71e8071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e759e640 1 -- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71e80c4020 msgr2=0x7f71e81ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:15.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e759e640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71e80c4020 0x7f71e81ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71e759e640 1 -- 172.21.15.7:0/3305846577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80094d0 con 0x7f71e81575c0 2023-11-28T18:50:15.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.390+0000 7f71dffff640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e8150b30 0x7f71e8071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:50:15.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71e759e640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e81575c0 0x7f71e806dc20 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f71e000cef0 tx=0x7f71e000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:15.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71df7fe640 1 -- 172.21.15.7:0/3305846577 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e000d840 con 0x7f71e81575c0 2023-11-28T18:50:15.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71df7fe640 1 -- 172.21.15.7:0/3305846577 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e000d9a0 con 0x7f71e81575c0 2023-11-28T18:50:15.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e81efcd0 con 0x7f71e81575c0 2023-11-28T18:50:15.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71df7fe640 1 -- 172.21.15.7:0/3305846577 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e000dd50 con 0x7f71e81575c0 2023-11-28T18:50:15.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e8071510 con 0x7f71e81575c0 2023-11-28T18:50:15.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e8071a30 con 0x7f71e81575c0 2023-11-28T18:50:15.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71df7fe640 1 -- 172.21.15.7:0/3305846577 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f71e001e3b0 con 0x7f71e81575c0 2023-11-28T18:50:15.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71df7fe640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f71c8066900 0x7f71c8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:15.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71df7fe640 1 -- 172.21.15.7:0/3305846577 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f71e00851e0 con 0x7f71e81575c0 2023-11-28T18:50:15.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.394+0000 7f71e6d9d640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f71c8066900 0x7f71c8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:15.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.398+0000 7f71e6d9d640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f71c8066900 0x7f71c8068dc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f71d80b2000 tx=0x7f71d80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:15.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.398+0000 7f71df7fe640 1 -- 172.21.15.7:0/3305846577 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f71e8071a30 con 0x7f71e81575c0 2023-11-28T18:50:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71e804e890 con 0x7f71e81575c0 2023-11-28T18:50:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71df7fe640 1 -- 172.21.15.7:0/3305846577 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f71e0052040 con 0x7f71e81575c0 2023-11-28T18:50:15.591 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:15.591 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f71c8066900 msgr2=0x7f71c8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f71c8066900 0x7f71c8068dc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f71d80b2000 tx=0x7f71d80aa070 comp rx=0 tx=0).stop 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e81575c0 msgr2=0x7f71e806dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e81575c0 0x7f71e806dc20 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f71e000cef0 tx=0x7f71e000a710 comp rx=0 tx=0).stop 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 shutdown_connections 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f71c8066900 0x7f71c8068dc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71e81575c0 0x7f71e806dc20 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f71e8150b30 0x7f71e8071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 --2- 172.21.15.7:0/3305846577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71e80c4020 0x7f71e81ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 >> 172.21.15.7:0/3305846577 conn(0x7f71e80662e0 msgr2=0x7f71e81568e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 shutdown_connections 2023-11-28T18:50:15.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:15.590+0000 7f71ed101640 1 -- 172.21.15.7:0/3305846577 wait complete. 2023-11-28T18:50:15.613 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:17.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:17.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10dd52a640 1 Processor -- start 2023-11-28T18:50:17.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10dd52a640 1 -- start start 2023-11-28T18:50:17.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10dd52a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10d815d350 0x7f10d806a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:17.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10dd52a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d80c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:17.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10dd52a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d80c8750 con 0x7f10d81540a0 2023-11-28T18:50:17.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10dd52a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d80c8a30 con 0x7f10d815d350 2023-11-28T18:50:17.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10dd52a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80731c0 con 0x7f10d806ab30 2023-11-28T18:50:17.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10cffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10d815d350 0x7f10d806a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:17.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.770+0000 7f10d6ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f10d81540a0 0x7f10d8072ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47696/0 (socket says 172.21.15.7:47696) 2023-11-28T18:50:17.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10d6ffd640 1 -- 172.21.15.7:0/4163551055 learned_addr learned my addr 172.21.15.7:0/4163551055 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:17.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10d77fe640 1 --2- 172.21.15.7:0/4163551055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d80c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:17.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10cffff640 1 -- 172.21.15.7:0/4163551055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 msgr2=0x7f10d80c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:17.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10cffff640 1 --2- 172.21.15.7:0/4163551055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d80c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10cffff640 1 -- 172.21.15.7:0/4163551055 >> v1:172.21.15.7:6789/0 conn(0x7f10d81540a0 legacy=0x7f10d8072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:17.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10cffff640 1 -- 172.21.15.7:0/4163551055 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d815d840 con 0x7f10d815d350 2023-11-28T18:50:17.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10d67fc640 1 -- 172.21.15.7:0/4163551055 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236395461 0 0) 0x7f10d80c8750 con 0x7f10d81540a0 2023-11-28T18:50:17.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10cffff640 1 --2- 172.21.15.7:0/4163551055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d815d350 0x7f10d806a5f0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f10c00097c0 tx=0x7f10c001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e370a062e0a47f67 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:17.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10d67fc640 1 -- 172.21.15.7:0/4163551055 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00ab600 con 0x7f10d815d350 2023-11-28T18:50:17.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10d67fc640 1 -- 172.21.15.7:0/4163551055 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c00a7050 con 0x7f10d815d350 2023-11-28T18:50:17.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.774+0000 7f10d67fc640 1 -- 172.21.15.7:0/4163551055 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00ab980 con 0x7f10d815d350 2023-11-28T18:50:17.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- 172.21.15.7:0/4163551055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d815d350 msgr2=0x7f10d806a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:17.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 --2- 172.21.15.7:0/4163551055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d815d350 0x7f10d806a5f0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f10c00097c0 tx=0x7f10c001c480 comp rx=0 tx=0).stop 2023-11-28T18:50:17.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- 172.21.15.7:0/4163551055 shutdown_connections 2023-11-28T18:50:17.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 --2- 172.21.15.7:0/4163551055 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d80c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 --2- 172.21.15.7:0/4163551055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d815d350 0x7f10d806a5f0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- 172.21.15.7:0/4163551055 >> 172.21.15.7:0/4163551055 conn(0x7f10d8073cd0 msgr2=0x7f10d80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:17.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- 172.21.15.7:0/4163551055 shutdown_connections 2023-11-28T18:50:17.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- 172.21.15.7:0/4163551055 wait complete. 2023-11-28T18:50:17.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 Processor -- start 2023-11-28T18:50:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- start start 2023-11-28T18:50:17.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d814dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:17.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d81540a0 0x7f10d8148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:17.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10d815d350 0x7f10d814fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:17.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d814f760 con 0x7f10d81540a0 2023-11-28T18:50:17.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d81ec020 con 0x7f10d815d350 2023-11-28T18:50:17.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10dd52a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80c5fa0 con 0x7f10d806ab30 2023-11-28T18:50:17.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10cffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d81540a0 0x7f10d8148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:17.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10cffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d81540a0 0x7f10d8148440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41210/0 (socket says 172.21.15.7:41210) 2023-11-28T18:50:17.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10cffff640 1 -- 172.21.15.7:0/786675286 learned_addr learned my addr 172.21.15.7:0/786675286 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:17.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10d6ffd640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d814dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:17.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10d77fe640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10d815d350 0x7f10d814fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:17.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10cffff640 1 -- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 msgr2=0x7f10d814dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10cffff640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d814dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10cffff640 1 -- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10d815d350 msgr2=0x7f10d814fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10cffff640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10d815d350 0x7f10d814fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10cffff640 1 -- 172.21.15.7:0/786675286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c0009420 con 0x7f10d81540a0 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10d77fe640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10d815d350 0x7f10d814fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.778+0000 7f10d6ffd640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d814dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10cffff640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d81540a0 0x7f10d8148440 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f10c001ab30 tx=0x7f10c0019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10cf7fe640 1 -- 172.21.15.7:0/786675286 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00b5670 con 0x7f10d81540a0 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10cf7fe640 1 -- 172.21.15.7:0/786675286 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c00abe90 con 0x7f10d81540a0 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10cf7fe640 1 -- 172.21.15.7:0/786675286 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00b5a30 con 0x7f10d81540a0 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d80c6240 con 0x7f10d81540a0 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d80c6520 con 0x7f10d81540a0 2023-11-28T18:50:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d80c7e30 con 0x7f10d81540a0 2023-11-28T18:50:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10cf7fe640 1 -- 172.21.15.7:0/786675286 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f10c00b2040 con 0x7f10d81540a0 2023-11-28T18:50:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10cf7fe640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f10b80668d0 0x7f10b8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10cf7fe640 1 -- 172.21.15.7:0/786675286 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f10c011ed70 con 0x7f10d81540a0 2023-11-28T18:50:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.782+0000 7f10d6ffd640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f10b80668d0 0x7f10b8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.786+0000 7f10d6ffd640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f10b80668d0 0x7f10b8068d90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f10c8000f70 tx=0x7f10c8007c40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:17.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.786+0000 7f10cf7fe640 1 -- 172.21.15.7:0/786675286 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f10c00ebb20 con 0x7f10d81540a0 2023-11-28T18:50:17.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.982+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10d804e890 con 0x7f10d81540a0 2023-11-28T18:50:17.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.982+0000 7f10cf7fe640 1 -- 172.21.15.7:0/786675286 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f10c00bc090 con 0x7f10d81540a0 2023-11-28T18:50:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:17.986 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f10b80668d0 msgr2=0x7f10b8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f10b80668d0 0x7f10b8068d90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f10c8000f70 tx=0x7f10c8007c40 comp rx=0 tx=0).stop 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d81540a0 msgr2=0x7f10d8148440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d81540a0 0x7f10d8148440 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f10c001ab30 tx=0x7f10c0019220 comp rx=0 tx=0).stop 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 shutdown_connections 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f10b80668d0 0x7f10b8068d90 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10d815d350 0x7f10d814fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10d81540a0 0x7f10d8148440 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 --2- 172.21.15.7:0/786675286 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f10d806ab30 0x7f10d814dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 >> 172.21.15.7:0/786675286 conn(0x7f10d8073cd0 msgr2=0x7f10d814b070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 shutdown_connections 2023-11-28T18:50:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:17.986+0000 7f10dd52a640 1 -- 172.21.15.7:0/786675286 wait complete. 2023-11-28T18:50:18.006 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:20.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.162+0000 7f054185f640 1 Processor -- start 2023-11-28T18:50:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.162+0000 7f054185f640 1 -- start start 2023-11-28T18:50:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.162+0000 7f054185f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c06e560 0x7f053c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:20.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.162+0000 7f054185f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c158000 con 0x7f053c151cd0 2023-11-28T18:50:20.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.162+0000 7f054185f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c071e60 con 0x7f053c157b90 2023-11-28T18:50:20.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.162+0000 7f054185f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c070930 con 0x7f053c06e560 2023-11-28T18:50:20.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.162+0000 7f053affd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f053c151cd0 0x7f053c0c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47712/0 (socket says 172.21.15.7:47712) 2023-11-28T18:50:20.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.162+0000 7f053affd640 1 -- 172.21.15.7:0/1779288107 learned_addr learned my addr 172.21.15.7:0/1779288107 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f0533fff640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c06e560 0x7f053c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337650393 0 0) 0x7f053c158000 con 0x7f053c151cd0 2023-11-28T18:50:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0528003530 con 0x7f053c151cd0 2023-11-28T18:50:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975520784 0 0) 0x7f0528003530 con 0x7f053c151cd0 2023-11-28T18:50:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c06e560 msgr2=0x7f053c15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c06e560 0x7f053c15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 >> v1:172.21.15.193:6789/0 conn(0x7f053c157b90 legacy=0x7f053c162510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c070bf0 con 0x7f053c151cd0 2023-11-28T18:50:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3650910908 0 0) 0x7f052c001680 con 0x7f053c151cd0 2023-11-28T18:50:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 >> v1:172.21.15.7:6789/0 conn(0x7f053c151cd0 legacy=0x7f053c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f0528003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c06e560 0x7f05280047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0528004d10 0x7f0528005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c070930 con 0x7f05280039b0 2023-11-28T18:50:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052800bac0 con 0x7f0528004d10 2023-11-28T18:50:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052800bc80 con 0x7f053c06e560 2023-11-28T18:50:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053affd640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f0528003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f0533fff640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c06e560 0x7f05280047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053b7fe640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0528004d10 0x7f0528005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053affd640 1 -- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c06e560 msgr2=0x7f05280047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053affd640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c06e560 0x7f05280047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053affd640 1 -- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0528004d10 msgr2=0x7f0528005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053affd640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0528004d10 0x7f0528005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.166+0000 7f053affd640 1 -- 172.21.15.7:0/1779288107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0528003530 con 0x7f05280039b0 2023-11-28T18:50:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f053affd640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f0528003dd0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f052c00c1c0 tx=0x7f052c01cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5307d0e300558bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c005ae0 con 0x7f05280039b0 2023-11-28T18:50:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c005c40 con 0x7f05280039b0 2023-11-28T18:50:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f053a7fc640 1 -- 172.21.15.7:0/1779288107 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c007070 con 0x7f05280039b0 2023-11-28T18:50:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f054185f640 1 -- 172.21.15.7:0/1779288107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 msgr2=0x7f0528003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f054185f640 1 --2- 172.21.15.7:0/1779288107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f0528003dd0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f052c00c1c0 tx=0x7f052c01cb60 comp rx=0 tx=0).stop 2023-11-28T18:50:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f053b7fe640 1 -- 172.21.15.7:0/1779288107 reap_dead start 2023-11-28T18:50:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f054185f640 1 -- 172.21.15.7:0/1779288107 shutdown_connections 2023-11-28T18:50:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f054185f640 1 -- 172.21.15.7:0/1779288107 >> 172.21.15.7:0/1779288107 conn(0x7f053c0662e0 msgr2=0x7f053c0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f054185f640 1 -- 172.21.15.7:0/1779288107 shutdown_connections 2023-11-28T18:50:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f054185f640 1 -- 172.21.15.7:0/1779288107 wait complete. 2023-11-28T18:50:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f054185f640 1 Processor -- start 2023-11-28T18:50:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.170+0000 7f054185f640 1 -- start start 2023-11-28T18:50:20.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f053c159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:20.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c0c52e0 0x7f053c0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:20.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c159640 0x7f053c1f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:20.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c1f6680 con 0x7f05280039b0 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c159b10 con 0x7f053c159640 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c072620 con 0x7f053c0c52e0 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f053c159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f053c159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41220/0 (socket says 172.21.15.7:41220) 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 -- 172.21.15.7:0/1790773362 learned_addr learned my addr 172.21.15.7:0/1790773362 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053b7fe640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c159640 0x7f053c1f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f0533fff640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c0c52e0 0x7f053c0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 -- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c0c52e0 msgr2=0x7f053c0c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c0c52e0 0x7f053c0c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 -- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c159640 msgr2=0x7f053c1f6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c159640 0x7f053c1f6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 -- 172.21.15.7:0/1790773362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c072900 con 0x7f05280039b0 2023-11-28T18:50:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053b7fe640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c159640 0x7f053c1f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:50:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f053affd640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f053c159e20 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f052c001940 tx=0x7f0528005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f0538ff9640 1 -- 172.21.15.7:0/1790773362 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0b4070 con 0x7f05280039b0 2023-11-28T18:50:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f0538ff9640 1 -- 172.21.15.7:0/1790773362 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c005e30 con 0x7f05280039b0 2023-11-28T18:50:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c072be0 con 0x7f05280039b0 2023-11-28T18:50:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f0538ff9640 1 -- 172.21.15.7:0/1790773362 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c001680 con 0x7f05280039b0 2023-11-28T18:50:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c158dc0 con 0x7f05280039b0 2023-11-28T18:50:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.174+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c04e890 con 0x7f05280039b0 2023-11-28T18:50:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.178+0000 7f0538ff9640 1 -- 172.21.15.7:0/1790773362 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f052c007040 con 0x7f05280039b0 2023-11-28T18:50:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.178+0000 7f0538ff9640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c066900 0x7f051c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.178+0000 7f0538ff9640 1 -- 172.21.15.7:0/1790773362 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f052c11e0f0 con 0x7f05280039b0 2023-11-28T18:50:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.178+0000 7f0533fff640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c066900 0x7f051c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:20.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.178+0000 7f0533fff640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c066900 0x7f051c068dc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f053c157180 tx=0x7f0524008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.182+0000 7f0538ff9640 1 -- 172.21.15.7:0/1790773362 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f052c0ee950 con 0x7f05280039b0 2023-11-28T18:50:20.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.374+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f053c1555b0 con 0x7f05280039b0 2023-11-28T18:50:20.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.374+0000 7f0538ff9640 1 -- 172.21.15.7:0/1790773362 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f052c0ee4e0 con 0x7f05280039b0 2023-11-28T18:50:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:20.375 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:20.376 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c066900 msgr2=0x7f051c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c066900 0x7f051c068dc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f053c157180 tx=0x7f0524008040 comp rx=0 tx=0).stop 2023-11-28T18:50:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 msgr2=0x7f053c159e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f053c159e20 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f052c001940 tx=0x7f0528005800 comp rx=0 tx=0).stop 2023-11-28T18:50:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 shutdown_connections 2023-11-28T18:50:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f051c066900 0x7f051c068dc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f053c159640 0x7f053c1f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f053c0c52e0 0x7f053c0c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 --2- 172.21.15.7:0/1790773362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05280039b0 0x7f053c159e20 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:20.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 >> 172.21.15.7:0/1790773362 conn(0x7f053c0662e0 msgr2=0x7f053c14b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:20.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 shutdown_connections 2023-11-28T18:50:20.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:20.378+0000 7f054185f640 1 -- 172.21.15.7:0/1790773362 wait complete. 2023-11-28T18:50:20.398 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:22.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~303s 2023-11-28T18:50:22.083 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-28T18:50:22.083 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:50:22.083 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T18:50:22.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:50:22.259 INFO:tasks.workunit.client.0.smithi007.stderr:rm: cannot remove 'tmp/meta': Software caused connection abort 2023-11-28T18:50:22.259 INFO:tasks.workunit.client.0.smithi007.stderr:rm: cannot remove 'ffsb': Transport endpoint is not connected 2023-11-28T18:50:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:50:22.261 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2023-11-28T18:50:22.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-28T18:50:22.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.306+0000 7fa23bd3e640 1 Processor -- start 2023-11-28T18:50:22.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.306+0000 7fa23bd3e640 1 -- start start 2023-11-28T18:50:22.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.306+0000 7fa23bd3e640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa2341641b0 0x7fa2340c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa23bd3e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2340c75a0 con 0x7fa23416ab90 2023-11-28T18:50:22.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa23bd3e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa234167020 con 0x7fa234164600 2023-11-28T18:50:22.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa23bd3e640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa23416a140 con 0x7fa2341641b0 2023-11-28T18:50:22.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa239ada640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa23416ab90 0x7fa2340c2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47714/0 (socket says 172.21.15.7:47714) 2023-11-28T18:50:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa239ada640 1 -- 172.21.15.7:0/4031672218 learned_addr learned my addr 172.21.15.7:0/4031672218 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa2392d9640 1 --2- 172.21.15.7:0/4031672218 >> v2:172.21.15.193:3300/0 conn(0x7fa2341641b0 0x7fa2340c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa238ad8640 1 -- 172.21.15.7:0/4031672218 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993295957 0 0) 0x7fa2340c75a0 con 0x7fa23416ab90 2023-11-28T18:50:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa238ad8640 1 -- 172.21.15.7:0/4031672218 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa220004640 con 0x7fa23416ab90 2023-11-28T18:50:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa2392d9640 1 -- 172.21.15.7:0/4031672218 >> v1:172.21.15.193:6789/0 conn(0x7fa234164600 legacy=0x7fa234171470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa238ad8640 1 -- 172.21.15.7:0/4031672218 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461188601 0 0) 0x7fa22c0014a0 con 0x7fa234164600 2023-11-28T18:50:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa2392d9640 1 -- 172.21.15.7:0/4031672218 >> v1:172.21.15.7:6789/0 conn(0x7fa23416ab90 legacy=0x7fa2340c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.310+0000 7fa2392d9640 1 -- 172.21.15.7:0/4031672218 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2340c8490 con 0x7fa2341641b0 2023-11-28T18:50:22.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa2392d9640 1 --2- 172.21.15.7:0/4031672218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 0x7fa2340c6660 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa224009fd0 tx=0x7fa224020e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84be3444b3bc7d01 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa238ad8640 1 -- 172.21.15.7:0/4031672218 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240aab80 con 0x7fa2341641b0 2023-11-28T18:50:22.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa238ad8640 1 -- 172.21.15.7:0/4031672218 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2240aace0 con 0x7fa2341641b0 2023-11-28T18:50:22.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa238ad8640 1 -- 172.21.15.7:0/4031672218 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240b2660 con 0x7fa2341641b0 2023-11-28T18:50:22.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa23bd3e640 1 -- 172.21.15.7:0/4031672218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 msgr2=0x7fa2340c6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa23bd3e640 1 --2- 172.21.15.7:0/4031672218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 0x7fa2340c6660 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa224009fd0 tx=0x7fa224020e80 comp rx=0 tx=0).stop 2023-11-28T18:50:22.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa23bd3e640 1 -- 172.21.15.7:0/4031672218 shutdown_connections 2023-11-28T18:50:22.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa23bd3e640 1 --2- 172.21.15.7:0/4031672218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 0x7fa2340c6660 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa23bd3e640 1 -- 172.21.15.7:0/4031672218 >> 172.21.15.7:0/4031672218 conn(0x7fa2340662e0 msgr2=0x7fa2340666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:22.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa23bd3e640 1 -- 172.21.15.7:0/4031672218 shutdown_connections 2023-11-28T18:50:22.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.314+0000 7fa23bd3e640 1 -- 172.21.15.7:0/4031672218 wait complete. 2023-11-28T18:50:22.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 Processor -- start 2023-11-28T18:50:22.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 -- start start 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 0x7fa23415e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa234164600 0x7fa234164d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa23416ab90 0x7fa234203c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234072c30 con 0x7fa234164600 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2341604d0 con 0x7fa2341641b0 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23415ee60 con 0x7fa23416ab90 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa234164600 0x7fa234164d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa234164600 0x7fa234164d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41230/0 (socket says 172.21.15.7:41230) 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 -- 172.21.15.7:0/2623820571 learned_addr learned my addr 172.21.15.7:0/2623820571 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa239ada640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 0x7fa23415e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 -- 172.21.15.7:0/2623820571 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa23416ab90 msgr2=0x7fa234203c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa23416ab90 0x7fa234203c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 -- 172.21.15.7:0/2623820571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 msgr2=0x7fa23415e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 0x7fa23415e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 -- 172.21.15.7:0/2623820571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa224009cb0 con 0x7fa234164600 2023-11-28T18:50:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa239ada640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 0x7fa23415e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T18:50:22.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa2392d9640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa234164600 0x7fa234164d10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa2240091a0 tx=0x7fa22401dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa22a7fc640 1 -- 172.21.15.7:0/2623820571 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240b2d70 con 0x7fa234164600 2023-11-28T18:50:22.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa22a7fc640 1 -- 172.21.15.7:0/2623820571 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa224006e20 con 0x7fa234164600 2023-11-28T18:50:22.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa22a7fc640 1 -- 172.21.15.7:0/2623820571 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240bb690 con 0x7fa234164600 2023-11-28T18:50:22.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2341606e0 con 0x7fa234164600 2023-11-28T18:50:22.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.318+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa234160ca0 con 0x7fa234164600 2023-11-28T18:50:22.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.322+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23404e890 con 0x7fa234164600 2023-11-28T18:50:22.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.322+0000 7fa22a7fc640 1 -- 172.21.15.7:0/2623820571 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa2240bb7f0 con 0x7fa234164600 2023-11-28T18:50:22.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.322+0000 7fa22a7fc640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa2180668d0 0x7fa218068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.322+0000 7fa22a7fc640 1 -- 172.21.15.7:0/2623820571 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fa224122a70 con 0x7fa234164600 2023-11-28T18:50:22.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.322+0000 7fa239ada640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa2180668d0 0x7fa218068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.322+0000 7fa239ada640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa2180668d0 0x7fa218068d90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa230005890 tx=0x7fa230009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.326+0000 7fa22a7fc640 1 -- 172.21.15.7:0/2623820571 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa2240ef700 con 0x7fa234164600 2023-11-28T18:50:22.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:22.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.486+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa234066be0 con 0x7fa234164600 2023-11-28T18:50:22.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.486+0000 7fa22a7fc640 1 -- 172.21.15.7:0/2623820571 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+18253 (secure 0 0 0) 0x7fa2240f2d20 con 0x7fa234164600 2023-11-28T18:50:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:22.488 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":34,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:45:32.502309+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6834/695953635":"2023-11-29T18:45:32.502253+0000","172.21.15.193:6832/2303207996":"2023-11-29T18:43:32.460743+0000","172.21.15.193:6837/1742899523":"2023-11-29T18:42:07.410735+0000","172.21.15.7:6835/1029915019":"2023-11-29T18:40:17.364225+0000","172.21.15.7:6834/1029915019":"2023-11-29T18:40:17.364225+0000","172.21.15.193:6835/650087483":"2023-11-29T18:38:37.328502+0000","172.21.15.193:6832/650087483":"2023-11-29T18:38:37.328502+0000","172.21.15.7:6838/2690556675":"2023-11-29T18:32:52.248615+0000","172.21.15.193:6834/1742899523":"2023-11-29T18:42:07.410735+0000","172.21.15.193:6832/4143265275":"2023-11-29T18:28:12.173848+0000","172.21.15.193:6834/3911527535":"2023-11-29T18:30:47.207685+0000","172.21.15.7:6839/1819203827":"2023-11-29T18:35:07.279902+0000","172.21.15.7:6837/1819203827":"2023-11-29T18:35:07.279902+0000","172.21.15.193:6835/2303207996":"2023-11-29T18:43:32.460743+0000","172.21.15.193:6836/1685294311":"2023-11-29T18:36:52.304067+0000","172.21.15.7:6834/1419029356":"2023-11-29T18:30:02.197504+0000","172.21.15.7:6835/695953635":"2023-11-29T18:45:32.502253+0000","172.21.15.7:6835/1419029356":"2023-11-29T18:30:02.197504+0000","172.21.15.7:6836/2690556675":"2023-11-29T18:32:52.248615+0000","172.21.15.193:6833/1372595381":"2023-11-29T18:25:37.139777+0000","172.21.15.193:6837/3911527535":"2023-11-29T18:30:47.207685+0000","172.21.15.193:6836/1372595381":"2023-11-29T18:25:37.139777+0000","172.21.15.193:6835/4143265275":"2023-11-29T18:28:12.173848+0000","172.21.15.193:6833/1685294311":"2023-11-29T18:36:52.304067+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:50:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa2180668d0 msgr2=0x7fa218068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa2180668d0 0x7fa218068d90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa230005890 tx=0x7fa230009040 comp rx=0 tx=0).stop 2023-11-28T18:50:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa234164600 msgr2=0x7fa234164d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa234164600 0x7fa234164d10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa2240091a0 tx=0x7fa22401dc20 comp rx=0 tx=0).stop 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 shutdown_connections 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fa2180668d0 0x7fa218068d90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa23416ab90 0x7fa234203c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa234164600 0x7fa234164d10 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 --2- 172.21.15.7:0/2623820571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2341641b0 0x7fa23415e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 >> 172.21.15.7:0/2623820571 conn(0x7fa2340662e0 msgr2=0x7fa234161dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 shutdown_connections 2023-11-28T18:50:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.490+0000 7fa23bd3e640 1 -- 172.21.15.7:0/2623820571 wait complete. 2023-11-28T18:50:22.514 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T18:23:05.706694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T18:23:07.048910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:50:22.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:50:22.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b9d9d640 1 Processor -- start 2023-11-28T18:50:22.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b9d9d640 1 -- start start 2023-11-28T18:50:22.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b9d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f52b4072950 0x7f52b4074030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b9d9d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b40c8490 con 0x7f52b4073070 2023-11-28T18:50:22.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b9d9d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b40c81f0 con 0x7f52b406c920 2023-11-28T18:50:22.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b9d9d640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b415ab00 con 0x7f52b4072950 2023-11-28T18:50:22.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b2ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f52b4073070 0x7f52b40c19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47716/0 (socket says 172.21.15.7:47716) 2023-11-28T18:50:22.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b2ffd640 1 -- 172.21.15.7:0/60718331 learned_addr learned my addr 172.21.15.7:0/60718331 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:22.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b37fe640 1 --2- 172.21.15.7:0/60718331 >> v2:172.21.15.193:3300/0 conn(0x7f52b4072950 0x7f52b4074030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.546+0000 7f52b27fc640 1 -- 172.21.15.7:0/60718331 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996646090 0 0) 0x7f52b40c8490 con 0x7f52b4073070 2023-11-28T18:50:22.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.550+0000 7f52b27fc640 1 -- 172.21.15.7:0/60718331 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f529c003530 con 0x7f52b4073070 2023-11-28T18:50:22.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.550+0000 7f52b27fc640 1 -- 172.21.15.7:0/60718331 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368076088 0 0) 0x7f52b40c81f0 con 0x7f52b406c920 2023-11-28T18:50:22.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.550+0000 7f52b27fc640 1 -- 172.21.15.7:0/60718331 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b40c8490 con 0x7f52b406c920 2023-11-28T18:50:22.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.550+0000 7f52b37fe640 1 -- 172.21.15.7:0/60718331 >> v1:172.21.15.193:6789/0 conn(0x7f52b406c920 legacy=0x7f52b41647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.550+0000 7f52b37fe640 1 -- 172.21.15.7:0/60718331 >> v1:172.21.15.7:6789/0 conn(0x7f52b4073070 legacy=0x7f52b40c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.550+0000 7f52b27fc640 1 -- 172.21.15.7:0/60718331 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537153139 0 0) 0x7f529c003530 con 0x7f52b4073070 2023-11-28T18:50:22.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.550+0000 7f52b37fe640 1 -- 172.21.15.7:0/60718331 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b415ade0 con 0x7f52b4072950 2023-11-28T18:50:22.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b37fe640 1 --2- 172.21.15.7:0/60718331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b4072950 0x7f52b4074030 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f52a40097c0 tx=0x7f52a401c280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b49b182b1235c3e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b27fc640 1 -- 172.21.15.7:0/60718331 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8001680 con 0x7f52b4072950 2023-11-28T18:50:22.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b27fc640 1 -- 172.21.15.7:0/60718331 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a40a7050 con 0x7f52b4072950 2023-11-28T18:50:22.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b27fc640 1 -- 172.21.15.7:0/60718331 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40ab620 con 0x7f52b4072950 2023-11-28T18:50:22.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b9d9d640 1 -- 172.21.15.7:0/60718331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b4072950 msgr2=0x7f52b4074030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b9d9d640 1 --2- 172.21.15.7:0/60718331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b4072950 0x7f52b4074030 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f52a40097c0 tx=0x7f52a401c280 comp rx=0 tx=0).stop 2023-11-28T18:50:22.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b9d9d640 1 -- 172.21.15.7:0/60718331 shutdown_connections 2023-11-28T18:50:22.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b9d9d640 1 --2- 172.21.15.7:0/60718331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b4072950 0x7f52b4074030 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b9d9d640 1 -- 172.21.15.7:0/60718331 >> 172.21.15.7:0/60718331 conn(0x7f52b40cbf80 msgr2=0x7f52b40cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:22.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b9d9d640 1 -- 172.21.15.7:0/60718331 shutdown_connections 2023-11-28T18:50:22.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.554+0000 7f52b9d9d640 1 -- 172.21.15.7:0/60718331 wait complete. 2023-11-28T18:50:22.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b9d9d640 1 Processor -- start 2023-11-28T18:50:22.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b9d9d640 1 -- start start 2023-11-28T18:50:22.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b9d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b406c920 0x7f52b40b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b9d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52b4072950 0x7f52b40c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b9d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b4073070 0x7f52b415c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b9d9d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b40b0550 con 0x7f52b4073070 2023-11-28T18:50:22.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b9d9d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b415d820 con 0x7f52b406c920 2023-11-28T18:50:22.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b9d9d640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b40caa10 con 0x7f52b4072950 2023-11-28T18:50:22.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b4073070 0x7f52b415c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b4073070 0x7f52b415c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41236/0 (socket says 172.21.15.7:41236) 2023-11-28T18:50:22.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b3fff640 1 -- 172.21.15.7:0/2797729570 learned_addr learned my addr 172.21.15.7:0/2797729570 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:22.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b37fe640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b406c920 0x7f52b40b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.558+0000 7f52b2ffd640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52b4072950 0x7f52b40c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b3fff640 1 -- 172.21.15.7:0/2797729570 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52b4072950 msgr2=0x7f52b40c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b3fff640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52b4072950 0x7f52b40c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b3fff640 1 -- 172.21.15.7:0/2797729570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b406c920 msgr2=0x7f52b40b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b3fff640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b406c920 0x7f52b40b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b3fff640 1 -- 172.21.15.7:0/2797729570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a4009420 con 0x7f52b4073070 2023-11-28T18:50:22.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b3fff640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b4073070 0x7f52b415c440 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c790 tx=0x7f52ac00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f529bfff640 1 -- 172.21.15.7:0/2797729570 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac00cea0 con 0x7f52b4073070 2023-11-28T18:50:22.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f529bfff640 1 -- 172.21.15.7:0/2797729570 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac003b80 con 0x7f52b4073070 2023-11-28T18:50:22.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f529bfff640 1 -- 172.21.15.7:0/2797729570 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac00a850 con 0x7f52b4073070 2023-11-28T18:50:22.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b414afd0 con 0x7f52b4073070 2023-11-28T18:50:22.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b40b1530 con 0x7f52b4073070 2023-11-28T18:50:22.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.562+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b415d9d0 con 0x7f52b4073070 2023-11-28T18:50:22.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.566+0000 7f529bfff640 1 -- 172.21.15.7:0/2797729570 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f52ac00a9b0 con 0x7f52b4073070 2023-11-28T18:50:22.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.566+0000 7f529bfff640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52a00668d0 0x7f52a0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.566+0000 7f529bfff640 1 -- 172.21.15.7:0/2797729570 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f52ac084760 con 0x7f52b4073070 2023-11-28T18:50:22.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.566+0000 7f52b37fe640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52a00668d0 0x7f52a0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.566+0000 7f52b37fe640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52a00668d0 0x7f52a0068d90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f52a40ad000 tx=0x7f52a40a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.570+0000 7f529bfff640 1 -- 172.21.15.7:0/2797729570 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f52b415d9d0 con 0x7f52b4073070 2023-11-28T18:50:22.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.670+0000 7f127e6c9640 1 Processor -- start 2023-11-28T18:50:22.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.670+0000 7f127e6c9640 1 -- start start 2023-11-28T18:50:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f127e6c9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1278154130 0x7f1278159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f127e6c9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12780caeb0 con 0x7f12780ca530 2023-11-28T18:50:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f127e6c9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127815d9b0 con 0x7f12780c4c80 2023-11-28T18:50:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f127e6c9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127815a3f0 con 0x7f1278154130 2023-11-28T18:50:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f1277fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1278154130 0x7f1278159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f127cc66640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f12780ca530 0x7f12780cb2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47732/0 (socket says 172.21.15.7:47732) 2023-11-28T18:50:22.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f127cc66640 1 -- 172.21.15.7:0/1418303056 learned_addr learned my addr 172.21.15.7:0/1418303056 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:22.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f1277fff640 1 -- 172.21.15.7:0/1418303056 >> v1:172.21.15.193:6789/0 conn(0x7f12780c4c80 legacy=0x7f127815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f1277fff640 1 -- 172.21.15.7:0/1418303056 >> v1:172.21.15.7:6789/0 conn(0x7f12780ca530 legacy=0x7f12780cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.674+0000 7f1277fff640 1 -- 172.21.15.7:0/1418303056 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127815a5d0 con 0x7f1278154130 2023-11-28T18:50:22.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f1277fff640 1 --2- 172.21.15.7:0/1418303056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f1278159eb0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f12680097c0 tx=0x7f126801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7679785a692bc6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f1276ffd640 1 -- 172.21.15.7:0/1418303056 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680a8970 con 0x7f1278154130 2023-11-28T18:50:22.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f1276ffd640 1 -- 172.21.15.7:0/1418303056 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12680a8ad0 con 0x7f1278154130 2023-11-28T18:50:22.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f1276ffd640 1 -- 172.21.15.7:0/1418303056 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680a8e60 con 0x7f1278154130 2023-11-28T18:50:22.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f127e6c9640 1 -- 172.21.15.7:0/1418303056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 msgr2=0x7f1278159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f127e6c9640 1 --2- 172.21.15.7:0/1418303056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f1278159eb0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f12680097c0 tx=0x7f126801ec70 comp rx=0 tx=0).stop 2023-11-28T18:50:22.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f127e6c9640 1 -- 172.21.15.7:0/1418303056 shutdown_connections 2023-11-28T18:50:22.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f127e6c9640 1 --2- 172.21.15.7:0/1418303056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f1278159eb0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f127e6c9640 1 -- 172.21.15.7:0/1418303056 >> 172.21.15.7:0/1418303056 conn(0x7f1278073ca0 msgr2=0x7f1278071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:22.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f127e6c9640 1 -- 172.21.15.7:0/1418303056 shutdown_connections 2023-11-28T18:50:22.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.678+0000 7f127e6c9640 1 -- 172.21.15.7:0/1418303056 wait complete. 2023-11-28T18:50:22.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 Processor -- start 2023-11-28T18:50:22.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 -- start start 2023-11-28T18:50:22.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12780c4c80 0x7f127814bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12780ca530 0x7f12781e6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f127814cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127814eba0 con 0x7f1278154130 2023-11-28T18:50:22.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12780cf760 con 0x7f12780c4c80 2023-11-28T18:50:22.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12780cfa00 con 0x7f12780ca530 2023-11-28T18:50:22.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f127814cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f127814cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41254/0 (socket says 172.21.15.7:41254) 2023-11-28T18:50:22.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 -- 172.21.15.7:0/140549205 learned_addr learned my addr 172.21.15.7:0/140549205 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:22.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f12777fe640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12780ca530 0x7f12781e6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f1277fff640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12780c4c80 0x7f127814bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 -- 172.21.15.7:0/140549205 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12780ca530 msgr2=0x7f12781e6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12780ca530 0x7f12781e6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 -- 172.21.15.7:0/140549205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12780c4c80 msgr2=0x7f127814bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12780c4c80 0x7f127814bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 -- 172.21.15.7:0/140549205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268009420 con 0x7f1278154130 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127cc66640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f127814cbb0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f127000cbc0 tx=0x7f127000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f1274ff9640 1 -- 172.21.15.7:0/140549205 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127000ad40 con 0x7f1278154130 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f1274ff9640 1 -- 172.21.15.7:0/140549205 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1270012400 con 0x7f1278154130 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f1274ff9640 1 -- 172.21.15.7:0/140549205 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127000aea0 con 0x7f1278154130 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127814c000 con 0x7f1278154130 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12780cf490 con 0x7f1278154130 2023-11-28T18:50:22.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.682+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12781523e0 con 0x7f1278154130 2023-11-28T18:50:22.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.686+0000 7f1274ff9640 1 -- 172.21.15.7:0/140549205 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1270012790 con 0x7f1278154130 2023-11-28T18:50:22.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.686+0000 7f1274ff9640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1264066900 0x7f1264068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:22.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.686+0000 7f1274ff9640 1 -- 172.21.15.7:0/140549205 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f1270084d70 con 0x7f1278154130 2023-11-28T18:50:22.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.686+0000 7f1277fff640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1264066900 0x7f1264068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:22.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.686+0000 7f1277fff640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1264066900 0x7f1264068dc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f12680b2000 tx=0x7f1268008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:22.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.690+0000 7f1274ff9640 1 -- 172.21.15.7:0/140549205 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1270051b40 con 0x7f1278154130 2023-11-28T18:50:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.774+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52b415d990 con 0x7f52b4073070 2023-11-28T18:50:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.774+0000 7f529bfff640 1 -- 172.21.15.7:0/2797729570 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f52ac054fb0 con 0x7f52b4073070 2023-11-28T18:50:22.776 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:22.776 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52a00668d0 msgr2=0x7f52a0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52a00668d0 0x7f52a0068d90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f52a40ad000 tx=0x7f52a40a6000 comp rx=0 tx=0).stop 2023-11-28T18:50:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b4073070 msgr2=0x7f52b415c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b4073070 0x7f52b415c440 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c790 tx=0x7f52ac00cc50 comp rx=0 tx=0).stop 2023-11-28T18:50:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 shutdown_connections 2023-11-28T18:50:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f52a00668d0 0x7f52a0068d90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b4073070 0x7f52b415c440 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52b4072950 0x7f52b40c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 --2- 172.21.15.7:0/2797729570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52b406c920 0x7f52b40b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 >> 172.21.15.7:0/2797729570 conn(0x7f52b40cbf80 msgr2=0x7f52b40c78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 shutdown_connections 2023-11-28T18:50:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.778+0000 7f52b9d9d640 1 -- 172.21.15.7:0/2797729570 wait complete. 2023-11-28T18:50:22.801 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-11-28T18:23:08.041625+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5559": { "addr": "172.21.15.7:6838/4097424196", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6836", "nonce": 4097424196, "type": "v2" }, { "addr": "172.21.15.7:6838", "nonce": 4097424196, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5559, "incarnation": 54, "join_fscid": -1, "laggy_since": "2023-11-28T18:45:57.508800+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_5700": { "addr": "172.21.15.7:6839/4048397809", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6837", "nonce": 4048397809, "type": "v2" }, { "addr": "172.21.15.7:6839", "nonce": 4048397809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5700, "incarnation": 60, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 29 }, "gid_5714": { "addr": "172.21.15.193:6835/1637641802", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6832", "nonce": 1637641802, "type": "v2" }, { "addr": "172.21.15.193:6835", "nonce": 1637641802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5714, "incarnation": 102, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:rejoin", "state_seq": 29 }, "gid_5734": { "addr": "172.21.15.193:6836/1268741263", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6833", "nonce": 1268741263, "type": "v2" }, { "addr": "172.21.15.193:6836", "nonce": 1268741263, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5734, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 29 }, "gid_6049": { "addr": "172.21.15.193:6837/1591261140", "addrs": { "addrvec": [ { "addr": "172.21.15.193:6834", "nonce": 1591261140, "type": "v2" }, { "addr": "172.21.15.193:6837", "nonce": 1591261140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6049, "incarnation": 93, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-11-28T18:46:39.535710+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5559, "mds_1": 5714, "mds_2": 6049, "mds_3": 5734, "mds_4": 5700 } } } ], "standbys": [] } 2023-11-28T18:50:22.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f127814c8d0 con 0x7f1278154130 2023-11-28T18:50:22.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f1274ff9640 1 -- 172.21.15.7:0/140549205 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f127814c8d0 con 0x7f1278154130 2023-11-28T18:50:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1264066900 msgr2=0x7f1264068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1264066900 0x7f1264068dc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f12680b2000 tx=0x7f1268008040 comp rx=0 tx=0).stop 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 msgr2=0x7f127814cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f127814cbb0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f127000cbc0 tx=0x7f127000a430 comp rx=0 tx=0).stop 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 shutdown_connections 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f1264066900 0x7f1264068dc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1278154130 0x7f127814cbb0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f12780ca530 0x7f12781e6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 --2- 172.21.15.7:0/140549205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12780c4c80 0x7f127814bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:22.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 >> 172.21.15.7:0/140549205 conn(0x7f1278073ca0 msgr2=0x7f127815de40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:22.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 shutdown_connections 2023-11-28T18:50:22.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:22.854+0000 7f127e6c9640 1 -- 172.21.15.7:0/140549205 wait complete. 2023-11-28T18:50:22.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T18:50:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31cf846640 1 Processor -- start 2023-11-28T18:50:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31cf846640 1 -- start start 2023-11-28T18:50:23.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31cf846640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f31c80c4c80 0x7f31c8159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31cf846640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c80caeb0 con 0x7f31c8154130 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31cf846640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c815d9b0 con 0x7f31c80ca530 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31cf846640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c815a3f0 con 0x7f31c80c4c80 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31cd5e2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f31c8154130 0x7f31c815de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47738/0 (socket says 172.21.15.7:47738) 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31cd5e2640 1 -- 172.21.15.7:0/3332601482 learned_addr learned my addr 172.21.15.7:0/3332601482 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31ccde1640 1 --2- 172.21.15.7:0/3332601482 >> v2:172.21.15.193:3300/0 conn(0x7f31c80c4c80 0x7f31c8159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31bffff640 1 -- 172.21.15.7:0/3332601482 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819968098 0 0) 0x7f31c80caeb0 con 0x7f31c8154130 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31bffff640 1 -- 172.21.15.7:0/3332601482 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b4003530 con 0x7f31c8154130 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31bffff640 1 -- 172.21.15.7:0/3332601482 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058629066 0 0) 0x7f31c815d9b0 con 0x7f31c80ca530 2023-11-28T18:50:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31bffff640 1 -- 172.21.15.7:0/3332601482 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c80caeb0 con 0x7f31c80ca530 2023-11-28T18:50:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31ccde1640 1 -- 172.21.15.7:0/3332601482 >> v1:172.21.15.193:6789/0 conn(0x7f31c80ca530 legacy=0x7f31c80cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31ccde1640 1 -- 172.21.15.7:0/3332601482 >> v1:172.21.15.7:6789/0 conn(0x7f31c8154130 legacy=0x7f31c815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.026+0000 7f31ccde1640 1 -- 172.21.15.7:0/3332601482 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c815a5d0 con 0x7f31c80c4c80 2023-11-28T18:50:23.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31ccde1640 1 --2- 172.21.15.7:0/3332601482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 0x7f31c8159eb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f31b80097c0 tx=0x7f31b801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e42ec0a9e2353e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31bffff640 1 -- 172.21.15.7:0/3332601482 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80a8970 con 0x7f31c80c4c80 2023-11-28T18:50:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31bffff640 1 -- 172.21.15.7:0/3332601482 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b80a8ad0 con 0x7f31c80c4c80 2023-11-28T18:50:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31bffff640 1 -- 172.21.15.7:0/3332601482 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80a8e60 con 0x7f31c80c4c80 2023-11-28T18:50:23.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31cf846640 1 -- 172.21.15.7:0/3332601482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 msgr2=0x7f31c8159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31cf846640 1 --2- 172.21.15.7:0/3332601482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 0x7f31c8159eb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f31b80097c0 tx=0x7f31b801ec70 comp rx=0 tx=0).stop 2023-11-28T18:50:23.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31cf846640 1 -- 172.21.15.7:0/3332601482 shutdown_connections 2023-11-28T18:50:23.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31cf846640 1 --2- 172.21.15.7:0/3332601482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 0x7f31c8159eb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31cf846640 1 -- 172.21.15.7:0/3332601482 >> 172.21.15.7:0/3332601482 conn(0x7f31c8073ca0 msgr2=0x7f31c8071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:23.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31cf846640 1 -- 172.21.15.7:0/3332601482 shutdown_connections 2023-11-28T18:50:23.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.030+0000 7f31cf846640 1 -- 172.21.15.7:0/3332601482 wait complete. 2023-11-28T18:50:23.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 Processor -- start 2023-11-28T18:50:23.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 -- start start 2023-11-28T18:50:23.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 0x7f31c80b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31c80ca530 0x7f31c8146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31c8154130 0x7f31c8146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c80b4700 con 0x7f31c8154130 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c80b3810 con 0x7f31c80c4c80 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c80b3af0 con 0x7f31c80ca530 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31c8154130 0x7f31c8146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31c8154130 0x7f31c8146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41262/0 (socket says 172.21.15.7:41262) 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 -- 172.21.15.7:0/2620306717 learned_addr learned my addr 172.21.15.7:0/2620306717 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31ccde1640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31c80ca530 0x7f31c8146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cd5e2640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 0x7f31c80b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 -- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31c80ca530 msgr2=0x7f31c8146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31c80ca530 0x7f31c8146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 -- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 msgr2=0x7f31c80b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 0x7f31c80b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 -- 172.21.15.7:0/2620306717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b8009420 con 0x7f31c8154130 2023-11-28T18:50:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cd5e2640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 0x7f31c80b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:50:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cdde3640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31c8154130 0x7f31c8146680 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f31c000c790 tx=0x7f31c000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31bdffb640 1 -- 172.21.15.7:0/2620306717 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c000cea0 con 0x7f31c8154130 2023-11-28T18:50:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31bdffb640 1 -- 172.21.15.7:0/2620306717 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c0003b80 con 0x7f31c8154130 2023-11-28T18:50:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c80b3dd0 con 0x7f31c8154130 2023-11-28T18:50:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31bdffb640 1 -- 172.21.15.7:0/2620306717 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c000a850 con 0x7f31c8154130 2023-11-28T18:50:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.034+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c80b4390 con 0x7f31c8154130 2023-11-28T18:50:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.038+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c80b4870 con 0x7f31c8154130 2023-11-28T18:50:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.038+0000 7f31bdffb640 1 -- 172.21.15.7:0/2620306717 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f31c000a9b0 con 0x7f31c8154130 2023-11-28T18:50:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.038+0000 7f31bdffb640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31ac0668d0 0x7f31ac068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.038+0000 7f31bdffb640 1 -- 172.21.15.7:0/2620306717 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f31c00846f0 con 0x7f31c8154130 2023-11-28T18:50:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.038+0000 7f31cd5e2640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31ac0668d0 0x7f31ac068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.038+0000 7f31cd5e2640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31ac0668d0 0x7f31ac068d90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f31c40011f0 tx=0x7f31c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.042+0000 7f31bdffb640 1 -- 172.21.15.7:0/2620306717 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f31c00514a0 con 0x7f31c8154130 2023-11-28T18:50:23.045 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-11-28T18:50:23.135 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d9d491af047e42f68657ad2fc302775b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-11-28T18:50:23.138 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-11-28T18:50:23.149 INFO:tasks.mds_thrash:joining mds_thrasher 2023-11-28T18:50:23.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.198+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f31c804e890 con 0x7f31c8154130 2023-11-28T18:50:23.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.198+0000 7f31bdffb640 1 -- 172.21.15.7:0/2620306717 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f31c0054ac0 con 0x7f31c8154130 2023-11-28T18:50:23.199 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-11-28T18:50:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31ac0668d0 msgr2=0x7f31ac068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31ac0668d0 0x7f31ac068d90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f31c40011f0 tx=0x7f31c4009040 comp rx=0 tx=0).stop 2023-11-28T18:50:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31c8154130 msgr2=0x7f31c8146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31c8154130 0x7f31c8146680 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f31c000c790 tx=0x7f31c000cc50 comp rx=0 tx=0).stop 2023-11-28T18:50:23.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 shutdown_connections 2023-11-28T18:50:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f31ac0668d0 0x7f31ac068d90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31c8154130 0x7f31c8146680 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f31c80ca530 0x7f31c8146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 --2- 172.21.15.7:0/2620306717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31c80c4c80 0x7f31c80b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 >> 172.21.15.7:0/2620306717 conn(0x7f31c8073ca0 msgr2=0x7f31c80c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 shutdown_connections 2023-11-28T18:50:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.202+0000 7f31cf846640 1 -- 172.21.15.7:0/2620306717 wait complete. 2023-11-28T18:50:23.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T18:50:23.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.366+0000 7f6b42d2a640 1 Processor -- start 2023-11-28T18:50:23.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.366+0000 7f6b42d2a640 1 -- start start 2023-11-28T18:50:23.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.366+0000 7f6b42d2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c141030 0x7f6b3c15be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.366+0000 7f6b42d2a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f6b3c06e3a0 0x7f6b3c0c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.366+0000 7f6b42d2a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b3c06e7c0 0x7f6b3c0c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.366+0000 7f6b42d2a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b3c15c390 con 0x7f6b3c06e7c0 2023-11-28T18:50:23.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.366+0000 7f6b42d2a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b3c157f60 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.366+0000 7f6b42d2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c158240 con 0x7f6b3c141030 2023-11-28T18:50:23.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b3c06e7c0 0x7f6b3c0c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b3c06e7c0 0x7f6b3c0c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41266/0 (socket says 172.21.15.7:41266) 2023-11-28T18:50:23.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 -- 172.21.15.7:0/3716558289 learned_addr learned my addr 172.21.15.7:0/3716558289 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:23.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b3bfff640 1 --2- 172.21.15.7:0/3716558289 >> v2:172.21.15.193:3300/0 conn(0x7f6b3c06e3a0 0x7f6b3c0c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b40ac6640 1 --2- 172.21.15.7:0/3716558289 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c141030 0x7f6b3c15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 -- 172.21.15.7:0/3716558289 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c141030 msgr2=0x7f6b3c15be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:23.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 --2- 172.21.15.7:0/3716558289 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c141030 0x7f6b3c15be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 -- 172.21.15.7:0/3716558289 >> v2:172.21.15.193:3300/0 conn(0x7f6b3c06e3a0 msgr2=0x7f6b3c0c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:23.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 --2- 172.21.15.7:0/3716558289 >> v2:172.21.15.193:3300/0 conn(0x7f6b3c06e3a0 0x7f6b3c0c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 -- 172.21.15.7:0/3716558289 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c158520 con 0x7f6b3c06e7c0 2023-11-28T18:50:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b412c7640 1 --2- 172.21.15.7:0/3716558289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e7c0 0x7f6b3c0c50d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c067000 tx=0x7f6b3401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec120d81fa2531dd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b3b7fe640 1 -- 172.21.15.7:0/3716558289 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b340af400 con 0x7f6b3c06e7c0 2023-11-28T18:50:23.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b3b7fe640 1 -- 172.21.15.7:0/3716558289 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b340ab050 con 0x7f6b3c06e7c0 2023-11-28T18:50:23.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.370+0000 7f6b3b7fe640 1 -- 172.21.15.7:0/3716558289 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b340af780 con 0x7f6b3c06e7c0 2023-11-28T18:50:23.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- 172.21.15.7:0/3716558289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e7c0 msgr2=0x7f6b3c0c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/3716558289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e7c0 0x7f6b3c0c50d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c067000 tx=0x7f6b3401f1c0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- 172.21.15.7:0/3716558289 shutdown_connections 2023-11-28T18:50:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/3716558289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e7c0 0x7f6b3c0c50d0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/3716558289 >> v2:172.21.15.193:3300/0 conn(0x7f6b3c06e3a0 0x7f6b3c0c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/3716558289 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c141030 0x7f6b3c15be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- 172.21.15.7:0/3716558289 >> 172.21.15.7:0/3716558289 conn(0x7f6b3c073f00 msgr2=0x7f6b3c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- 172.21.15.7:0/3716558289 shutdown_connections 2023-11-28T18:50:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- 172.21.15.7:0/3716558289 wait complete. 2023-11-28T18:50:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 Processor -- start 2023-11-28T18:50:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- start start 2023-11-28T18:50:23.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e3a0 0x7f6b3c13a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c06e7c0 0x7f6b3c13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c141030 0x7f6b3c0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c0cada0 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c0c9280 con 0x7f6b3c141030 2023-11-28T18:50:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b42d2a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c138140 con 0x7f6b3c06e7c0 2023-11-28T18:50:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b40ac6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e3a0 0x7f6b3c13a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b40ac6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e3a0 0x7f6b3c13a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41282/0 (socket says 172.21.15.7:41282) 2023-11-28T18:50:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b40ac6640 1 -- 172.21.15.7:0/2324407492 learned_addr learned my addr 172.21.15.7:0/2324407492 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b3bfff640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c06e7c0 0x7f6b3c13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b412c7640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c141030 0x7f6b3c0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b40ac6640 1 -- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c06e7c0 msgr2=0x7f6b3c13a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b40ac6640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c06e7c0 0x7f6b3c13a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b40ac6640 1 -- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c141030 msgr2=0x7f6b3c0cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b40ac6640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c141030 0x7f6b3c0cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b40ac6640 1 -- 172.21.15.7:0/2324407492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b34009b30 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b3bfff640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c06e7c0 0x7f6b3c13a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:50:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.374+0000 7f6b412c7640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c141030 0x7f6b3c0cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:50:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b40ac6640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e3a0 0x7f6b3c13a290 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000d5f0 tx=0x7f6b3000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b397fa640 1 -- 172.21.15.7:0/2324407492 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b300125f0 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b397fa640 1 -- 172.21.15.7:0/2324407492 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b30012750 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c13e890 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b397fa640 1 -- 172.21.15.7:0/2324407492 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30012ae0 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c138390 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c0c6920 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b397fa640 1 -- 172.21.15.7:0/2324407492 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6b30012c40 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b397fa640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b200668d0 0x7f6b20068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b397fa640 1 -- 172.21.15.7:0/2324407492 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f6b30084e60 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.378+0000 7f6b3bfff640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b200668d0 0x7f6b20068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.382+0000 7f6b3bfff640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b200668d0 0x7f6b20068d90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6b28004660 tx=0x7f6b28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.382+0000 7f6b397fa640 1 -- 172.21.15.7:0/2324407492 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6b30051b80 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6b3c04e890 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b397fa640 1 -- 172.21.15.7:0/2324407492 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6b30007380 con 0x7f6b3c06e3a0 2023-11-28T18:50:23.544 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-11-28T18:50:23.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b200668d0 msgr2=0x7f6b20068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b200668d0 0x7f6b20068d90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6b28004660 tx=0x7f6b28008040 comp rx=0 tx=0).stop 2023-11-28T18:50:23.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e3a0 msgr2=0x7f6b3c13a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e3a0 0x7f6b3c13a290 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000d5f0 tx=0x7f6b3000dab0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 shutdown_connections 2023-11-28T18:50:23.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f6b200668d0 0x7f6b20068d90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c141030 0x7f6b3c0cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6b3c06e7c0 0x7f6b3c13a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 --2- 172.21.15.7:0/2324407492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b3c06e3a0 0x7f6b3c13a290 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 >> 172.21.15.7:0/2324407492 conn(0x7f6b3c073f00 msgr2=0x7f6b3c0c7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:23.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 shutdown_connections 2023-11-28T18:50:23.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.542+0000 7f6b42d2a640 1 -- 172.21.15.7:0/2324407492 wait complete. 2023-11-28T18:50:23.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T18:50:23.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.706+0000 7fd37fc09640 1 Processor -- start 2023-11-28T18:50:23.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.706+0000 7fd37fc09640 1 -- start start 2023-11-28T18:50:23.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37fc09640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd3780c50b0 0x7fd3781584c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37fc09640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fd3780c5490 0x7fd3780c6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37fc09640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd378070630 con 0x7fd3780c6840 2023-11-28T18:50:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37fc09640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd378071cb0 con 0x7fd3780c50b0 2023-11-28T18:50:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37fc09640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd378070910 con 0x7fd3780c5490 2023-11-28T18:50:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37d9a5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd3780c50b0 0x7fd3781584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37e1a6640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd3780c6840 0x7fd3781516e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47988/0 (socket says 172.21.15.7:47988) 2023-11-28T18:50:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37e1a6640 1 -- 172.21.15.7:0/813495306 learned_addr learned my addr 172.21.15.7:0/813495306 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37d1a4640 1 --2- 172.21.15.7:0/813495306 >> v2:172.21.15.193:3300/0 conn(0x7fd3780c5490 0x7fd3780c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37d9a5640 1 -- 172.21.15.7:0/813495306 >> v2:172.21.15.193:3300/0 conn(0x7fd3780c5490 msgr2=0x7fd3780c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37d9a5640 1 --2- 172.21.15.7:0/813495306 >> v2:172.21.15.193:3300/0 conn(0x7fd3780c5490 0x7fd3780c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37d9a5640 1 -- 172.21.15.7:0/813495306 >> v1:172.21.15.7:6789/0 conn(0x7fd3780c6840 legacy=0x7fd3781516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37d9a5640 1 -- 172.21.15.7:0/813495306 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd378070bf0 con 0x7fd3780c50b0 2023-11-28T18:50:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.710+0000 7fd37c9a3640 1 -- 172.21.15.7:0/813495306 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906208614 0 0) 0x7fd378070630 con 0x7fd3780c6840 2023-11-28T18:50:23.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37d9a5640 1 --2- 172.21.15.7:0/813495306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c50b0 0x7fd3781584c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd37806ed40 tx=0x7fd37401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c833e1be6e0a83 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37c9a3640 1 -- 172.21.15.7:0/813495306 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3740a7070 con 0x7fd3780c50b0 2023-11-28T18:50:23.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37c9a3640 1 -- 172.21.15.7:0/813495306 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3740ab430 con 0x7fd3780c50b0 2023-11-28T18:50:23.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37c9a3640 1 -- 172.21.15.7:0/813495306 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3740ab7c0 con 0x7fd3780c50b0 2023-11-28T18:50:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 -- 172.21.15.7:0/813495306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c50b0 msgr2=0x7fd3781584c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 --2- 172.21.15.7:0/813495306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c50b0 0x7fd3781584c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd37806ed40 tx=0x7fd37401c430 comp rx=0 tx=0).stop 2023-11-28T18:50:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 -- 172.21.15.7:0/813495306 shutdown_connections 2023-11-28T18:50:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 --2- 172.21.15.7:0/813495306 >> v2:172.21.15.193:3300/0 conn(0x7fd3780c5490 0x7fd3780c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 --2- 172.21.15.7:0/813495306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c50b0 0x7fd3781584c0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 -- 172.21.15.7:0/813495306 >> 172.21.15.7:0/813495306 conn(0x7fd3780662e0 msgr2=0x7fd3780666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 -- 172.21.15.7:0/813495306 shutdown_connections 2023-11-28T18:50:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 -- 172.21.15.7:0/813495306 wait complete. 2023-11-28T18:50:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 Processor -- start 2023-11-28T18:50:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.714+0000 7fd37fc09640 1 -- start start 2023-11-28T18:50:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3780c50b0 0x7fd3780c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3780c5490 0x7fd3781474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c6840 0x7fd378158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780c41c0 con 0x7fd3780c6840 2023-11-28T18:50:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780c3530 con 0x7fd3780c50b0 2023-11-28T18:50:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780c4980 con 0x7fd3780c5490 2023-11-28T18:50:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c6840 0x7fd378158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c6840 0x7fd378158d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43554/0 (socket says 172.21.15.7:43554) 2023-11-28T18:50:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 -- 172.21.15.7:0/2707214992 learned_addr learned my addr 172.21.15.7:0/2707214992 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37d9a5640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3780c50b0 0x7fd3780c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37d1a4640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3780c5490 0x7fd3781474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 -- 172.21.15.7:0/2707214992 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3780c5490 msgr2=0x7fd3781474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3780c5490 0x7fd3781474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 -- 172.21.15.7:0/2707214992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3780c50b0 msgr2=0x7fd3780c39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3780c50b0 0x7fd3780c39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 -- 172.21.15.7:0/2707214992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd374009420 con 0x7fd3780c6840 2023-11-28T18:50:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37e1a6640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c6840 0x7fd378158d70 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd37000c780 tx=0x7fd37000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd36e7fc640 1 -- 172.21.15.7:0/2707214992 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37000ce90 con 0x7fd3780c6840 2023-11-28T18:50:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd36e7fc640 1 -- 172.21.15.7:0/2707214992 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37000a640 con 0x7fd3780c6840 2023-11-28T18:50:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37815b320 con 0x7fd3780c6840 2023-11-28T18:50:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd36e7fc640 1 -- 172.21.15.7:0/2707214992 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37000a9d0 con 0x7fd3780c6840 2023-11-28T18:50:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3780c4c20 con 0x7fd3780c6840 2023-11-28T18:50:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.718+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37804e890 con 0x7fd3780c6840 2023-11-28T18:50:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.722+0000 7fd36e7fc640 1 -- 172.21.15.7:0/2707214992 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd37000ab30 con 0x7fd3780c6840 2023-11-28T18:50:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.722+0000 7fd36e7fc640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd35c066900 0x7fd35c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.722+0000 7fd36e7fc640 1 -- 172.21.15.7:0/2707214992 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fd370083ff0 con 0x7fd3780c6840 2023-11-28T18:50:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.722+0000 7fd37d9a5640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd35c066900 0x7fd35c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.722+0000 7fd37d9a5640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd35c066900 0x7fd35c068dc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd3740af000 tx=0x7fd3740a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.726+0000 7fd36e7fc640 1 -- 172.21.15.7:0/2707214992 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd3700b5280 con 0x7fd3780c6840 2023-11-28T18:50:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.898+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd37806e5f0 con 0x7fd3780c6840 2023-11-28T18:50:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.898+0000 7fd36e7fc640 1 -- 172.21.15.7:0/2707214992 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 24 entries v34) v1 ==== 81+0+1464 (secure 0 0 0) 0x7fd370018020 con 0x7fd3780c6840 2023-11-28T18:50:23.900 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6834/695953635 2023-11-29T18:45:32.502253+0000 2023-11-28T18:50:23.900 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6832/2303207996 2023-11-29T18:43:32.460743+0000 2023-11-28T18:50:23.901 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6837/1742899523 2023-11-29T18:42:07.410735+0000 2023-11-28T18:50:23.901 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6835/1029915019 2023-11-29T18:40:17.364225+0000 2023-11-28T18:50:23.901 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6834/1029915019 2023-11-29T18:40:17.364225+0000 2023-11-28T18:50:23.901 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6835/650087483 2023-11-29T18:38:37.328502+0000 2023-11-28T18:50:23.901 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6832/650087483 2023-11-29T18:38:37.328502+0000 2023-11-28T18:50:23.901 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6838/2690556675 2023-11-29T18:32:52.248615+0000 2023-11-28T18:50:23.901 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6834/1742899523 2023-11-29T18:42:07.410735+0000 2023-11-28T18:50:23.902 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6832/4143265275 2023-11-29T18:28:12.173848+0000 2023-11-28T18:50:23.902 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6834/3911527535 2023-11-29T18:30:47.207685+0000 2023-11-28T18:50:23.902 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6839/1819203827 2023-11-29T18:35:07.279902+0000 2023-11-28T18:50:23.902 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6837/1819203827 2023-11-29T18:35:07.279902+0000 2023-11-28T18:50:23.902 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6835/2303207996 2023-11-29T18:43:32.460743+0000 2023-11-28T18:50:23.902 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6836/1685294311 2023-11-29T18:36:52.304067+0000 2023-11-28T18:50:23.902 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6834/1419029356 2023-11-29T18:30:02.197504+0000 2023-11-28T18:50:23.903 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6835/695953635 2023-11-29T18:45:32.502253+0000 2023-11-28T18:50:23.903 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6835/1419029356 2023-11-29T18:30:02.197504+0000 2023-11-28T18:50:23.903 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6836/2690556675 2023-11-29T18:32:52.248615+0000 2023-11-28T18:50:23.903 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6833/1372595381 2023-11-29T18:25:37.139777+0000 2023-11-28T18:50:23.903 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6837/3911527535 2023-11-29T18:30:47.207685+0000 2023-11-28T18:50:23.903 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6836/1372595381 2023-11-29T18:25:37.139777+0000 2023-11-28T18:50:23.903 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6835/4143265275 2023-11-29T18:28:12.173848+0000 2023-11-28T18:50:23.903 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6833/1685294311 2023-11-29T18:36:52.304067+0000 2023-11-28T18:50:23.904 INFO:teuthology.orchestra.run.smithi007.stderr:listed 24 entries 2023-11-28T18:50:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd35c066900 msgr2=0x7fd35c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd35c066900 0x7fd35c068dc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd3740af000 tx=0x7fd3740a7000 comp rx=0 tx=0).stop 2023-11-28T18:50:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c6840 msgr2=0x7fd378158d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c6840 0x7fd378158d70 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd37000c780 tx=0x7fd37000cc40 comp rx=0 tx=0).stop 2023-11-28T18:50:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 shutdown_connections 2023-11-28T18:50:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fd35c066900 0x7fd35c068dc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3780c6840 0x7fd378158d70 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd3780c5490 0x7fd3781474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 --2- 172.21.15.7:0/2707214992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3780c50b0 0x7fd3780c39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 >> 172.21.15.7:0/2707214992 conn(0x7fd3780662e0 msgr2=0x7fd3781515d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 shutdown_connections 2023-11-28T18:50:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:23.902+0000 7fd37fc09640 1 -- 172.21.15.7:0/2707214992 wait complete. 2023-11-28T18:50:23.922 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:50:23.922 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T18:50:23.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T18:50:24.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc82112a640 1 Processor -- start 2023-11-28T18:50:24.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc82112a640 1 -- start start 2023-11-28T18:50:24.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc82112a640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fc81c1542e0 0x7fc81c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc82112a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc81c072450 0x7fc81c072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc82112a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c0ca920 con 0x7fc81c0cbb70 2023-11-28T18:50:24.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc82112a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c071d50 con 0x7fc81c072450 2023-11-28T18:50:24.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc82112a640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c072030 con 0x7fc81c1542e0 2023-11-28T18:50:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81b59e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc81c072450 0x7fc81c072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81b59e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc81c072450 0x7fc81c072870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43558/0 (socket says 172.21.15.7:43558) 2023-11-28T18:50:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81b59e640 1 -- 172.21.15.7:0/989779613 learned_addr learned my addr 172.21.15.7:0/989779613 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81ad9d640 1 --2- 172.21.15.7:0/989779613 >> v2:172.21.15.193:3300/0 conn(0x7fc81c1542e0 0x7fc81c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81b59e640 1 -- 172.21.15.7:0/989779613 >> v2:172.21.15.193:3300/0 conn(0x7fc81c1542e0 msgr2=0x7fc81c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81b59e640 1 --2- 172.21.15.7:0/989779613 >> v2:172.21.15.193:3300/0 conn(0x7fc81c1542e0 0x7fc81c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81b59e640 1 -- 172.21.15.7:0/989779613 >> v1:172.21.15.193:6789/0 conn(0x7fc81c0cbb70 legacy=0x7fc81c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81b59e640 1 -- 172.21.15.7:0/989779613 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c15e550 con 0x7fc81c072450 2023-11-28T18:50:24.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.118+0000 7fc81ad9d640 1 --2- 172.21.15.7:0/989779613 >> v2:172.21.15.193:3300/0 conn(0x7fc81c1542e0 0x7fc81c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:50:24.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc81b59e640 1 --2- 172.21.15.7:0/989779613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c072450 0x7fc81c072870 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc81c071d20 tx=0x7fc81401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c026bd896194f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc819d9b640 1 -- 172.21.15.7:0/989779613 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8140a57a0 con 0x7fc81c072450 2023-11-28T18:50:24.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc819d9b640 1 -- 172.21.15.7:0/989779613 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8140a5900 con 0x7fc81c072450 2023-11-28T18:50:24.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc819d9b640 1 -- 172.21.15.7:0/989779613 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8140a5c90 con 0x7fc81c072450 2023-11-28T18:50:24.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc82112a640 1 -- 172.21.15.7:0/989779613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c072450 msgr2=0x7fc81c072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc82112a640 1 --2- 172.21.15.7:0/989779613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c072450 0x7fc81c072870 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc81c071d20 tx=0x7fc81401c9d0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc82112a640 1 -- 172.21.15.7:0/989779613 shutdown_connections 2023-11-28T18:50:24.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc82112a640 1 --2- 172.21.15.7:0/989779613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c072450 0x7fc81c072870 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc82112a640 1 --2- 172.21.15.7:0/989779613 >> v2:172.21.15.193:3300/0 conn(0x7fc81c1542e0 0x7fc81c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc82112a640 1 -- 172.21.15.7:0/989779613 >> 172.21.15.7:0/989779613 conn(0x7fc81c0717e0 msgr2=0x7fc81c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:24.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc82112a640 1 -- 172.21.15.7:0/989779613 shutdown_connections 2023-11-28T18:50:24.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.122+0000 7fc82112a640 1 -- 172.21.15.7:0/989779613 wait complete. 2023-11-28T18:50:24.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 Processor -- start 2023-11-28T18:50:24.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 -- start start 2023-11-28T18:50:24.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc81c072450 0x7fc81c0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc81c0cbb70 0x7fc81c0c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c1542e0 0x7fc81c152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c154c00 con 0x7fc81c1542e0 2023-11-28T18:50:24.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0cd740 con 0x7fc81c072450 2023-11-28T18:50:24.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0cf020 con 0x7fc81c0cbb70 2023-11-28T18:50:24.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c1542e0 0x7fc81c152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c1542e0 0x7fc81c152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43574/0 (socket says 172.21.15.7:43574) 2023-11-28T18:50:24.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 -- 172.21.15.7:0/4019508835 learned_addr learned my addr 172.21.15.7:0/4019508835 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:24.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81ad9d640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc81c072450 0x7fc81c0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81a59c640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc81c0cbb70 0x7fc81c0c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 -- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc81c0cbb70 msgr2=0x7fc81c0c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc81c0cbb70 0x7fc81c0c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 -- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc81c072450 msgr2=0x7fc81c0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc81c072450 0x7fc81c0cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 -- 172.21.15.7:0/4019508835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc814009b30 con 0x7fc81c1542e0 2023-11-28T18:50:24.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81a59c640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc81c0cbb70 0x7fc81c0c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:50:24.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc81b59e640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c1542e0 0x7fc81c152ad0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc8140af040 tx=0x7fc814019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc8037fe640 1 -- 172.21.15.7:0/4019508835 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8140b8670 con 0x7fc81c1542e0 2023-11-28T18:50:24.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc8037fe640 1 -- 172.21.15.7:0/4019508835 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8140b8810 con 0x7fc81c1542e0 2023-11-28T18:50:24.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc8037fe640 1 -- 172.21.15.7:0/4019508835 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8140b8bc0 con 0x7fc81c1542e0 2023-11-28T18:50:24.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c0cdcc0 con 0x7fc81c1542e0 2023-11-28T18:50:24.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.126+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c0cd9e0 con 0x7fc81c1542e0 2023-11-28T18:50:24.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.130+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c06c670 con 0x7fc81c1542e0 2023-11-28T18:50:24.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.130+0000 7fc8037fe640 1 -- 172.21.15.7:0/4019508835 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc8140b8d20 con 0x7fc81c1542e0 2023-11-28T18:50:24.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.130+0000 7fc8037fe640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8080668d0 0x7fc808068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.130+0000 7fc8037fe640 1 -- 172.21.15.7:0/4019508835 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fc81411fd70 con 0x7fc81c1542e0 2023-11-28T18:50:24.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.130+0000 7fc81ad9d640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8080668d0 0x7fc808068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.130+0000 7fc81ad9d640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8080668d0 0x7fc808068d90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc81c1f0230 tx=0x7fc80c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.134+0000 7fc8037fe640 1 -- 172.21.15.7:0/4019508835 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc8140ecaa0 con 0x7fc81c1542e0 2023-11-28T18:50:24.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.294+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc81c04e890 con 0x7fc81c1542e0 2023-11-28T18:50:24.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.294+0000 7fc8037fe640 1 -- 172.21.15.7:0/4019508835 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+18253 (secure 0 0 0) 0x7fc8140f00c0 con 0x7fc81c1542e0 2023-11-28T18:50:24.296 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:24.296 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":34,"fsid":"b4396e54-0cbb-426d-8760-05a802ba48a2","created":"2023-11-28T18:22:42.315615+0000","modified":"2023-11-28T18:45:32.502309+0000","last_up_change":"2023-11-28T18:22:54.338513+0000","last_in_change":"2023-11-28T18:22:45.160970+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T18:22:55.523418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T18:23:05.706694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T18:23:07.048910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e137b39-c9c2-4354-b748-63bb98afe2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6812","nonce":1682271586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6816","nonce":1682271586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6823","nonce":1682271586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1682271586},{"type":"v1","addr":"172.21.15.7:6820","nonce":1682271586}]},"public_addr":"172.21.15.7:6812/1682271586","cluster_addr":"172.21.15.7:6816/1682271586","heartbeat_back_addr":"172.21.15.7:6823/1682271586","heartbeat_front_addr":"172.21.15.7:6820/1682271586","state":["exists","up"]},{"osd":1,"uuid":"95148a41-c6a1-4b3d-a5cd-76a64e27aa52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6825","nonce":927014245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6827","nonce":927014245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6831","nonce":927014245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":927014245},{"type":"v1","addr":"172.21.15.7:6829","nonce":927014245}]},"public_addr":"172.21.15.7:6825/927014245","cluster_addr":"172.21.15.7:6827/927014245","heartbeat_back_addr":"172.21.15.7:6831/927014245","heartbeat_front_addr":"172.21.15.7:6829/927014245","state":["exists","up"]},{"osd":2,"uuid":"a436ddaf-975a-4147-8e22-a17992126b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6809","nonce":2282553034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6813","nonce":2282553034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6819","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6821","nonce":2282553034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2282553034},{"type":"v1","addr":"172.21.15.7:6817","nonce":2282553034}]},"public_addr":"172.21.15.7:6809/2282553034","cluster_addr":"172.21.15.7:6813/2282553034","heartbeat_back_addr":"172.21.15.7:6821/2282553034","heartbeat_front_addr":"172.21.15.7:6817/2282553034","state":["exists","up"]},{"osd":3,"uuid":"857e014e-444b-407f-a223-62e7384e471c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6801","nonce":2659836262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6803","nonce":2659836262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6807","nonce":2659836262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2659836262},{"type":"v1","addr":"172.21.15.7:6805","nonce":2659836262}]},"public_addr":"172.21.15.7:6801/2659836262","cluster_addr":"172.21.15.7:6803/2659836262","heartbeat_back_addr":"172.21.15.7:6807/2659836262","heartbeat_front_addr":"172.21.15.7:6805/2659836262","state":["exists","up"]},{"osd":4,"uuid":"b6d20668-0212-4e4f-84fd-968ea13370b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6817","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6819","nonce":2413083632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6821","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6823","nonce":2413083632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6829","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6831","nonce":2413083632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6825","nonce":2413083632},{"type":"v1","addr":"172.21.15.193:6827","nonce":2413083632}]},"public_addr":"172.21.15.193:6819/2413083632","cluster_addr":"172.21.15.193:6823/2413083632","heartbeat_back_addr":"172.21.15.193:6831/2413083632","heartbeat_front_addr":"172.21.15.193:6827/2413083632","state":["exists","up"]},{"osd":5,"uuid":"d01156aa-e11d-4fec-9f46-8fcd16feb3bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6801","nonce":4136487123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6803","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6805","nonce":4136487123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6811","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6813","nonce":4136487123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6807","nonce":4136487123},{"type":"v1","addr":"172.21.15.193:6809","nonce":4136487123}]},"public_addr":"172.21.15.193:6801/4136487123","cluster_addr":"172.21.15.193:6805/4136487123","heartbeat_back_addr":"172.21.15.193:6813/4136487123","heartbeat_front_addr":"172.21.15.193:6809/4136487123","state":["exists","up"]},{"osd":6,"uuid":"2846c4d0-bec5-4f62-a048-e8a0c163134d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6818","nonce":667378636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6822","nonce":667378636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6830","nonce":667378636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":667378636},{"type":"v1","addr":"172.21.15.193:6826","nonce":667378636}]},"public_addr":"172.21.15.193:6818/667378636","cluster_addr":"172.21.15.193:6822/667378636","heartbeat_back_addr":"172.21.15.193:6830/667378636","heartbeat_front_addr":"172.21.15.193:6826/667378636","state":["exists","up"]},{"osd":7,"uuid":"ce5ddd1e-0005-4bc1-a5b8-eeedf0a21ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6804","nonce":1530886252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6808","nonce":1530886252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6815","nonce":1530886252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1530886252},{"type":"v1","addr":"172.21.15.193:6812","nonce":1530886252}]},"public_addr":"172.21.15.193:6804/1530886252","cluster_addr":"172.21.15.193:6808/1530886252","heartbeat_back_addr":"172.21.15.193:6815/1530886252","heartbeat_front_addr":"172.21.15.193:6812/1530886252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.942055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.941804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.736152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.738020+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.825852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.896099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.930352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T18:22:50.797065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6834/695953635":"2023-11-29T18:45:32.502253+0000","172.21.15.193:6832/2303207996":"2023-11-29T18:43:32.460743+0000","172.21.15.193:6837/1742899523":"2023-11-29T18:42:07.410735+0000","172.21.15.7:6835/1029915019":"2023-11-29T18:40:17.364225+0000","172.21.15.7:6834/1029915019":"2023-11-29T18:40:17.364225+0000","172.21.15.193:6835/650087483":"2023-11-29T18:38:37.328502+0000","172.21.15.193:6832/650087483":"2023-11-29T18:38:37.328502+0000","172.21.15.7:6838/2690556675":"2023-11-29T18:32:52.248615+0000","172.21.15.193:6834/1742899523":"2023-11-29T18:42:07.410735+0000","172.21.15.193:6832/4143265275":"2023-11-29T18:28:12.173848+0000","172.21.15.193:6834/3911527535":"2023-11-29T18:30:47.207685+0000","172.21.15.7:6839/1819203827":"2023-11-29T18:35:07.279902+0000","172.21.15.7:6837/1819203827":"2023-11-29T18:35:07.279902+0000","172.21.15.193:6835/2303207996":"2023-11-29T18:43:32.460743+0000","172.21.15.193:6836/1685294311":"2023-11-29T18:36:52.304067+0000","172.21.15.7:6834/1419029356":"2023-11-29T18:30:02.197504+0000","172.21.15.7:6835/695953635":"2023-11-29T18:45:32.502253+0000","172.21.15.7:6835/1419029356":"2023-11-29T18:30:02.197504+0000","172.21.15.7:6836/2690556675":"2023-11-29T18:32:52.248615+0000","172.21.15.193:6833/1372595381":"2023-11-29T18:25:37.139777+0000","172.21.15.193:6837/3911527535":"2023-11-29T18:30:47.207685+0000","172.21.15.193:6836/1372595381":"2023-11-29T18:25:37.139777+0000","172.21.15.193:6835/4143265275":"2023-11-29T18:28:12.173848+0000","172.21.15.193:6833/1685294311":"2023-11-29T18:36:52.304067+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T18:50:24.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8080668d0 msgr2=0x7fc808068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8080668d0 0x7fc808068d90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc81c1f0230 tx=0x7fc80c008040 comp rx=0 tx=0).stop 2023-11-28T18:50:24.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c1542e0 msgr2=0x7fc81c152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c1542e0 0x7fc81c152ad0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc8140af040 tx=0x7fc814019770 comp rx=0 tx=0).stop 2023-11-28T18:50:24.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 shutdown_connections 2023-11-28T18:50:24.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7fc8080668d0 0x7fc808068d90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc81c1542e0 0x7fc81c152ad0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fc81c0cbb70 0x7fc81c0c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 --2- 172.21.15.7:0/4019508835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc81c072450 0x7fc81c0cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 >> 172.21.15.7:0/4019508835 conn(0x7fc81c0717e0 msgr2=0x7fc81c15b750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:24.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 shutdown_connections 2023-11-28T18:50:24.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.298+0000 7fc82112a640 1 -- 172.21.15.7:0/4019508835 wait complete. 2023-11-28T18:50:24.317 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T18:22:55.523418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T18:23:05.706694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T18:23:07.048910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T18:50:24.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T18:50:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69cb6fe640 1 Processor -- start 2023-11-28T18:50:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69cb6fe640 1 -- start start 2023-11-28T18:50:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69cb6fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69c415a5c0 0x7f69c40c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69cb6fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f69c415a9a0 0x7f69c415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69cb6fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c40c8530 con 0x7f69c40c3ba0 2023-11-28T18:50:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69cb6fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c40c7680 con 0x7f69c415a5c0 2023-11-28T18:50:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69cb6fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c40c7920 con 0x7f69c415a9a0 2023-11-28T18:50:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69c949a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69c415a5c0 0x7f69c40c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69c949a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69c415a5c0 0x7f69c40c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43580/0 (socket says 172.21.15.7:43580) 2023-11-28T18:50:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69c949a640 1 -- 172.21.15.7:0/419292084 learned_addr learned my addr 172.21.15.7:0/419292084 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69c8c99640 1 --2- 172.21.15.7:0/419292084 >> v2:172.21.15.193:3300/0 conn(0x7f69c415a9a0 0x7f69c415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69c949a640 1 -- 172.21.15.7:0/419292084 >> v2:172.21.15.193:3300/0 conn(0x7f69c415a9a0 msgr2=0x7f69c415adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69c949a640 1 --2- 172.21.15.7:0/419292084 >> v2:172.21.15.193:3300/0 conn(0x7f69c415a9a0 0x7f69c415adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69c949a640 1 -- 172.21.15.7:0/419292084 >> v1:172.21.15.193:6789/0 conn(0x7f69c40c3ba0 legacy=0x7f69c40c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.462+0000 7f69c949a640 1 -- 172.21.15.7:0/419292084 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c40c7c00 con 0x7f69c415a5c0 2023-11-28T18:50:24.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69c949a640 1 --2- 172.21.15.7:0/419292084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c415a5c0 0x7f69c40c6890 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f69c4073720 tx=0x7f69c0021380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963a9675e7f045c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69bbfff640 1 -- 172.21.15.7:0/419292084 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00b1600 con 0x7f69c415a5c0 2023-11-28T18:50:24.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69bbfff640 1 -- 172.21.15.7:0/419292084 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c00ad050 con 0x7f69c415a5c0 2023-11-28T18:50:24.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69bbfff640 1 -- 172.21.15.7:0/419292084 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00b1980 con 0x7f69c415a5c0 2023-11-28T18:50:24.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69cb6fe640 1 -- 172.21.15.7:0/419292084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c415a5c0 msgr2=0x7f69c40c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/419292084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c415a5c0 0x7f69c40c6890 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f69c4073720 tx=0x7f69c0021380 comp rx=0 tx=0).stop 2023-11-28T18:50:24.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69cb6fe640 1 -- 172.21.15.7:0/419292084 shutdown_connections 2023-11-28T18:50:24.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/419292084 >> v2:172.21.15.193:3300/0 conn(0x7f69c415a9a0 0x7f69c415adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/419292084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c415a5c0 0x7f69c40c6890 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69cb6fe640 1 -- 172.21.15.7:0/419292084 >> 172.21.15.7:0/419292084 conn(0x7f69c4073f00 msgr2=0x7f69c40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:24.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69cb6fe640 1 -- 172.21.15.7:0/419292084 shutdown_connections 2023-11-28T18:50:24.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.466+0000 7f69cb6fe640 1 -- 172.21.15.7:0/419292084 wait complete. 2023-11-28T18:50:24.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69cb6fe640 1 Processor -- start 2023-11-28T18:50:24.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69cb6fe640 1 -- start start 2023-11-28T18:50:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69cb6fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c40c3ba0 0x7f69c4154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69cb6fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69c415a5c0 0x7f69c4144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69cb6fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69c415a9a0 0x7f69c4145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69cb6fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41efc00 con 0x7f69c40c3ba0 2023-11-28T18:50:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69cb6fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41efe10 con 0x7f69c415a9a0 2023-11-28T18:50:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69cb6fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41f00f0 con 0x7f69c415a5c0 2023-11-28T18:50:24.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c40c3ba0 0x7f69c4154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c40c3ba0 0x7f69c4154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43588/0 (socket says 172.21.15.7:43588) 2023-11-28T18:50:24.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 -- 172.21.15.7:0/3058444144 learned_addr learned my addr 172.21.15.7:0/3058444144 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:24.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c8c99640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69c415a5c0 0x7f69c4144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c9c9b640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69c415a9a0 0x7f69c4145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 -- 172.21.15.7:0/3058444144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69c415a5c0 msgr2=0x7f69c4144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69c415a5c0 0x7f69c4144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 -- 172.21.15.7:0/3058444144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69c415a9a0 msgr2=0x7f69c4145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69c415a9a0 0x7f69c4145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 -- 172.21.15.7:0/3058444144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c000b770 con 0x7f69c40c3ba0 2023-11-28T18:50:24.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.470+0000 7f69c949a640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c40c3ba0 0x7f69c4154ec0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f69c000b2c0 tx=0x7f69c001fbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69b9ffb640 1 -- 172.21.15.7:0/3058444144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00bc650 con 0x7f69c40c3ba0 2023-11-28T18:50:24.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69b9ffb640 1 -- 172.21.15.7:0/3058444144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c00bc7b0 con 0x7f69c40c3ba0 2023-11-28T18:50:24.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69b9ffb640 1 -- 172.21.15.7:0/3058444144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00bcb60 con 0x7f69c40c3ba0 2023-11-28T18:50:24.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c41f03d0 con 0x7f69c40c3ba0 2023-11-28T18:50:24.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c41f0990 con 0x7f69c40c3ba0 2023-11-28T18:50:24.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c404e890 con 0x7f69c40c3ba0 2023-11-28T18:50:24.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69b9ffb640 1 -- 172.21.15.7:0/3058444144 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f69c00b8070 con 0x7f69c40c3ba0 2023-11-28T18:50:24.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69b9ffb640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69a80668d0 0x7f69a8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69b9ffb640 1 -- 172.21.15.7:0/3058444144 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f69c0123090 con 0x7f69c40c3ba0 2023-11-28T18:50:24.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.474+0000 7f69c8c99640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69a80668d0 0x7f69a8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.478+0000 7f69c8c99640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69a80668d0 0x7f69a8068d90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f69c4150b90 tx=0x7f69b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.478+0000 7f69b9ffb640 1 -- 172.21.15.7:0/3058444144 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f69c00efdb0 con 0x7f69c40c3ba0 2023-11-28T18:50:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f69c414bfd0 con 0x7f69c40c3ba0 2023-11-28T18:50:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69b9ffb640 1 -- 172.21.15.7:0/3058444144 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f69c00b6020 con 0x7f69c40c3ba0 2023-11-28T18:50:24.638 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-11-28T18:50:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69a80668d0 msgr2=0x7f69a8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69a80668d0 0x7f69a8068d90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f69c4150b90 tx=0x7f69b4008040 comp rx=0 tx=0).stop 2023-11-28T18:50:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c40c3ba0 msgr2=0x7f69c4154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c40c3ba0 0x7f69c4154ec0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f69c000b2c0 tx=0x7f69c001fbc0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 shutdown_connections 2023-11-28T18:50:24.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f69a80668d0 0x7f69a8068d90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69c415a9a0 0x7f69c4145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69c415a5c0 0x7f69c4144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 --2- 172.21.15.7:0/3058444144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69c40c3ba0 0x7f69c4154ec0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 >> 172.21.15.7:0/3058444144 conn(0x7f69c4073f00 msgr2=0x7f69c40c3070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:24.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 shutdown_connections 2023-11-28T18:50:24.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.638+0000 7f69cb6fe640 1 -- 172.21.15.7:0/3058444144 wait complete. 2023-11-28T18:50:24.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T18:50:24.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:24.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f470dcfe640 1 Processor -- start 2023-11-28T18:50:24.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f470dcfe640 1 -- start start 2023-11-28T18:50:24.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f470dcfe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f47080c4c80 0x7f4708159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f470dcfe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f47080ca530 0x7f47080cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f470dcfe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080caeb0 con 0x7f4708154130 2023-11-28T18:50:24.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f470dcfe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470815a3f0 con 0x7f47080ca530 2023-11-28T18:50:24.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f470dcfe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470815d9b0 con 0x7f47080c4c80 2023-11-28T18:50:24.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f4707fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f47080ca530 0x7f47080cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f4707fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f47080ca530 0x7f47080cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43600/0 (socket says 172.21.15.7:43600) 2023-11-28T18:50:24.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f47077fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4708154130 0x7f470815de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47992/0 (socket says 172.21.15.7:47992) 2023-11-28T18:50:24.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f4707fff640 1 -- 172.21.15.7:0/1562249188 learned_addr learned my addr 172.21.15.7:0/1562249188 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f4706ffd640 1 --2- 172.21.15.7:0/1562249188 >> v2:172.21.15.193:3300/0 conn(0x7f47080c4c80 0x7f4708159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f4707fff640 1 -- 172.21.15.7:0/1562249188 >> v2:172.21.15.193:3300/0 conn(0x7f47080c4c80 msgr2=0x7f4708159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f4707fff640 1 --2- 172.21.15.7:0/1562249188 >> v2:172.21.15.193:3300/0 conn(0x7f47080c4c80 0x7f4708159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f4707fff640 1 -- 172.21.15.7:0/1562249188 >> v1:172.21.15.7:6789/0 conn(0x7f4708154130 legacy=0x7f470815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.806+0000 7f4707fff640 1 -- 172.21.15.7:0/1562249188 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470815a5d0 con 0x7f47080ca530 2023-11-28T18:50:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f4707fff640 1 --2- 172.21.15.7:0/1562249188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47080ca530 0x7f47080cb2b0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f47080c9e60 tx=0x7f470001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a278b602bc2c301 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f47067fc640 1 -- 172.21.15.7:0/1562249188 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000af400 con 0x7f47080ca530 2023-11-28T18:50:24.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f47067fc640 1 -- 172.21.15.7:0/1562249188 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47000ab050 con 0x7f47080ca530 2023-11-28T18:50:24.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f47067fc640 1 -- 172.21.15.7:0/1562249188 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000af780 con 0x7f47080ca530 2023-11-28T18:50:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f470dcfe640 1 -- 172.21.15.7:0/1562249188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47080ca530 msgr2=0x7f47080cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f470dcfe640 1 --2- 172.21.15.7:0/1562249188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47080ca530 0x7f47080cb2b0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f47080c9e60 tx=0x7f470001f1c0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f470dcfe640 1 -- 172.21.15.7:0/1562249188 shutdown_connections 2023-11-28T18:50:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f470dcfe640 1 --2- 172.21.15.7:0/1562249188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47080ca530 0x7f47080cb2b0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f470dcfe640 1 --2- 172.21.15.7:0/1562249188 >> v2:172.21.15.193:3300/0 conn(0x7f47080c4c80 0x7f4708159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f470dcfe640 1 -- 172.21.15.7:0/1562249188 >> 172.21.15.7:0/1562249188 conn(0x7f4708073ca0 msgr2=0x7f4708071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f470dcfe640 1 -- 172.21.15.7:0/1562249188 shutdown_connections 2023-11-28T18:50:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.810+0000 7f470dcfe640 1 -- 172.21.15.7:0/1562249188 wait complete. 2023-11-28T18:50:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 Processor -- start 2023-11-28T18:50:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 -- start start 2023-11-28T18:50:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f47080c4c80 0x7f47081f0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47080ca530 0x7f47081f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708154130 0x7f4708149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4708155e10 con 0x7f4708154130 2023-11-28T18:50:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470815ccd0 con 0x7f47080ca530 2023-11-28T18:50:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470815bde0 con 0x7f47080c4c80 2023-11-28T18:50:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708154130 0x7f4708149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708154130 0x7f4708149f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43612/0 (socket says 172.21.15.7:43612) 2023-11-28T18:50:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 -- 172.21.15.7:0/3824352849 learned_addr learned my addr 172.21.15.7:0/3824352849 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f47077fe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f47080c4c80 0x7f47081f0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4706ffd640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47080ca530 0x7f47081f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 -- 172.21.15.7:0/3824352849 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f47080c4c80 msgr2=0x7f47081f0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f47080c4c80 0x7f47081f0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 -- 172.21.15.7:0/3824352849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47080ca530 msgr2=0x7f47081f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47080ca530 0x7f47081f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 -- 172.21.15.7:0/3824352849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4700009b30 con 0x7f4708154130 2023-11-28T18:50:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f4707fff640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708154130 0x7f4708149f50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f470001cdf0 tx=0x7f470001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f46effff640 1 -- 172.21.15.7:0/3824352849 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000afdc0 con 0x7f4708154130 2023-11-28T18:50:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f46effff640 1 -- 172.21.15.7:0/3824352849 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4700009510 con 0x7f4708154130 2023-11-28T18:50:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470815c0c0 con 0x7f4708154130 2023-11-28T18:50:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f46effff640 1 -- 172.21.15.7:0/3824352849 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000ba680 con 0x7f4708154130 2023-11-28T18:50:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470815c680 con 0x7f4708154130 2023-11-28T18:50:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.814+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47081f0900 con 0x7f4708154130 2023-11-28T18:50:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.818+0000 7f46effff640 1 -- 172.21.15.7:0/3824352849 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f47000b6040 con 0x7f4708154130 2023-11-28T18:50:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.818+0000 7f46effff640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46f4066900 0x7f46f4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.818+0000 7f46effff640 1 -- 172.21.15.7:0/3824352849 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f4700122c60 con 0x7f4708154130 2023-11-28T18:50:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.818+0000 7f47077fe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46f4066900 0x7f46f4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.818+0000 7f47077fe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46f4066900 0x7f46f4068dc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f46f8000f70 tx=0x7f46f8007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.822+0000 7f46effff640 1 -- 172.21.15.7:0/3824352849 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f47081f0900 con 0x7f4708154130 2023-11-28T18:50:24.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.946+0000 7f4f430bb640 1 Processor -- start 2023-11-28T18:50:24.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f430bb640 1 -- start start 2023-11-28T18:50:24.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f430bb640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4f3c15ce10 0x7f4f3c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f430bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c0cafe0 0x7f4f3c0ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f430bb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c15a3a0 con 0x7f4f3c15ca30 2023-11-28T18:50:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f430bb640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3c15a620 con 0x7f4f3c15ce10 2023-11-28T18:50:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f430bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c15a900 con 0x7f4f3c0cafe0 2023-11-28T18:50:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f418b8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4f3c15ce10 0x7f4f3c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f418b8640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4f3c15ce10 0x7f4f3c0ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.7:58334/0 (socket says 172.21.15.7:58334) 2023-11-28T18:50:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f418b8640 1 -- 172.21.15.7:0/1683183171 learned_addr learned my addr 172.21.15.7:0/1683183171 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f420b9640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f4f3c15ca30 0x7f4f3c159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:53958/0 (socket says 172.21.15.7:53958) 2023-11-28T18:50:24.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f428ba640 1 --2- 172.21.15.7:0/1683183171 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c0cafe0 0x7f4f3c0ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f418b8640 1 -- 172.21.15.7:0/1683183171 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c0cafe0 msgr2=0x7f4f3c0ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:24.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f418b8640 1 --2- 172.21.15.7:0/1683183171 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c0cafe0 0x7f4f3c0ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f418b8640 1 -- 172.21.15.7:0/1683183171 >> v1:172.21.15.193:6789/0 conn(0x7f4f3c15ca30 legacy=0x7f4f3c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f418b8640 1 -- 172.21.15.7:0/1683183171 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c15abe0 con 0x7f4f3c15ce10 2023-11-28T18:50:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.950+0000 7f4f410b7640 1 -- 172.21.15.7:0/1683183171 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691584797 0 0) 0x7f4f3c15a3a0 con 0x7f4f3c15ca30 2023-11-28T18:50:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f418b8640 1 --2- 172.21.15.7:0/1683183171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c0ca1d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009870 tx=0x7f4f2c01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81ffcb2878e8c419 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f410b7640 1 -- 172.21.15.7:0/1683183171 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c0a8a30 con 0x7f4f3c15ce10 2023-11-28T18:50:24.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f410b7640 1 -- 172.21.15.7:0/1683183171 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c0a8b90 con 0x7f4f3c15ce10 2023-11-28T18:50:24.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f410b7640 1 -- 172.21.15.7:0/1683183171 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c0b0440 con 0x7f4f3c15ce10 2023-11-28T18:50:24.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f430bb640 1 -- 172.21.15.7:0/1683183171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c15ce10 msgr2=0x7f4f3c0ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f430bb640 1 --2- 172.21.15.7:0/1683183171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c0ca1d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009870 tx=0x7f4f2c01ed30 comp rx=0 tx=0).stop 2023-11-28T18:50:24.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f430bb640 1 -- 172.21.15.7:0/1683183171 shutdown_connections 2023-11-28T18:50:24.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f430bb640 1 --2- 172.21.15.7:0/1683183171 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c0cafe0 0x7f4f3c0ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f430bb640 1 --2- 172.21.15.7:0/1683183171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c0ca1d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f430bb640 1 -- 172.21.15.7:0/1683183171 >> 172.21.15.7:0/1683183171 conn(0x7f4f3c06c060 msgr2=0x7f4f3c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:24.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f430bb640 1 -- 172.21.15.7:0/1683183171 shutdown_connections 2023-11-28T18:50:24.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.954+0000 7f4f430bb640 1 -- 172.21.15.7:0/1683183171 wait complete. 2023-11-28T18:50:24.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f430bb640 1 Processor -- start 2023-11-28T18:50:24.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f430bb640 1 -- start start 2023-11-28T18:50:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f430bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c0cafe0 0x7f4f3c1572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f430bb640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c15ca30 0x7f4f3c1f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f430bb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c1f08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f430bb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c156610 con 0x7f4f3c15ce10 2023-11-28T18:50:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f430bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c1f3590 con 0x7f4f3c0cafe0 2023-11-28T18:50:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f430bb640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c155ce0 con 0x7f4f3c15ca30 2023-11-28T18:50:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c1f08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c1f08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43616/0 (socket says 172.21.15.7:43616) 2023-11-28T18:50:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 -- 172.21.15.7:0/314285823 learned_addr learned my addr 172.21.15.7:0/314285823 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f418b8640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c15ca30 0x7f4f3c1f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f420b9640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c0cafe0 0x7f4f3c1572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 -- 172.21.15.7:0/314285823 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c15ca30 msgr2=0x7f4f3c1f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c15ca30 0x7f4f3c1f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 -- 172.21.15.7:0/314285823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c0cafe0 msgr2=0x7f4f3c1572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c0cafe0 0x7f4f3c1572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 -- 172.21.15.7:0/314285823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0094d0 con 0x7f4f3c15ce10 2023-11-28T18:50:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.958+0000 7f4f428ba640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c1f08d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4f3400efc0 tx=0x7f4f3400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.962+0000 7f4f32ffd640 1 -- 172.21.15.7:0/314285823 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f34014480 con 0x7f4f3c15ce10 2023-11-28T18:50:24.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.962+0000 7f4f32ffd640 1 -- 172.21.15.7:0/314285823 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3400f040 con 0x7f4f3c15ce10 2023-11-28T18:50:24.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.962+0000 7f4f32ffd640 1 -- 172.21.15.7:0/314285823 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f34014820 con 0x7f4f3c15ce10 2023-11-28T18:50:24.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.962+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c155fc0 con 0x7f4f3c15ce10 2023-11-28T18:50:24.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.962+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c156aa0 con 0x7f4f3c15ce10 2023-11-28T18:50:24.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.962+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c156780 con 0x7f4f3c15ce10 2023-11-28T18:50:24.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.962+0000 7f4f32ffd640 1 -- 172.21.15.7:0/314285823 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4f34014980 con 0x7f4f3c15ce10 2023-11-28T18:50:24.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.966+0000 7f4f32ffd640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f200668d0 0x7f4f20068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:24.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.966+0000 7f4f32ffd640 1 -- 172.21.15.7:0/314285823 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f4f34085350 con 0x7f4f3c15ce10 2023-11-28T18:50:24.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.966+0000 7f4f420b9640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f200668d0 0x7f4f20068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:24.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.966+0000 7f4f420b9640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f200668d0 0x7f4f20068d90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4f38007fd0 tx=0x7f4f38007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:24.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.970+0000 7f4f32ffd640 1 -- 172.21.15.7:0/314285823 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4f34052130 con 0x7f4f3c15ce10 2023-11-28T18:50:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f47081f0900 con 0x7f4708154130 2023-11-28T18:50:24.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f46effff640 1 -- 172.21.15.7:0/3824352849 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f47081f0900 con 0x7f4708154130 2023-11-28T18:50:24.986 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-11-28T18:50:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46f4066900 msgr2=0x7f46f4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46f4066900 0x7f46f4068dc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f46f8000f70 tx=0x7f46f8007c90 comp rx=0 tx=0).stop 2023-11-28T18:50:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708154130 msgr2=0x7f4708149f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708154130 0x7f4708149f50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f470001cdf0 tx=0x7f470001bf60 comp rx=0 tx=0).stop 2023-11-28T18:50:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 shutdown_connections 2023-11-28T18:50:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f46f4066900 0x7f46f4068dc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708154130 0x7f4708149f50 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47080ca530 0x7f47081f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 --2- 172.21.15.7:0/3824352849 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f47080c4c80 0x7f47081f0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:24.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 >> 172.21.15.7:0/3824352849 conn(0x7f4708073ca0 msgr2=0x7f47081518d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:24.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 shutdown_connections 2023-11-28T18:50:24.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:24.986+0000 7f470dcfe640 1 -- 172.21.15.7:0/3824352849 wait complete. 2023-11-28T18:50:25.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T18:50:25.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.162+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f3c0cb6c0 con 0x7f4f3c15ce10 2023-11-28T18:50:25.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f32ffd640 1 -- 172.21.15.7:0/314285823 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f4f34018090 con 0x7f4f3c15ce10 2023-11-28T18:50:25.165 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:25.165 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:25.166 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:25.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f200668d0 msgr2=0x7f4f20068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f200668d0 0x7f4f20068d90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4f38007fd0 tx=0x7f4f38007d60 comp rx=0 tx=0).stop 2023-11-28T18:50:25.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c15ce10 msgr2=0x7f4f3c1f08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c1f08d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4f3400efc0 tx=0x7f4f3400c490 comp rx=0 tx=0).stop 2023-11-28T18:50:25.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 shutdown_connections 2023-11-28T18:50:25.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f4f200668d0 0x7f4f20068d90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c15ce10 0x7f4f3c1f08d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4f3c15ca30 0x7f4f3c1f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 --2- 172.21.15.7:0/314285823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f3c0cafe0 0x7f4f3c1572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 >> 172.21.15.7:0/314285823 conn(0x7f4f3c06c060 msgr2=0x7f4f3c0cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:25.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 shutdown_connections 2023-11-28T18:50:25.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.166+0000 7f4f430bb640 1 -- 172.21.15.7:0/314285823 wait complete. 2023-11-28T18:50:25.187 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc67bdf640 1 Processor -- start 2023-11-28T18:50:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc67bdf640 1 -- start start 2023-11-28T18:50:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc67bdf640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7efc601540d0 0x7efc60072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc67bdf640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc6015db70 0x7efc6015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc67bdf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc6015ae90 con 0x7efc600cd560 2023-11-28T18:50:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc67bdf640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6015e4c0 con 0x7efc6015db70 2023-11-28T18:50:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc67bdf640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6015a370 con 0x7efc601540d0 2023-11-28T18:50:25.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc6517a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc6015db70 0x7efc6015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc6517a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc6015db70 0x7efc6015df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43630/0 (socket says 172.21.15.7:43630) 2023-11-28T18:50:25.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc6517a640 1 -- 172.21.15.7:0/2235172358 learned_addr learned my addr 172.21.15.7:0/2235172358 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:25.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.194+0000 7efc6597b640 1 --2- 172.21.15.7:0/2235172358 >> v2:172.21.15.193:3300/0 conn(0x7efc601540d0 0x7efc60072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.198+0000 7efc6517a640 1 -- 172.21.15.7:0/2235172358 >> v2:172.21.15.193:3300/0 conn(0x7efc601540d0 msgr2=0x7efc60072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:25.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.198+0000 7efc6517a640 1 --2- 172.21.15.7:0/2235172358 >> v2:172.21.15.193:3300/0 conn(0x7efc601540d0 0x7efc60072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.198+0000 7efc6517a640 1 -- 172.21.15.7:0/2235172358 >> v1:172.21.15.193:6789/0 conn(0x7efc600cd560 legacy=0x7efc600ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.198+0000 7efc6517a640 1 -- 172.21.15.7:0/2235172358 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6015a610 con 0x7efc6015db70 2023-11-28T18:50:25.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.198+0000 7efc6517a640 1 --2- 172.21.15.7:0/2235172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc6015db70 0x7efc6015df80 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7efc6015a800 tx=0x7efc5001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ea2675bb5ee0e3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:25.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.198+0000 7efc64979640 1 -- 172.21.15.7:0/2235172358 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500ab5f0 con 0x7efc6015db70 2023-11-28T18:50:25.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.198+0000 7efc64979640 1 -- 172.21.15.7:0/2235172358 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc500ab750 con 0x7efc6015db70 2023-11-28T18:50:25.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.198+0000 7efc64979640 1 -- 172.21.15.7:0/2235172358 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500abae0 con 0x7efc6015db70 2023-11-28T18:50:25.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- 172.21.15.7:0/2235172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc6015db70 msgr2=0x7efc6015df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 --2- 172.21.15.7:0/2235172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc6015db70 0x7efc6015df80 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7efc6015a800 tx=0x7efc5001c500 comp rx=0 tx=0).stop 2023-11-28T18:50:25.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- 172.21.15.7:0/2235172358 shutdown_connections 2023-11-28T18:50:25.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 --2- 172.21.15.7:0/2235172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc6015db70 0x7efc6015df80 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 --2- 172.21.15.7:0/2235172358 >> v2:172.21.15.193:3300/0 conn(0x7efc601540d0 0x7efc60072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- 172.21.15.7:0/2235172358 >> 172.21.15.7:0/2235172358 conn(0x7efc600662e0 msgr2=0x7efc600666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:25.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- 172.21.15.7:0/2235172358 shutdown_connections 2023-11-28T18:50:25.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- 172.21.15.7:0/2235172358 wait complete. 2023-11-28T18:50:25.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 Processor -- start 2023-11-28T18:50:25.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- start start 2023-11-28T18:50:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc600cd560 0x7efc6013cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efc601540d0 0x7efc6013d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc6015db70 0x7efc601444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc600c45c0 con 0x7efc600cd560 2023-11-28T18:50:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc601401d0 con 0x7efc6015db70 2023-11-28T18:50:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc67bdf640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc60140880 con 0x7efc601540d0 2023-11-28T18:50:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6597b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc600cd560 0x7efc6013cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6597b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc600cd560 0x7efc6013cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43646/0 (socket says 172.21.15.7:43646) 2023-11-28T18:50:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6597b640 1 -- 172.21.15.7:0/1831723604 learned_addr learned my addr 172.21.15.7:0/1831723604 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6517a640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efc601540d0 0x7efc6013d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6617c640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc6015db70 0x7efc601444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6597b640 1 -- 172.21.15.7:0/1831723604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efc601540d0 msgr2=0x7efc6013d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6597b640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efc601540d0 0x7efc6013d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6597b640 1 -- 172.21.15.7:0/1831723604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc6015db70 msgr2=0x7efc601444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6597b640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc6015db70 0x7efc601444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.202+0000 7efc6597b640 1 -- 172.21.15.7:0/1831723604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500094d0 con 0x7efc600cd560 2023-11-28T18:50:25.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc6597b640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc600cd560 0x7efc6013cd80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00e9b0 tx=0x7efc5c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:25.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc567fc640 1 -- 172.21.15.7:0/1831723604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c0096b0 con 0x7efc600cd560 2023-11-28T18:50:25.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc567fc640 1 -- 172.21.15.7:0/1831723604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5c00c9a0 con 0x7efc600cd560 2023-11-28T18:50:25.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc600cb570 con 0x7efc600cd560 2023-11-28T18:50:25.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc567fc640 1 -- 172.21.15.7:0/1831723604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c00cd30 con 0x7efc600cd560 2023-11-28T18:50:25.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc60153550 con 0x7efc600cd560 2023-11-28T18:50:25.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc60144270 con 0x7efc600cd560 2023-11-28T18:50:25.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc567fc640 1 -- 172.21.15.7:0/1831723604 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7efc5c013470 con 0x7efc600cd560 2023-11-28T18:50:25.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc567fc640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efc44066900 0x7efc44068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc567fc640 1 -- 172.21.15.7:0/1831723604 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7efc5c084ce0 con 0x7efc600cd560 2023-11-28T18:50:25.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.206+0000 7efc6517a640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efc44066900 0x7efc44068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.210+0000 7efc6517a640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efc44066900 0x7efc44068dc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efc500af000 tx=0x7efc500a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:25.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.210+0000 7efc567fc640 1 -- 172.21.15.7:0/1831723604 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efc5c051970 con 0x7efc600cd560 2023-11-28T18:50:25.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efc6004e890 con 0x7efc600cd560 2023-11-28T18:50:25.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc567fc640 1 -- 172.21.15.7:0/1831723604 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7efc5c0210a0 con 0x7efc600cd560 2023-11-28T18:50:25.370 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-11-28T18:50:25.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efc44066900 msgr2=0x7efc44068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efc44066900 0x7efc44068dc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efc500af000 tx=0x7efc500a7000 comp rx=0 tx=0).stop 2023-11-28T18:50:25.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc600cd560 msgr2=0x7efc6013cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc600cd560 0x7efc6013cd80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00e9b0 tx=0x7efc5c00ee70 comp rx=0 tx=0).stop 2023-11-28T18:50:25.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 shutdown_connections 2023-11-28T18:50:25.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7efc44066900 0x7efc44068dc0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc6015db70 0x7efc601444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efc601540d0 0x7efc6013d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 --2- 172.21.15.7:0/1831723604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc600cd560 0x7efc6013cd80 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 >> 172.21.15.7:0/1831723604 conn(0x7efc600662e0 msgr2=0x7efc600c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 shutdown_connections 2023-11-28T18:50:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.370+0000 7efc67bdf640 1 -- 172.21.15.7:0/1831723604 wait complete. 2023-11-28T18:50:25.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T18:50:25.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.574+0000 7f618ec32640 1 Processor -- start 2023-11-28T18:50:25.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.574+0000 7f618ec32640 1 -- start start 2023-11-28T18:50:25.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f618ec32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61881516e0 0x7f6188151b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f618ec32640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188152080 con 0x7f61880c5490 2023-11-28T18:50:25.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f618ec32640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188070630 con 0x7f61880c50b0 2023-11-28T18:50:25.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f618ec32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6188071cb0 con 0x7f61881516e0 2023-11-28T18:50:25.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f6187fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f61880c5490 0x7f61880c6300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47996/0 (socket says 172.21.15.7:47996) 2023-11-28T18:50:25.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f6187fff640 1 -- 172.21.15.7:0/2117653512 learned_addr learned my addr 172.21.15.7:0/2117653512 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:25.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f618d1cf640 1 --2- 172.21.15.7:0/2117653512 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61881516e0 0x7f6188151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f61877fe640 1 -- 172.21.15.7:0/2117653512 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650985784 0 0) 0x7f6188152080 con 0x7f61880c5490 2023-11-28T18:50:25.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f61877fe640 1 -- 172.21.15.7:0/2117653512 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6178003530 con 0x7f61880c5490 2023-11-28T18:50:25.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f61877fe640 1 -- 172.21.15.7:0/2117653512 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156406297 0 0) 0x7f6188070630 con 0x7f61880c50b0 2023-11-28T18:50:25.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f61877fe640 1 -- 172.21.15.7:0/2117653512 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188152080 con 0x7f61880c50b0 2023-11-28T18:50:25.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f61877fe640 1 -- 172.21.15.7:0/2117653512 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 957779382 0 0) 0x7f6178003530 con 0x7f61880c5490 2023-11-28T18:50:25.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f618d1cf640 1 -- 172.21.15.7:0/2117653512 >> v1:172.21.15.193:6789/0 conn(0x7f61880c50b0 legacy=0x7f61881584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f618d1cf640 1 -- 172.21.15.7:0/2117653512 >> v1:172.21.15.7:6789/0 conn(0x7f61880c5490 legacy=0x7f61880c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.578+0000 7f618d1cf640 1 -- 172.21.15.7:0/2117653512 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6188070910 con 0x7f61881516e0 2023-11-28T18:50:25.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618d1cf640 1 --2- 172.21.15.7:0/2117653512 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61881516e0 0x7f6188151b40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6180009dc0 tx=0x7f618001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87eaf9e387cdfd39 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:25.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f61877fe640 1 -- 172.21.15.7:0/2117653512 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174001680 con 0x7f61881516e0 2023-11-28T18:50:25.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f61877fe640 1 -- 172.21.15.7:0/2117653512 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61800a55c0 con 0x7f61881516e0 2023-11-28T18:50:25.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f61877fe640 1 -- 172.21.15.7:0/2117653512 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800a5950 con 0x7f61881516e0 2023-11-28T18:50:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 -- 172.21.15.7:0/2117653512 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61881516e0 msgr2=0x7f6188151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 --2- 172.21.15.7:0/2117653512 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61881516e0 0x7f6188151b40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6180009dc0 tx=0x7f618001c890 comp rx=0 tx=0).stop 2023-11-28T18:50:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 -- 172.21.15.7:0/2117653512 shutdown_connections 2023-11-28T18:50:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 --2- 172.21.15.7:0/2117653512 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61881516e0 0x7f6188151b40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 -- 172.21.15.7:0/2117653512 >> 172.21.15.7:0/2117653512 conn(0x7f61880662e0 msgr2=0x7f61880666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:25.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 -- 172.21.15.7:0/2117653512 shutdown_connections 2023-11-28T18:50:25.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 -- 172.21.15.7:0/2117653512 wait complete. 2023-11-28T18:50:25.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 Processor -- start 2023-11-28T18:50:25.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.582+0000 7f618ec32640 1 -- start start 2023-11-28T18:50:25.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61880c50b0 0x7f6188148c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61880c5490 0x7f6188147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61881516e0 0x7f6188149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61881626e0 con 0x7f61880c5490 2023-11-28T18:50:25.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61881475e0 con 0x7f61881516e0 2023-11-28T18:50:25.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6188146fd0 con 0x7f61880c50b0 2023-11-28T18:50:25.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61880c5490 0x7f6188147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61880c5490 0x7f6188147a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43658/0 (socket says 172.21.15.7:43658) 2023-11-28T18:50:25.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 -- 172.21.15.7:0/4076189895 learned_addr learned my addr 172.21.15.7:0/4076189895 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:25.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618d1cf640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61881516e0 0x7f6188149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618c9ce640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61880c50b0 0x7f6188148c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 -- 172.21.15.7:0/4076189895 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61880c50b0 msgr2=0x7f6188148c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61880c50b0 0x7f6188148c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 -- 172.21.15.7:0/4076189895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61881516e0 msgr2=0x7f6188149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61881516e0 0x7f6188149900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 -- 172.21.15.7:0/4076189895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180009a70 con 0x7f61880c5490 2023-11-28T18:50:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f6187fff640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61880c5490 0x7f6188147a70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f6174005890 tx=0x7f617400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:25.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f61857fa640 1 -- 172.21.15.7:0/4076189895 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174005ae0 con 0x7f61880c5490 2023-11-28T18:50:25.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f61857fa640 1 -- 172.21.15.7:0/4076189895 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6174005c40 con 0x7f61880c5490 2023-11-28T18:50:25.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f61857fa640 1 -- 172.21.15.7:0/4076189895 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174007070 con 0x7f61880c5490 2023-11-28T18:50:25.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6188149e40 con 0x7f61880c5490 2023-11-28T18:50:25.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6188147270 con 0x7f61880c5490 2023-11-28T18:50:25.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.586+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6188147fb0 con 0x7f61880c5490 2023-11-28T18:50:25.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.590+0000 7f61857fa640 1 -- 172.21.15.7:0/4076189895 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f61740071d0 con 0x7f61880c5490 2023-11-28T18:50:25.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.590+0000 7f61857fa640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f616c0668d0 0x7f616c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:25.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.590+0000 7f61857fa640 1 -- 172.21.15.7:0/4076189895 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f6174083ff0 con 0x7f61880c5490 2023-11-28T18:50:25.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.590+0000 7f618c9ce640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f616c0668d0 0x7f616c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:25.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.590+0000 7f618c9ce640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f616c0668d0 0x7f616c068d90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f617c0011f0 tx=0x7f617c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:25.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.594+0000 7f61857fa640 1 -- 172.21.15.7:0/4076189895 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f61740b5280 con 0x7f61880c5490 2023-11-28T18:50:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6188147790 con 0x7f61880c5490 2023-11-28T18:50:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f61857fa640 1 -- 172.21.15.7:0/4076189895 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 24 entries v34) v1 ==== 81+0+1464 (secure 0 0 0) 0x7f6174017070 con 0x7f61880c5490 2023-11-28T18:50:25.763 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6834/695953635 2023-11-29T18:45:32.502253+0000 2023-11-28T18:50:25.763 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6832/2303207996 2023-11-29T18:43:32.460743+0000 2023-11-28T18:50:25.763 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6837/1742899523 2023-11-29T18:42:07.410735+0000 2023-11-28T18:50:25.763 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6835/1029915019 2023-11-29T18:40:17.364225+0000 2023-11-28T18:50:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6834/1029915019 2023-11-29T18:40:17.364225+0000 2023-11-28T18:50:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6835/650087483 2023-11-29T18:38:37.328502+0000 2023-11-28T18:50:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6832/650087483 2023-11-29T18:38:37.328502+0000 2023-11-28T18:50:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6838/2690556675 2023-11-29T18:32:52.248615+0000 2023-11-28T18:50:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6834/1742899523 2023-11-29T18:42:07.410735+0000 2023-11-28T18:50:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6832/4143265275 2023-11-29T18:28:12.173848+0000 2023-11-28T18:50:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6834/3911527535 2023-11-29T18:30:47.207685+0000 2023-11-28T18:50:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6839/1819203827 2023-11-29T18:35:07.279902+0000 2023-11-28T18:50:25.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6837/1819203827 2023-11-29T18:35:07.279902+0000 2023-11-28T18:50:25.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6835/2303207996 2023-11-29T18:43:32.460743+0000 2023-11-28T18:50:25.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6836/1685294311 2023-11-29T18:36:52.304067+0000 2023-11-28T18:50:25.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6834/1419029356 2023-11-29T18:30:02.197504+0000 2023-11-28T18:50:25.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6835/695953635 2023-11-29T18:45:32.502253+0000 2023-11-28T18:50:25.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6835/1419029356 2023-11-29T18:30:02.197504+0000 2023-11-28T18:50:25.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6836/2690556675 2023-11-29T18:32:52.248615+0000 2023-11-28T18:50:25.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6833/1372595381 2023-11-29T18:25:37.139777+0000 2023-11-28T18:50:25.766 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6837/3911527535 2023-11-29T18:30:47.207685+0000 2023-11-28T18:50:25.766 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6836/1372595381 2023-11-29T18:25:37.139777+0000 2023-11-28T18:50:25.766 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6835/4143265275 2023-11-29T18:28:12.173848+0000 2023-11-28T18:50:25.766 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.193:6833/1685294311 2023-11-29T18:36:52.304067+0000 2023-11-28T18:50:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:listed 24 entries 2023-11-28T18:50:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f616c0668d0 msgr2=0x7f616c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f616c0668d0 0x7f616c068d90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f617c0011f0 tx=0x7f617c009040 comp rx=0 tx=0).stop 2023-11-28T18:50:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61880c5490 msgr2=0x7f6188147a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61880c5490 0x7f6188147a70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f6174005890 tx=0x7f617400cbe0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 shutdown_connections 2023-11-28T18:50:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f616c0668d0 0x7f616c068d90 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61881516e0 0x7f6188149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61880c5490 0x7f6188147a70 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 --2- 172.21.15.7:0/4076189895 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f61880c50b0 0x7f6188148c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 >> 172.21.15.7:0/4076189895 conn(0x7f61880662e0 msgr2=0x7f618806e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.762+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 shutdown_connections 2023-11-28T18:50:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:25.766+0000 7f618ec32640 1 -- 172.21.15.7:0/4076189895 wait complete. 2023-11-28T18:50:25.784 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi007.front.sepia.ceph.com... 2023-11-28T18:50:25.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:50:25.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-28T18:50:25.819 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:50:25.820 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:50:25.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:50:25.820 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T18:50:27.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:27.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.342+0000 7f21d10fe640 1 Processor -- start 2023-11-28T18:50:27.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.342+0000 7f21d10fe640 1 -- start start 2023-11-28T18:50:27.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.342+0000 7f21d10fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc150b30 0x7f21cc0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:27.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.342+0000 7f21d10fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f21cc1575c0 0x7f21cc0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:27.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.342+0000 7f21d10fe640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc0c6140 con 0x7f21cc0c4020 2023-11-28T18:50:27.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.342+0000 7f21d10fe640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc0c72b0 con 0x7f21cc1575c0 2023-11-28T18:50:27.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.342+0000 7f21d10fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc070c00 con 0x7f21cc150b30 2023-11-28T18:50:27.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21c3fff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f21cc1575c0 0x7f21cc0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:27.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21cb59e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f21cc0c4020 0x7f21cc0c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.7:53992/0 (socket says 172.21.15.7:53992) 2023-11-28T18:50:27.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21cad9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc150b30 0x7f21cc0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:27.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21cb59e640 1 -- 172.21.15.7:0/1854204887 learned_addr learned my addr 172.21.15.7:0/1854204887 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-28T18:50:27.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21ca59c640 1 -- 172.21.15.7:0/1854204887 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855852164 0 0) 0x7f21cc0c6140 con 0x7f21cc0c4020 2023-11-28T18:50:27.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21ca59c640 1 -- 172.21.15.7:0/1854204887 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b8005740 con 0x7f21cc0c4020 2023-11-28T18:50:27.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21cad9d640 1 -- 172.21.15.7:0/1854204887 >> v2:172.21.15.193:3300/0 conn(0x7f21cc1575c0 msgr2=0x7f21cc0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T18:50:27.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21cad9d640 1 --2- 172.21.15.7:0/1854204887 >> v2:172.21.15.193:3300/0 conn(0x7f21cc1575c0 0x7f21cc0c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21cad9d640 1 -- 172.21.15.7:0/1854204887 >> v1:172.21.15.193:6789/0 conn(0x7f21cc0c4020 legacy=0x7f21cc0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:27.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21cad9d640 1 -- 172.21.15.7:0/1854204887 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0c6370 con 0x7f21cc150b30 2023-11-28T18:50:27.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21c3fff640 1 --2- 172.21.15.7:0/1854204887 >> v2:172.21.15.193:3300/0 conn(0x7f21cc1575c0 0x7f21cc0c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T18:50:27.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.346+0000 7f21cad9d640 1 --2- 172.21.15.7:0/1854204887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc150b30 0x7f21cc0c24f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0097c0 tx=0x7f21bc01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a79cde3fb93585 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21ca59c640 1 -- 172.21.15.7:0/1854204887 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0a8970 con 0x7f21cc150b30 2023-11-28T18:50:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21ca59c640 1 -- 172.21.15.7:0/1854204887 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc0a8ad0 con 0x7f21cc150b30 2023-11-28T18:50:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21ca59c640 1 -- 172.21.15.7:0/1854204887 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0a8e60 con 0x7f21cc150b30 2023-11-28T18:50:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- 172.21.15.7:0/1854204887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc150b30 msgr2=0x7f21cc0c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 --2- 172.21.15.7:0/1854204887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc150b30 0x7f21cc0c24f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0097c0 tx=0x7f21bc01ec70 comp rx=0 tx=0).stop 2023-11-28T18:50:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- 172.21.15.7:0/1854204887 shutdown_connections 2023-11-28T18:50:27.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 --2- 172.21.15.7:0/1854204887 >> v2:172.21.15.193:3300/0 conn(0x7f21cc1575c0 0x7f21cc0c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 --2- 172.21.15.7:0/1854204887 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc150b30 0x7f21cc0c24f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- 172.21.15.7:0/1854204887 >> 172.21.15.7:0/1854204887 conn(0x7f21cc0662e0 msgr2=0x7f21cc0666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:27.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- 172.21.15.7:0/1854204887 shutdown_connections 2023-11-28T18:50:27.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- 172.21.15.7:0/1854204887 wait complete. 2023-11-28T18:50:27.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 Processor -- start 2023-11-28T18:50:27.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- start start 2023-11-28T18:50:27.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21cc0c4020 0x7f21cc1de360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:27.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21cc150b30 0x7f21cc1ebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:27.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc1575c0 0x7f21cc1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc067fa0 con 0x7f21cc0c4020 2023-11-28T18:50:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1ec510 con 0x7f21cc150b30 2023-11-28T18:50:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.350+0000 7f21d10fe640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc155c20 con 0x7f21cc1575c0 2023-11-28T18:50:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21cc0c4020 0x7f21cc1de360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21cc0c4020 0x7f21cc1de360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43660/0 (socket says 172.21.15.7:43660) 2023-11-28T18:50:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 -- 172.21.15.7:0/2969312311 learned_addr learned my addr 172.21.15.7:0/2969312311 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-28T18:50:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cb59e640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc1575c0 0x7f21cc1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21c3fff640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21cc150b30 0x7f21cc1ebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 -- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc1575c0 msgr2=0x7f21cc1556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc1575c0 0x7f21cc1556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 -- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21cc150b30 msgr2=0x7f21cc1ebfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21cc150b30 0x7f21cc1ebfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 -- 172.21.15.7:0/2969312311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b4009420 con 0x7f21cc0c4020 2023-11-28T18:50:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cb59e640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc1575c0 0x7f21cc1556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T18:50:27.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21cad9d640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21cc0c4020 0x7f21cc1de360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0b3040 tx=0x7f21bc01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21c37fe640 1 -- 172.21.15.7:0/2969312311 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0b0d60 con 0x7f21cc0c4020 2023-11-28T18:50:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21c37fe640 1 -- 172.21.15.7:0/2969312311 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc0b9430 con 0x7f21cc0c4020 2023-11-28T18:50:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21bc009420 con 0x7f21cc0c4020 2023-11-28T18:50:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21c37fe640 1 -- 172.21.15.7:0/2969312311 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0b97e0 con 0x7f21cc0c4020 2023-11-28T18:50:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc155400 con 0x7f21cc0c4020 2023-11-28T18:50:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc04e890 con 0x7f21cc0c4020 2023-11-28T18:50:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.354+0000 7f21c37fe640 1 -- 172.21.15.7:0/2969312311 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f21bc0b7040 con 0x7f21cc0c4020 2023-11-28T18:50:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.358+0000 7f21c37fe640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21ac0668d0 0x7f21ac068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.358+0000 7f21c37fe640 1 -- 172.21.15.7:0/2969312311 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f21bc122400 con 0x7f21cc0c4020 2023-11-28T18:50:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.358+0000 7f21c3fff640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21ac0668d0 0x7f21ac068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T18:50:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.358+0000 7f21c3fff640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21ac0668d0 0x7f21ac068d90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f21cc154a20 tx=0x7f21b4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T18:50:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.358+0000 7f21c37fe640 1 -- 172.21.15.7:0/2969312311 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f21bc0ef130 con 0x7f21cc0c4020 2023-11-28T18:50:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21cc06e950 con 0x7f21cc0c4020 2023-11-28T18:50:27.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21c37fe640 1 -- 172.21.15.7:0/2969312311 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5822 (secure 0 0 0) 0x7f21bc0b5090 con 0x7f21cc0c4020 2023-11-28T18:50:27.550 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-28T18:50:27.551 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T18:23:08.041625+0000","modified":"2023-11-28T18:46:39.535710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5559,"mds_1":5714,"mds_2":6049,"mds_3":5734,"mds_4":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":35,"addr":"172.21.15.7:6838/4097424196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":4097424196},{"type":"v1","addr":"172.21.15.7:6838","nonce":4097424196}]},"join_fscid":-1,"laggy_since":"2023-11-28T18:45:57.508800+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5700":{"gid":5700,"name":"e","rank":4,"incarnation":60,"state":"up:active","state_seq":29,"addr":"172.21.15.7:6839/4048397809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6837","nonce":4048397809},{"type":"v1","addr":"172.21.15.7:6839","nonce":4048397809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5714":{"gid":5714,"name":"f","rank":1,"incarnation":102,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.193:6835/1637641802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1637641802},{"type":"v1","addr":"172.21.15.193:6835","nonce":1637641802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5734":{"gid":5734,"name":"b","rank":3,"incarnation":66,"state":"up:active","state_seq":29,"addr":"172.21.15.193:6836/1268741263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6833","nonce":1268741263},{"type":"v1","addr":"172.21.15.193:6836","nonce":1268741263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6049":{"gid":6049,"name":"d","rank":2,"incarnation":93,"state":"up:active","state_seq":23,"addr":"172.21.15.193:6837/1591261140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1591261140},{"type":"v1","addr":"172.21.15.193:6837","nonce":1591261140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T18:50:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 108 2023-11-28T18:50:27.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21ac0668d0 msgr2=0x7f21ac068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:27.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21ac0668d0 0x7f21ac068d90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f21cc154a20 tx=0x7f21b4009190 comp rx=0 tx=0).stop 2023-11-28T18:50:27.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21cc0c4020 msgr2=0x7f21cc1de360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T18:50:27.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21cc0c4020 0x7f21cc1de360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0b3040 tx=0x7f21bc01ba10 comp rx=0 tx=0).stop 2023-11-28T18:50:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 shutdown_connections 2023-11-28T18:50:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:6832/264069339,v1:172.21.15.7:6833/264069339] conn(0x7f21ac0668d0 0x7f21ac068d90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f21cc1575c0 0x7f21cc1556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21cc150b30 0x7f21cc1ebfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 --2- 172.21.15.7:0/2969312311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21cc0c4020 0x7f21cc1de360 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 >> 172.21.15.7:0/2969312311 conn(0x7f21cc0662e0 msgr2=0x7f21cc0c8420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T18:50:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 shutdown_connections 2023-11-28T18:50:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:27.550+0000 7f21d10fe640 1 -- 172.21.15.7:0/2969312311 wait complete. 2023-11-28T18:50:27.572 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-11-28T18:50:27.582 INFO:tasks.ceph.osd.0:Sent signal 15 2023-11-28T18:50:27.583 INFO:tasks.ceph.osd.1:Sent signal 15 2023-11-28T18:50:27.583 INFO:tasks.ceph.osd.2:Sent signal 15 2023-11-28T18:50:27.583 INFO:tasks.ceph.osd.3:Sent signal 15 2023-11-28T18:50:27.584 INFO:tasks.ceph.osd.4:Sent signal 15 2023-11-28T18:50:27.584 INFO:tasks.ceph.osd.5:Sent signal 15 2023-11-28T18:50:27.584 INFO:tasks.ceph.osd.6:Sent signal 15 2023-11-28T18:50:27.585 INFO:tasks.ceph.osd.7:Sent signal 15 2023-11-28T18:50:27.585 INFO:tasks.ceph.mds.e:Sent signal 15 2023-11-28T18:50:27.585 INFO:tasks.ceph.mds.b:Sent signal 15 2023-11-28T18:50:27.585 INFO:tasks.ceph.mds.d:Sent signal 15 2023-11-28T18:50:27.586 INFO:tasks.ceph.mds.f:Sent signal 15 2023-11-28T18:50:27.586 INFO:tasks.ceph.mon.a:Sent signal 15 2023-11-28T18:50:27.586 INFO:tasks.ceph.mon.b:Sent signal 15 2023-11-28T18:50:27.586 INFO:tasks.ceph.mon.c:Sent signal 15 2023-11-28T18:50:27.587 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-11-28T18:50:27.587 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-11-28T18:50:27.588 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f74589c5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38537) UID: 0 2023-11-28T18:50:27.588 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f74589c5640 -1 osd.4 34 *** Got signal Terminated *** 2023-11-28T18:50:27.588 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f74589c5640 -1 osd.4 34 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T18:50:27.589 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-28T18:50:27.582+0000 7fe7bc9ec640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 38279) UID: 0 2023-11-28T18:50:27.589 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-28T18:50:27.582+0000 7fe7bc9ec640 -1 osd.0 34 *** Got signal Terminated *** 2023-11-28T18:50:27.589 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-28T18:50:27.582+0000 7fe7bc9ec640 -1 osd.0 34 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T18:50:27.590 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-28T18:50:27.582+0000 7f4f43953640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38281) UID: 0 2023-11-28T18:50:27.590 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-28T18:50:27.582+0000 7f4f43953640 -1 osd.1 34 *** Got signal Terminated *** 2023-11-28T18:50:27.590 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-28T18:50:27.582+0000 7f4f43953640 -1 osd.1 34 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T18:50:27.590 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fe238e12640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38538) UID: 0 2023-11-28T18:50:27.590 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fe238e12640 -1 osd.5 34 *** Got signal Terminated *** 2023-11-28T18:50:27.591 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fe238e12640 -1 osd.5 34 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T18:50:27.591 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f8a05edb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38541) UID: 0 2023-11-28T18:50:27.591 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f8a05edb640 -1 osd.6 34 *** Got signal Terminated *** 2023-11-28T18:50:27.591 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f8a05edb640 -1 osd.6 34 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T18:50:27.592 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-28T18:50:27.582+0000 7f815132b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 38282) UID: 0 2023-11-28T18:50:27.592 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-28T18:50:27.582+0000 7f815132b640 -1 osd.2 34 *** Got signal Terminated *** 2023-11-28T18:50:27.592 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-28T18:50:27.582+0000 7f815132b640 -1 osd.2 34 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T18:50:27.593 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f3054ab0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38543) UID: 0 2023-11-28T18:50:27.593 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f3054ab0640 -1 osd.7 34 *** Got signal Terminated *** 2023-11-28T18:50:27.593 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f3054ab0640 -1 osd.7 34 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T18:50:27.593 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-28T18:50:27.582+0000 7fb422acc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38285) UID: 0 2023-11-28T18:50:27.593 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-28T18:50:27.582+0000 7fb422acc640 -1 osd.3 34 *** Got signal Terminated *** 2023-11-28T18:50:27.593 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-28T18:50:27.582+0000 7fb422acc640 -1 osd.3 34 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T18:50:27.594 INFO:tasks.ceph.mds.b.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fdef9047640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i b (PID: 44414) UID: 0 2023-11-28T18:50:27.594 INFO:tasks.ceph.mds.b.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fdef9047640 -1 mds.b *** got signal Terminated *** 2023-11-28T18:50:27.594 INFO:tasks.ceph.mds.e.smithi007.stderr:2023-11-28T18:50:27.586+0000 7f425d79e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i e (PID: 51120) UID: 0 2023-11-28T18:50:27.595 INFO:tasks.ceph.mds.e.smithi007.stderr:2023-11-28T18:50:27.586+0000 7f425d79e640 -1 mds.e *** got signal Terminated *** 2023-11-28T18:50:27.595 INFO:tasks.ceph.mds.d.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fd3e0b3c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i d (PID: 44570) UID: 0 2023-11-28T18:50:27.595 INFO:tasks.ceph.mds.d.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fd3e0b3c640 -1 mds.d *** got signal Terminated *** 2023-11-28T18:50:27.596 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-11-28T18:50:27.586+0000 7f5aabd4e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 38166) UID: 0 2023-11-28T18:50:27.596 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-11-28T18:50:27.586+0000 7f5aabd4e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-28T18:50:27.596 INFO:tasks.ceph.mds.f.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fc7df7ab640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i f (PID: 44612) UID: 0 2023-11-28T18:50:27.596 INFO:tasks.ceph.mds.f.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fc7df7ab640 -1 mds.f *** got signal Terminated *** 2023-11-28T18:50:27.597 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:50:27.586+0000 7fd84daa9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster ceph -i y (PID: 38171) UID: 0 2023-11-28T18:50:27.597 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-28T18:50:27.586+0000 7fd84daa9640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-28T18:50:27.597 INFO:tasks.ceph.mon.b.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fa6626ad640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 38217) UID: 0 2023-11-28T18:50:27.597 INFO:tasks.ceph.mon.b.smithi193.stderr:2023-11-28T18:50:27.583+0000 7fa6626ad640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-11-28T18:50:27.598 INFO:tasks.ceph.mon.c.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f8988246640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 38219) UID: 0 2023-11-28T18:50:27.598 INFO:tasks.ceph.mon.c.smithi193.stderr:2023-11-28T18:50:27.583+0000 7f8988246640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-11-28T18:50:27.789 INFO:tasks.ceph.mgr.x.smithi193.stderr:daemon-helper: command crashed with signal 15 2023-11-28T18:50:29.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T18:50:29.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba895d1640 1 Processor -- start 2023-11-28T18:50:29.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba895d1640 1 -- start start 2023-11-28T18:50:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba895d1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 0x7fba840ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba895d1640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba840cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba895d1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8415e380 con 0x7fba8415a8a0 2023-11-28T18:50:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba895d1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba841540e0 2023-11-28T18:50:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba895d1640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba840cf6c0 2023-11-28T18:50:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 msgr2=0x7fba840ca4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 0x7fba840ca4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:29.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba8415dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:29.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba840cfb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:29.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.729+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba840cfb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:29.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.933+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 msgr2=0x7fba840ca4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:29.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.933+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 0x7fba840ca4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:29.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.933+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba840cfb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:29.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.933+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba840cfb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:29.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:29.933+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba8415dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:30.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:30.337+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 msgr2=0x7fba840ca4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:30.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:30.337+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 0x7fba840ca4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:30.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:30.337+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba8415dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:30.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:30.337+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba840cfb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:30.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:30.337+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba840cfb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:31.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:31.141+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 msgr2=0x7fba840ca4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:31.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:31.141+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 0x7fba840ca4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:31.141+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba8415dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:31.141+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba840cfb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:31.141+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba840cfb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.729+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba840cfb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.729+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba840cfb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.729+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 msgr2=0x7fba840ca4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.729+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba841540e0 0x7fba840ca4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.729+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba8415dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 0x7fba64009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.733+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba640036d0 2023-11-28T18:50:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.733+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64007380 2023-11-28T18:50:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.733+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.733+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 msgr2=0x7fba64009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.733+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 0x7fba64009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:32.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.937+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:32.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.937+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 msgr2=0x7fba64009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:32.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:32.937+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 0x7fba64009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:33.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:33.341+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:33.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:33.341+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 msgr2=0x7fba64009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:33.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:33.341+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 0x7fba64009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:34.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:34.145+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:34.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:34.145+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 msgr2=0x7fba64009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:34.145+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 0x7fba64009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:35.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 msgr2=0x7fba64009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:35.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007380 0x7fba64009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:35.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:35.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:50:35.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba64009ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:35.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400af60 con 0x7fba640036d0 2023-11-28T18:50:35.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba8415a8a0 2023-11-28T18:50:35.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba840cf6c0 2023-11-28T18:50:35.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:35.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba64009ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:35.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:35.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.733+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba64009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.937+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.937+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba64009ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.937+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:35.937+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba64009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:36.341+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:36.341+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba64009ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:36.341+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:36.341+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba64009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:37.145+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:37.145+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba64009ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:37.145+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:37.145+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba64009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:38.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 msgr2=0x7fba64009ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:38.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba840cf6c0 0x7fba64009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:38.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:38.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640036d0 legacy=0x7fba64005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:38.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:38.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:38.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba640108e0 2023-11-28T18:50:38.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64007830 2023-11-28T18:50:38.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:38.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:38.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:38.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.733+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:38.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.937+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:38.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.937+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:38.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.937+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:38.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:38.937+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:39.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:39.341+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:39.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:39.341+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:39.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:39.341+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:39.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:39.341+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:40.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:40.145+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:40.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:40.145+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:40.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:40.145+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:40.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:40.145+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:41.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:41.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:41.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:41.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:41.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:50:41.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:41.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:41.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba8415a8a0 2023-11-28T18:50:41.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba640108e0 2023-11-28T18:50:41.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64007830 2023-11-28T18:50:41.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:41.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:41.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:41.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:41.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.733+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:41.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.937+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:41.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.937+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.937+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.937+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:41.937+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:42.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:50:42.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:50:42.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:42.341+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:42.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:42.341+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:42.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:42.341+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:42.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:42.341+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:42.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:42.341+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:43.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:43.145+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:43.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:43.145+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:43.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:43.145+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:43.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:43.145+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:43.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:43.145+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 msgr2=0x7fba64010280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64007830 0x7fba64010280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 msgr2=0x7fba64012d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba640108e0 0x7fba64012d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400af60 con 0x7fba64009f10 2023-11-28T18:50:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006990 2023-11-28T18:50:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64009f10 legacy=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.733+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:44.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.937+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64009f10 legacy=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:44.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.937+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:44.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:44.937+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:45.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:45.341+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64009f10 legacy=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:45.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:45.341+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:45.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:45.341+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:46.145+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64009f10 legacy=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:46.145+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:46.145+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64009f10 legacy=0x7fba64006280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:50:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006dd0 2023-11-28T18:50:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400af60 con 0x7fba8415a8a0 2023-11-28T18:50:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006990 2023-11-28T18:50:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.733+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.937+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.937+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:47.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.937+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:47.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:47.937+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:48.341+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:48.341+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:48.341+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:48.341+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:49.145+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:49.145+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:49.145+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:49.145+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba8415a8a0 legacy=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 0x7fba64005710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 0x7fba64003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64010280 2023-11-28T18:50:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64003ab0 2023-11-28T18:50:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 msgr2=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 0x7fba64005710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 msgr2=0x7fba64003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.733+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 0x7fba64003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:50.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.937+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 msgr2=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:50.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.937+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 0x7fba64005710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:50.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.937+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 msgr2=0x7fba64003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:50.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:50.937+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 0x7fba64003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:51.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:51.341+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 msgr2=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:51.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:51.341+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 0x7fba64005710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:51.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:51.341+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 msgr2=0x7fba64003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:51.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:51.341+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 0x7fba64003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:52.145+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 msgr2=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:52.145+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 0x7fba64005710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:52.145+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 msgr2=0x7fba64003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:52.145+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 0x7fba64003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:53.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 msgr2=0x7fba64003e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:53.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64003ab0 0x7fba64003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:53.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 msgr2=0x7fba64005710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:53.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64010280 0x7fba64005710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:53.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:53.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:50:53.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:53.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba64003ab0 2023-11-28T18:50:53.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006dd0 2023-11-28T18:50:53.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64006990 2023-11-28T18:50:53.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:53.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64003ab0 legacy=0x7fba64010cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:53.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:53.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:53.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.733+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.937+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64003ab0 legacy=0x7fba64010cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:53.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.937+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:53.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.937+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:53.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.937+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:53.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:53.937+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:54.341+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:54.341+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:54.341+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64003ab0 legacy=0x7fba64010cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:54.341+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:54.341+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:55.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:55.145+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:55.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:55.145+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64003ab0 legacy=0x7fba64010cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:50:55.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:55.145+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:50:55.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:55.145+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:55.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:55.145+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:56.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64006280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:56.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64006280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:56.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:56.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:56.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64003ab0 legacy=0x7fba64010cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:56.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:56.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba640123e0 2023-11-28T18:50:56.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400e270 2023-11-28T18:50:56.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:56.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:56.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.737+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:56.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.941+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:56.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.941+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:56.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:56.941+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:57.345+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:57.345+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:57.345+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:50:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:58.149+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:58.149+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:58.149+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:50:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:50:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:50:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:50:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:50:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba640123e0 2023-11-28T18:50:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400e270 2023-11-28T18:50:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64006dd0 2023-11-28T18:50:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.737+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:50:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.941+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:50:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.941+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:50:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.941+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:50:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.941+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:50:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:50:59.941+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:00.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:00.345+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:00.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:00.345+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:00.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:00.345+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:00.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:00.345+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:00.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:00.345+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:01.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:01.149+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:01.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:01.149+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:01.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:01.149+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:01.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:01.149+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:01.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:01.149+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006dd0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640123e0 legacy=0x7fba640127c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 0x7fba64006260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006990 2023-11-28T18:51:02.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64003ab0 2023-11-28T18:51:02.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006990 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:02.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 msgr2=0x7fba64006260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:02.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.737+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 0x7fba64006260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:02.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.941+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 msgr2=0x7fba64006260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:02.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.941+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006990 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:02.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:02.941+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 0x7fba64006260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:03.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:03.345+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006990 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:03.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:03.345+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 msgr2=0x7fba64006260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:03.345+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 0x7fba64006260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:04.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:04.149+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006990 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:04.149+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 msgr2=0x7fba64006260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:04.149+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 0x7fba64006260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 msgr2=0x7fba64006260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64003ab0 0x7fba64006260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006990 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:05.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:05.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:05.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:05.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba64003ab0 2023-11-28T18:51:05.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400e270 2023-11-28T18:51:05.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64006990 2023-11-28T18:51:05.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:05.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64003ab0 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.737+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:05.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.941+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64003ab0 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:05.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.941+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:05.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.941+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:05.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.941+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:05.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:05.941+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:06.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:06.345+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64003ab0 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:06.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:06.345+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:06.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:06.345+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:06.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:06.345+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:06.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:06.345+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:07.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:07.149+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64003ab0 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:07.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:07.149+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:07.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:07.149+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:07.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:07.149+0000 7fba837fe640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:07.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:07.149+0000 7fba837fe640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:08.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:08.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:08.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:08.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:08.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64003ab0 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:08.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 0x7fba6400f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:08.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006dd0 2023-11-28T18:51:08.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006260 2023-11-28T18:51:08.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:08.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:08.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.737+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.941+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.941+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:08.941+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:09.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:09.345+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:09.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:09.345+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:09.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:09.345+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:10.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:10.149+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:10.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:10.149+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:10.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:10.149+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:11.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:11.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64006260 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:11.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:11.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:11.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:11.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:11.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba64006dd0 2023-11-28T18:51:11.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400e270 2023-11-28T18:51:11.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64006990 2023-11-28T18:51:11.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:11.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:11.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:11.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:11.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.737+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:11.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.941+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:11.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.941+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:11.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.941+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:11.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.941+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:11.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:11.941+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:12.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:51:12.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:51:12.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:12.345+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:12.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:12.345+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:12.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:12.345+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:12.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:12.345+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:12.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:12.345+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:13.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:13.149+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:13.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:13.149+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:13.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:13.149+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:13.149+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:13.149+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:14.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 msgr2=0x7fba6400e650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e270 0x7fba6400e650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006260 2023-11-28T18:51:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64004ac0 2023-11-28T18:51:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.737+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.941+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.941+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:14.941+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:15.345+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:15.345+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:15.345+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:16.149+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:16.149+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:16.149+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba64006260 2023-11-28T18:51:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64004ac0 2023-11-28T18:51:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64006990 2023-11-28T18:51:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:18.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.737+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:18.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.941+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:18.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.941+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:18.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.941+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:18.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.941+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:18.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:17.941+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:18.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:18.345+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:18.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:18.345+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:18.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:18.345+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:18.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:18.345+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:18.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:18.345+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:19.149+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:19.149+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:19.149+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:19.149+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:19.149+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:20.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:20.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:20.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:20.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba64004ac0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:20.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400eb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:20.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 0x7fba6400d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:20.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006dd0 2023-11-28T18:51:20.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400d1e0 2023-11-28T18:51:20.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:20.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 0x7fba6400d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:20.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.737+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:20.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.941+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:20.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.941+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 0x7fba6400d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:20.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:20.941+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:21.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:21.345+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:21.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:21.345+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 0x7fba6400d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:21.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:21.345+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:22.149+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:22.149+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 0x7fba6400d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:22.149+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400d660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d1e0 0x7fba6400d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba64006260 2023-11-28T18:51:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006dd0 2023-11-28T18:51:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006990 2023-11-28T18:51:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.737+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:23.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.941+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:23.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.941+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:23.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.941+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:23.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:23.941+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:24.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:24.344+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:24.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:24.344+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:24.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:24.344+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:24.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:24.344+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:25.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:25.148+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:25.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:25.148+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:25.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:25.148+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:25.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:25.148+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:26.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:26.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400d600 2023-11-28T18:51:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba6400d1e0 2023-11-28T18:51:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.740+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:26.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.944+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:26.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.944+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:26.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.944+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:26.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:26.944+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:27.348+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:27.348+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:27.348+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:27.348+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:28.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:28.152+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:28.152+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:28.152+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:28.152+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:29.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400ebb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400af60 con 0x7fba64006260 2023-11-28T18:51:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba64006dd0 2023-11-28T18:51:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400d1e0 2023-11-28T18:51:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.740+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:29.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.944+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:29.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.944+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:29.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.944+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:29.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:29.944+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:30.348+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:30.348+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:30.348+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:30.348+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:31.152+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:31.152+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:31.152+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:31.152+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:32.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:32.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400f800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:32.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba6400f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:32.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400d600 2023-11-28T18:51:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64006990 2023-11-28T18:51:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400fd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.740+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.944+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.944+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.944+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400fd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:32.944+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:33.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:33.348+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:33.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:33.348+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:33.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:33.348+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400fd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:33.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:33.348+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:34.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:34.152+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:34.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:34.152+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:34.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:34.152+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400fd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:34.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:34.152+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:35.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba6400fd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:35.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba6400fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:35.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:35.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400d600 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:35.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:35.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:35.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:35.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006dd0 2023-11-28T18:51:35.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400e9f0 2023-11-28T18:51:35.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba6400d1e0 2023-11-28T18:51:35.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:35.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:35.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:35.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:35.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.740+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.944+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.944+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.944+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.944+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:35.944+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:36.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:36.348+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:36.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:36.348+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:36.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:36.348+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:36.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:36.348+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:36.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:36.348+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:37.152+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:37.152+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:37.152+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:37.152+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:37.152+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:38.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:38.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:38.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:38.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:38.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:38.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba64003ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:38.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400af60 con 0x7fba64006990 2023-11-28T18:51:38.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006260 2023-11-28T18:51:38.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006990 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:38.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:38.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.740+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:38.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.944+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006990 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:38.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.944+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:38.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:38.944+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:39.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:39.348+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006990 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:39.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:39.348+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:39.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:39.348+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:40.152+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006990 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:40.152+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:40.152+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:41.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:41.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:41.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006990 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:41.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:41.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:41.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:41.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006dd0 2023-11-28T18:51:41.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400e9f0 2023-11-28T18:51:41.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba6400d1e0 2023-11-28T18:51:41.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:41.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:41.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:41.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:41.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.740+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.944+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.944+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.944+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.944+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:41.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:41.944+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:42.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:51:42.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:51:42.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:42.348+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:42.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:42.348+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:42.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:42.348+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:42.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:42.348+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:42.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:42.348+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:43.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:43.152+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:43.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:43.152+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:43.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:43.152+0000 7fba837fe640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:43.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:43.152+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:43.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:43.152+0000 7fba837fe640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640051d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba64005710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64003ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:44.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400af60 con 0x7fba64006260 2023-11-28T18:51:44.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006990 2023-11-28T18:51:44.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:44.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:44.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.740+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:44.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.944+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:44.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.944+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:44.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:44.944+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:45.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:45.348+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:45.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:45.348+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:45.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:45.348+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:46.152+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:46.152+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:46.152+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64003ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006260 2023-11-28T18:51:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400e9f0 2023-11-28T18:51:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba6400d1e0 2023-11-28T18:51:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.740+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.944+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.944+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.944+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.944+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:47.944+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:48.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:48.348+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:48.348+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:48.348+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:48.348+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:48.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:48.348+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:49.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:49.152+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:49.152+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:49.152+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:49.152+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:49.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:49.152+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba6400e9f0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400af60 con 0x7fba64006dd0 2023-11-28T18:51:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006990 2023-11-28T18:51:50.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:50.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:50.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.740+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:50.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.944+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:50.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.944+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:50.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:50.944+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:51.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:51.348+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:51.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:51.348+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:51.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:51.348+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:52.152+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:52.152+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:52.152+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006990 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006260 2023-11-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400af60 con 0x7fba64006dd0 2023-11-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400d1e0 2023-11-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.744+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:53.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.948+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:53.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.948+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:53.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.948+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:53.948+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:54.352+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:54.352+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:54.352+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:54.352+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:55.156+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:55.156+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:55.156+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:51:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:55.156+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:56.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006dd0 legacy=0x7fba640055e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 0x7fba64004ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 0x7fba640050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006960 2023-11-28T18:51:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba6400fde0 2023-11-28T18:51:56.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:56.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:56.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:56.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.744+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:56.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.948+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:56.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.948+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:56.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.948+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:56.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:56.948+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:57.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:57.352+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:57.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:57.352+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:57.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:57.352+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:57.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:57.352+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:51:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:58.156+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:58.156+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:58.156+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:51:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:58.156+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:51:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 msgr2=0x7fba640050a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fde0 0x7fba640050a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 msgr2=0x7fba64004ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:51:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006960 0x7fba64004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:51:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:51:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba6400f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:51:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba64006260 2023-11-28T18:51:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006dd0 2023-11-28T18:51:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba6400d1e0 2023-11-28T18:51:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400f100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba6400f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:59.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:59.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.744+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:51:59.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.948+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400f100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:51:59.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.948+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:51:59.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.948+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba6400f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:51:59.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.948+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:51:59.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:51:59.948+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:00.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:00.352+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400f100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:00.352+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:00.352+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba6400f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:00.352+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:00.352+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:01.156+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400f100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:01.156+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:01.156+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba6400f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:01.156+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:01.156+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:02.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:02.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:02.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba6400f100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:02.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba6400f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:02.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64006260 legacy=0x7fba6400d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:02.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:02.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64006960 2023-11-28T18:52:02.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400fdf0 2023-11-28T18:52:02.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:02.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006960 legacy=0x7fba6400f640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:02.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.744+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:02.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.948+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:02.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.948+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006960 legacy=0x7fba6400f640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:02.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:02.948+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:03.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:03.352+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:03.352+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006960 legacy=0x7fba6400f640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:03.352+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:04.156+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:04.156+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006960 legacy=0x7fba6400f640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:04.156+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:05.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:05.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:05.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006960 legacy=0x7fba6400f640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:52:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba64006260 2023-11-28T18:52:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400d1e0 2023-11-28T18:52:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba6400fdf0 2023-11-28T18:52:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006260 legacy=0x7fba6400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.744+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.948+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006260 legacy=0x7fba6400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.948+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.948+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.948+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:05.948+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:06.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:06.352+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:06.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:06.352+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:06.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:06.352+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006260 legacy=0x7fba6400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:06.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:06.352+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:06.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:06.352+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:07.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:07.156+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:07.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:07.156+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006260 legacy=0x7fba6400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:07.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:07.156+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:07.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:07.156+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:07.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:07.156+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:08.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 msgr2=0x7fba6400c1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:08.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba6400fdf0 0x7fba6400c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:08.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:08.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:08.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba64006260 legacy=0x7fba6400c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:08.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:08.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba64005ac0 2023-11-28T18:52:08.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006dd0 2023-11-28T18:52:08.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64005ac0 legacy=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:08.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:08.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.744+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:08.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.948+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64005ac0 legacy=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:08.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.948+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:08.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:08.948+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:09.352+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64005ac0 legacy=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:09.352+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:09.352+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:10.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:10.156+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64005ac0 legacy=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:10.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:10.156+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:10.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:10.156+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:11.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:11.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:11.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba64005ac0 legacy=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:11.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:52:11.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:11.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:11.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba640098d0 2023-11-28T18:52:11.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006dd0 2023-11-28T18:52:11.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba6400d1e0 2023-11-28T18:52:11.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640098d0 legacy=0x7fba64019000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.744+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:11.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.948+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:11.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.948+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640098d0 legacy=0x7fba64019000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:11.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.948+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:11.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.948+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:11.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:11.948+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:12.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:12.352+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:12.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:12.352+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640098d0 legacy=0x7fba64019000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:12.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:12.352+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:12.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:12.352+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:12.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:12.352+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:12.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:52:12.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:52:13.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:13.156+0000 7fba837fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:13.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:13.156+0000 7fba82ffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640098d0 legacy=0x7fba64019000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:13.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:13.156+0000 7fba837fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:13.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:13.156+0000 7fba7bfff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:13.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:13.156+0000 7fba7bfff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba640098d0 legacy=0x7fba64019000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba640068d0 2023-11-28T18:52:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006260 2023-11-28T18:52:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.744+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:14.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.948+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:14.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.948+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:14.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:14.948+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:15.352+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:15.352+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:15.352+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:16.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:16.156+0000 7fba7bfff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:16.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:16.156+0000 7fba82ffd640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:16.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:16.156+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:17.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:17.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:17.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:17.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:17.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:17.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:52:17.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba640068d0 2023-11-28T18:52:17.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400d1e0 2023-11-28T18:52:17.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64006260 2023-11-28T18:52:17.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:17.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:17.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:17.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.744+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.948+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.948+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.948+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.948+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:17.948+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:18.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:18.352+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:18.352+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:18.352+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:18.352+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:18.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:18.352+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:19.156+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:19.156+0000 7fba837fe640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:19.156+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:19.156+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:19.156+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:20.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:20.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:20.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:20.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:20.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba817fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba640086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:20.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:20.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba6400c1c0 2023-11-28T18:52:20.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006dd0 2023-11-28T18:52:20.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:20.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:20.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.744+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:20.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.948+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:20.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.948+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:20.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:20.948+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:21.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:21.352+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:21.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:21.352+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:21.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:21.352+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:22.156+0000 7fba82ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:22.156+0000 7fba7bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:22.156+0000 7fba82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba817fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba837fe640 1 -- reap_dead start 2023-11-28T18:52:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba817fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840cb850 con 0x7fba6400c1c0 2023-11-28T18:52:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba817fa640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba6400d1e0 2023-11-28T18:52:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba817fa640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840cab80 con 0x7fba64006260 2023-11-28T18:52:23.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:23.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:23.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:23.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:23.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.747+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.951+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.951+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.951+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.951+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:23.951+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:24.355+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:24.355+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:24.355+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:24.355+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:24.355+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:25.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:25.159+0000 7fba837fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:25.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:25.159+0000 7fba82ffd640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:25.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:25.159+0000 7fba7bfff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:25.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:25.159+0000 7fba82ffd640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:25.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:25.159+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba817fa640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 msgr2=0x7fba640081b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba817fa640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fba64006260 0x7fba640081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba817fa640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 msgr2=0x7fba6400a420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba817fa640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fba6400d1e0 0x7fba6400a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba817fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fba6400c1c0 legacy=0x7fba6400f9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba817fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba817fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6400acb0 con 0x7fba640068d0 2023-11-28T18:52:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba817fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415d7e0 con 0x7fba64006dd0 2023-11-28T18:52:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.747+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba6400c5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:26.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.951+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba6400c5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:26.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.951+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:26.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:26.951+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:27.355+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:27.355+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:27.355+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba6400c5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:28.159+0000 7fba7bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 msgr2=0x7fba64007c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:28.159+0000 7fba7bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba64006dd0 0x7fba64007c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:28.159+0000 7fba82ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fba640068d0 legacy=0x7fba6400c5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:52:29.608 ERROR:tasks.mds_thrash.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mds_thrash.py", line 124, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mds_thrash.py", line 312, in do_thrash status = self.wait_for_stable(rank, gid) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mds_thrash.py", line 188, in wait_for_stable status = self.fs.status() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 396, in status return FSStatus(self.mon_manager, epoch) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 88, in __init__ self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2023-11-28T18:52:29.622 INFO:tasks.mds_thrash:done joining 2023-11-28T18:52:29.622 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-11-28T18:52:29.636 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-11-28T18:52:29.636 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-28T18:52:29.636 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T18:52:29.652 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357768k,nr_inodes=4089442,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276004k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=927 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276000k,nr_inodes=819000,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276004k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2023-11-28T18:52:29.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:52:29.653 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2023-11-28T18:52:29.749 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T18:52:29.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:52:29.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T18:52:29.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T18:52:29.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2023-11-28T18:52:30.004 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-28T18:52:30.014 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T18:52:30.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T18:52:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.163+0000 7fa02cbf8640 1 Processor -- start 2023-11-28T18:52:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa02cbf8640 1 -- start start 2023-11-28T18:52:30.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa02cbf8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 0x7fa028157b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:30.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa02cbf8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa028151cd0 2023-11-28T18:52:30.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa02cbf8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa02806e560 2023-11-28T18:52:30.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa02cbf8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa02815bd00 2023-11-28T18:52:30.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 msgr2=0x7fa028157b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:30.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa0280c4e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:30.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 0x7fa028157b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:30.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.167+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa02806e560 legacy=0x7fa028160500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:30.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.371+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa0280c4e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:30.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.371+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 msgr2=0x7fa028157b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:30.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.371+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 0x7fa028157b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:30.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.371+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa02806e560 legacy=0x7fa028160500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:30.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.775+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa0280c4e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.775+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 msgr2=0x7fa028157b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.775+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 0x7fa028157b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:30.775+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa02806e560 legacy=0x7fa028160500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:31.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:31.579+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa0280c4e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:31.579+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 msgr2=0x7fa028157b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:31.579+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 0x7fa028157b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:31.579+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa02806e560 legacy=0x7fa028160500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:33.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 msgr2=0x7fa028157b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:33.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa02815bd00 0x7fa028157b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:33.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa02806e560 legacy=0x7fa028160500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:33.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa0280c4e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:33.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 0x7fa008005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:33.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 0x7fa0080085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:33.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080036b0 2023-11-28T18:52:33.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008006120 2023-11-28T18:52:33.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:33.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 msgr2=0x7fa008005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:33.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:33.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.171+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 0x7fa008005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:33.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.375+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:33.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.375+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 msgr2=0x7fa008005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:33.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.375+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:33.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.375+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 0x7fa008005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.779+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 msgr2=0x7fa008005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.779+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.779+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 0x7fa008005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:33.779+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:34.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:34.583+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:34.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:34.583+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 msgr2=0x7fa008005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:34.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:34.583+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:34.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:34.583+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 0x7fa008005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008006120 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 msgr2=0x7fa008005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa0080036b0 0x7fa008005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:36.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:52:36.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 0x7fa00800b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:36.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:36.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa028151cd0 2023-11-28T18:52:36.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008008ff0 2023-11-28T18:52:36.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008005b60 2023-11-28T18:52:36.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:36.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 msgr2=0x7fa00800b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:36.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:36.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:36.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.171+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 0x7fa00800b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.375+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.375+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.375+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 msgr2=0x7fa00800b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.375+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.375+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 0x7fa00800b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:36.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.779+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:36.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.779+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:36.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.779+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 msgr2=0x7fa00800b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:36.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.779+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:36.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:36.779+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 0x7fa00800b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:37.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:37.583+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:37.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:37.583+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:37.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:37.583+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 msgr2=0x7fa00800b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:37.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:37.583+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:37.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:37.583+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 0x7fa00800b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:39.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:39.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:39.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 msgr2=0x7fa00800b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:39.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008ff0 0x7fa00800b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:39.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:39.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:39.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa00800d220 2023-11-28T18:52:39.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080113a0 2023-11-28T18:52:39.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:39.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:39.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.171+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d220 legacy=0x7fa00800d690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.375+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.375+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.375+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d220 legacy=0x7fa00800d690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:39.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.779+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:39.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.779+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:39.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:39.779+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d220 legacy=0x7fa00800d690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:40.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:40.583+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:40.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:40.583+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:40.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:40.583+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d220 legacy=0x7fa00800d690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:42.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:42.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:42.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d220 legacy=0x7fa00800d690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:42.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:52:42.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:42.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:42.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa028151cd0 2023-11-28T18:52:42.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080113a0 2023-11-28T18:52:42.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008005b60 2023-11-28T18:52:42.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:42.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:42.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:42.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:42.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.171+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:42.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.375+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:42.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.375+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:42.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.375+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:42.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.375+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:42.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.375+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:42.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:52:42.427 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:52:42.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.779+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:42.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.779+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:42.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.779+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:42.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.779+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:42.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:42.779+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:43.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:43.583+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:43.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:43.583+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:43.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:43.583+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:43.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:43.583+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:43.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:43.583+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 msgr2=0x7fa0080085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008005b60 0x7fa0080085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa028151cd0 legacy=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 0x7fa00800faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa00800d690 2023-11-28T18:52:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800b3e0 2023-11-28T18:52:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 msgr2=0x7fa00800faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d690 legacy=0x7fa00800e380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.171+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 0x7fa00800faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:45.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.375+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 msgr2=0x7fa00800faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:45.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.375+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d690 legacy=0x7fa00800e380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:45.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.375+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 0x7fa00800faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:45.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.779+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 msgr2=0x7fa00800faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:45.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.779+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d690 legacy=0x7fa00800e380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:45.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:45.779+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 0x7fa00800faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:46.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:46.583+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 msgr2=0x7fa00800faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:46.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:46.583+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d690 legacy=0x7fa00800e380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:46.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:46.583+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 0x7fa00800faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 msgr2=0x7fa00800faa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800b3e0 0x7fa00800faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800d690 legacy=0x7fa00800e380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:52:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa00800b9c0 2023-11-28T18:52:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080113a0 2023-11-28T18:52:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa00800c4b0 2023-11-28T18:52:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.171+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.375+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.375+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.375+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:48.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.375+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:48.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.375+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:48.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.779+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:48.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.779+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:48.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.779+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:48.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.779+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:48.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:48.779+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:49.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:49.583+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:49.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:49.583+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:49.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:49.583+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:49.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:49.583+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:49.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:49.583+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 msgr2=0x7fa008013790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080113a0 0x7fa008013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 0x7fa008010a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa00800c8c0 2023-11-28T18:52:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008005b70 2023-11-28T18:52:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 msgr2=0x7fa008010a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.171+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 0x7fa008010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.375+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 msgr2=0x7fa008010a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.375+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.375+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 0x7fa008010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.779+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 msgr2=0x7fa008010a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.779+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:51.779+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 0x7fa008010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:52.583+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 msgr2=0x7fa008010a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:52.583+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:52.583+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 0x7fa008010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 msgr2=0x7fa008010a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008005b70 0x7fa008010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008010370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:52:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa00800b9c0 2023-11-28T18:52:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800c8c0 2023-11-28T18:52:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa00800c4b0 2023-11-28T18:52:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.171+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.375+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.375+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.375+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.375+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.375+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:54.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.779+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:54.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.779+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:54.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.779+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:54.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.779+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:54.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:54.779+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:55.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:55.583+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:52:55.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:55.583+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:52:55.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:55.583+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:55.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:55.583+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:52:55.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:55.583+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:52:57.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:57.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c4b0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:52:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:52:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:52:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa00800cca0 2023-11-28T18:52:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008013180 2023-11-28T18:52:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:57.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800cca0 legacy=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:57.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.171+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:52:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.375+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:57.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.375+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800cca0 legacy=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:57.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.375+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:52:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.779+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.779+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800cca0 legacy=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:57.779+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:52:58.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:58.583+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:52:58.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:58.583+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800cca0 legacy=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:52:58.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:52:58.583+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:00.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:00.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:00.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800cca0 legacy=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:00.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:00.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:00.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:00.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa00800b9c0 2023-11-28T18:53:00.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800c8c0 2023-11-28T18:53:00.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008013180 2023-11-28T18:53:00.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:00.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:00.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.175+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:00.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.379+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:00.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.379+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:00.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.379+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:00.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.379+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:00.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.379+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:00.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.783+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:00.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.783+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:00.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.783+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:00.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.783+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:00.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:00.783+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:01.587+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:01.587+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:01.587+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:01.587+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:01.587+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 msgr2=0x7fa0080155a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013180 0x7fa0080155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa008010370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa008010370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa0080108b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:03.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa00800c4b0 2023-11-28T18:53:03.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800cca0 2023-11-28T18:53:03.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:03.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:03.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.175+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa0080108b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.379+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.379+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.379+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa0080108b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:03.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.783+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:03.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.783+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:03.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:03.783+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa0080108b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:04.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:04.587+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:04.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:04.587+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:04.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:04.587+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa0080108b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa0080108b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa0080108b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa0080085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa00800b9c0 2023-11-28T18:53:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa00800c4b0 2023-11-28T18:53:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800c8c0 2023-11-28T18:53:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.175+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:06.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.379+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:06.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.379+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:06.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.379+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:06.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.379+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.783+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.783+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.783+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:06.783+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:07.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:07.587+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:07.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:07.587+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:07.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:07.587+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:07.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:07.587+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:09.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa0080085c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:09.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa0080085c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c4b0 legacy=0x7fa008015c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800b9c0 legacy=0x7fa00800bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa008016370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800cca0 2023-11-28T18:53:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008010380 2023-11-28T18:53:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa008016370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa008016370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.175+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.379+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa008016370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.379+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.379+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa008016370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.379+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:09.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.783+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa008016370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:09.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.783+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:09.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.783+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa008016370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:09.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:09.783+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:10.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:10.587+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa008016370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:10.587+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:10.587+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa008016370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:10.587+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 msgr2=0x7fa008016370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800cca0 0x7fa008016370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa00800bf00 2023-11-28T18:53:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800c8c0 2023-11-28T18:53:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008010380 2023-11-28T18:53:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa026d9d640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.175+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:12.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.379+0000 7fa026d9d640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:12.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.379+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:12.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.379+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:12.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.379+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:12.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.379+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:12.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:53:12.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:53:12.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.783+0000 7fa026d9d640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:12.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.783+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:12.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.783+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:12.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.783+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:12.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:12.783+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:13.587+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:13.587+0000 7fa026d9d640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:13.587+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:13.587+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:13.587+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:15.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:15.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:15.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:15.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:15.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:15.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:15.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa008004000 2023-11-28T18:53:15.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800c310 2023-11-28T18:53:15.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:15.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:15.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.175+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:15.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.379+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:15.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.379+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.379+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.783+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.783+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:15.783+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:16.587+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:16.587+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:16.587+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:18.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:18.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:18.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:18.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:18.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:18.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:18.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa008004000 2023-11-28T18:53:18.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800c8c0 2023-11-28T18:53:18.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa00800c310 2023-11-28T18:53:18.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:18.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:18.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:18.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:18.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.175+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:18.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.379+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:18.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.379+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:18.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.379+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:18.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.379+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:18.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.379+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.783+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.783+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.783+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.783+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:18.783+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:19.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:19.587+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:19.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:19.587+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:19.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:19.587+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:19.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:19.587+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:19.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:19.587+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa00800bf00 2023-11-28T18:53:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008010380 2023-11-28T18:53:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.175+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:21.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.379+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:21.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.379+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:21.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.379+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.783+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.783+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:21.783+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:22.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:22.587+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:22.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:22.587+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:22.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:22.587+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800bf00 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800c8c0 2023-11-28T18:53:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa00800c310 2023-11-28T18:53:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.175+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00800bf00 con 0x7fa008010380 2023-11-28T18:53:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.179+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:24.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.179+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:24.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.179+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:24.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.179+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:24.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.179+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:24.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.179+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:24.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.383+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:24.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.383+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:24.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.383+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:24.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.383+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:24.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.383+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:24.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.383+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:24.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.787+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:24.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.787+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:24.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.787+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:24.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.787+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:24.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.787+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:24.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:24.787+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:25.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:25.591+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:25.591+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:25.591+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:25.591+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:25.591+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:25.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:25.591+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800c8c0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa008008580 2023-11-28T18:53:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa008004000 2023-11-28T18:53:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.178+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:27.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.382+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:27.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.382+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.786+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:27.786+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:28.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:28.590+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:28.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:28.590+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:30.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008004000 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:30.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa008008580 2023-11-28T18:53:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800c310 2023-11-28T18:53:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008010380 2023-11-28T18:53:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.178+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:30.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.382+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:30.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.382+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:30.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.382+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:30.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.382+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:30.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.382+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.786+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.786+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.786+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.786+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:30.786+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:31.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:31.590+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:31.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:31.590+0000 7fa026d9d640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:31.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:31.590+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:31.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:31.590+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:31.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:31.590+0000 7fa026d9d640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:33.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 msgr2=0x7fa008010780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:33.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008010380 0x7fa008010780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:33.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:33.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa00800c310 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:33.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008008580 legacy=0x7fa008015150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:33.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:33.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa00800c8c0 2023-11-28T18:53:33.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008004000 2023-11-28T18:53:33.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:33.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:33.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.178+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:33.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.382+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:33.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.382+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:33.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.382+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.786+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.786+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:33.786+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:34.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:34.590+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:34.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:34.590+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:34.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:34.590+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa00800c8c0 legacy=0x7fa008004410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa008010350 2023-11-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008004000 2023-11-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008008580 2023-11-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.178+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.382+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.382+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.382+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.382+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.382+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:36.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.786+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:36.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.786+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:36.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.786+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:36.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.786+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:36.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:36.786+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:37.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:37.590+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:37.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:37.590+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:37.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:37.590+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:37.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:37.590+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:37.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:37.590+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:39.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:39.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:39.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:39.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:39.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:39.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 0x7fa008005420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:39.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa008013070 2023-11-28T18:53:39.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008010780 2023-11-28T18:53:39.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa0080154f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:39.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 msgr2=0x7fa008005420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:39.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.178+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 0x7fa008005420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:39.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.382+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa0080154f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.382+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 msgr2=0x7fa008005420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.382+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 0x7fa008005420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:39.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.786+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa0080154f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:39.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.786+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 msgr2=0x7fa008005420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:39.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:39.786+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 0x7fa008005420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:40.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:40.590+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa0080154f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:40.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:40.590+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 msgr2=0x7fa008005420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:40.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:40.590+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 0x7fa008005420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:42.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 msgr2=0x7fa008005420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:42.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008010780 0x7fa008005420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:42.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa0080154f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:42.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:42.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa008017070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:42.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:42.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa008010350 2023-11-28T18:53:42.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008016c90 2023-11-28T18:53:42.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008008580 2023-11-28T18:53:42.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:42.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa008017070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:42.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:42.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa008017070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:42.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.178+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:42.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.382+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:42.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.382+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:42.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.382+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa008017070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:42.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.382+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:42.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.382+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa008017070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:42.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:53:42.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:53:42.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.786+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa008017070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.786+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.786+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.786+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa008017070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:42.786+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:43.590+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:43.590+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:43.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:43.590+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa008017070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:43.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:43.590+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa008017070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:43.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:43.590+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa008017070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa008017070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa008013070 2023-11-28T18:53:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008004000 2023-11-28T18:53:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa008005420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.178+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:45.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.382+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa008005420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:45.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.382+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:45.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.382+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:45.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.786+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa008005420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:45.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.786+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:45.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:45.786+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:46.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:46.590+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa008005420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:46.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:46.590+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:46.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:46.590+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008004000 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008013070 legacy=0x7fa008005420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa0080165c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa008010350 2023-11-28T18:53:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008016c90 2023-11-28T18:53:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008008580 2023-11-28T18:53:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa0080165c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa0080165c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.178+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:48.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.382+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:48.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.382+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa0080165c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:48.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.382+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:48.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.382+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:48.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.382+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa0080165c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:48.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.786+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:48.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.786+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:48.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.786+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa0080165c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:48.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.786+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:48.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:48.786+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa0080165c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:49.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:49.590+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:49.590+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:49.590+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa0080165c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:49.590+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:49.590+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa0080165c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa0080165c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa0080165c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa008004000 2023-11-28T18:53:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008013070 2023-11-28T18:53:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004000 legacy=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.178+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.382+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.382+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004000 legacy=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.382+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.786+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.786+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004000 legacy=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:51.786+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:52.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:52.590+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:52.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:52.590+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004000 legacy=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:52.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:52.590+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa008013070 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004000 legacy=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:53:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa008017070 2023-11-28T18:53:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028071e60 con 0x7fa008010350 2023-11-28T18:53:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008008580 2023-11-28T18:53:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa008019460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.178+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.382+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa008019460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.382+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.382+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.382+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:54.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.786+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa008019460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:54.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.786+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:54.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.786+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:54.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:54.786+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:55.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:55.590+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa008019460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:53:55.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:55.590+0000 7fa026d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:53:55.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:55.590+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:53:55.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:55.590+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:57.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:57.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:53:57.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008010350 legacy=0x7fa00800c0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:57.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa008019460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:53:57.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:57.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa0080053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:53:57.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080165f0 2023-11-28T18:53:57.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008016c90 2023-11-28T18:53:57.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:57.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:57.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:57.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.178+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:53:57.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.382+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.382+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.382+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.382+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:53:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.786+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.786+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.786+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:57.786+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:53:58.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:58.590+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:53:58.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:58.590+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:53:58.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:58.590+0000 7fa02659c640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:53:58.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:53:58.590+0000 7fa02659c640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:00.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.178+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:00.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008016c90 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:00.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.178+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:00.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:00.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.178+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:54:00.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:00.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.178+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:00.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.182+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa008017070 2023-11-28T18:54:00.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.182+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080165f0 2023-11-28T18:54:00.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.182+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008008580 2023-11-28T18:54:00.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.182+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:00.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.182+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:00.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.182+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:00.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.182+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:00.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.182+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:00.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.386+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.386+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.386+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.386+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.386+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:00.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.790+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:00.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.790+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:00.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.790+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:00.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.790+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:00.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:00.790+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:01.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:01.594+0000 7fa026d9d640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:01.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:01.594+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:01.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:01.594+0000 7fa026d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:01.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:01.594+0000 7fa01ffff640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:01.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:01.594+0000 7fa01ffff640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa008016c60 2023-11-28T18:54:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008013070 2023-11-28T18:54:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.182+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.386+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.386+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.386+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:03.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.790+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:03.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.790+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:03.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:03.790+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:04.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:04.594+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:04.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:04.594+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:04.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:04.594+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:06.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:54:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa008017070 2023-11-28T18:54:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080165f0 2023-11-28T18:54:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa008008580 2023-11-28T18:54:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.182+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:06.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.386+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:06.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.386+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:06.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.386+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:06.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.386+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:06.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.386+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.790+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.790+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.790+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.790+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:06.790+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:07.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:07.594+0000 7fa02659c640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:07.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:07.594+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:07.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:07.594+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:07.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:07.594+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:07.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:07.594+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:09.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008017070 legacy=0x7fa0080154b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:09.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:09.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa008016c60 2023-11-28T18:54:09.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008013070 2023-11-28T18:54:09.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:09.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:09.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.182+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.386+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.386+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.386+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:09.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.790+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:09.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.790+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:09.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:09.790+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:10.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:10.594+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:10.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:10.594+0000 7fa01ffff640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:10.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:10.594+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa008013070 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016c60 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008015bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa008016100 2023-11-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008008580 2023-11-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa0080102d0 2023-11-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008015bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016100 legacy=0x7fa00801c480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.182+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008015bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.386+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.386+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008015bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.386+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016100 legacy=0x7fa00801c480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.386+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.386+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008015bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:12.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:54:12.769 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.790+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.790+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008015bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.790+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016100 legacy=0x7fa00801c480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.790+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:12.790+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008015bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:13.594+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016100 legacy=0x7fa00801c480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:13.594+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008015bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:13.594+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:13.594+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:13.594+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008015bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008015bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008015bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008016100 legacy=0x7fa00801c480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa00800c040 2023-11-28T18:54:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080165f0 2023-11-28T18:54:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa008017070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:15.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.182+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.386+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa008017070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.386+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.386+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:15.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.790+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa008017070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:15.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.790+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:15.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:15.790+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:16.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:16.594+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:16.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:16.594+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa008017070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:16.594+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080165f0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa008017070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:54:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa0080053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa00800c040 2023-11-28T18:54:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008008580 2023-11-28T18:54:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa0080102d0 2023-11-28T18:54:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa0080043e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.182+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:18.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.386+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa0080043e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.386+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.386+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.386+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.386+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.790+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa0080043e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.790+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.790+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.790+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:18.790+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:19.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:19.594+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa0080043e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:19.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:19.594+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:19.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:19.594+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:19.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:19.594+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:19.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:19.594+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa0080053f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa0080053f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 msgr2=0x7fa008003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008008580 0x7fa008003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa024d99640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa00800c040 legacy=0x7fa0080043e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 0x7fa0080154a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa0080165f0 2023-11-28T18:54:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa00800b9c0 2023-11-28T18:54:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 msgr2=0x7fa0080154a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 0x7fa0080154a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.182+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa0080165f0 legacy=0x7fa008004af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:21.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.386+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 msgr2=0x7fa0080154a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:21.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.386+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 0x7fa0080154a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.386+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa0080165f0 legacy=0x7fa008004af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:21.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.790+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 msgr2=0x7fa0080154a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:21.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.790+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 0x7fa0080154a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:21.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:21.790+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa0080165f0 legacy=0x7fa008004af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:22.594+0000 7fa01ffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 msgr2=0x7fa0080154a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:22.594+0000 7fa01ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 0x7fa0080154a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:22.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:22.594+0000 7fa02659c640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa0080165f0 legacy=0x7fa008004af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa024d99640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 msgr2=0x7fa0080154a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:24.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa00800b9c0 0x7fa0080154a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:24.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa0080165f0 legacy=0x7fa008004af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:24.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:24.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa026d9d640 1 -- reap_dead start 2023-11-28T18:54:24.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 0x7fa00800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:24.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa024d99640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028070bf0 con 0x7fa008008580 2023-11-28T18:54:24.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa024d99640 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa008003ac0 2023-11-28T18:54:24.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa024d99640 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008008cb0 con 0x7fa0080102d0 2023-11-28T18:54:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008005200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008008580 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.182+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:24.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.386+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008005200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:24.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.386+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008008580 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:24.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.386+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:24.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.386+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:24.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.386+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:24.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.790+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008005200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:24.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.790+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.790+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008008580 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.790+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:24.790+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:25.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:25.594+0000 7fa026d9d640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3300/0 2023-11-28T18:54:25.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:25.594+0000 7fa01ffff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008008580 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2023-11-28T18:54:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:25.594+0000 7fa02659c640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008005200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:3301/0 2023-11-28T18:54:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:25.594+0000 7fa026d9d640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:25.594+0000 7fa02659c640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:27.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa024d99640 1 -- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 msgr2=0x7fa008005200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:27.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa024d99640 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa0080102d0 0x7fa008005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:27.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa024d99640 1 -- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 msgr2=0x7fa00800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:27.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa024d99640 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fa008003ac0 0x7fa00800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T18:54:27.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa024d99640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa008008580 legacy=0x7fa00800c040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T18:54:27.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa024d99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 0x7fa0080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T18:54:27.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa024d99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02815c110 con 0x7fa008004d00 2023-11-28T18:54:27.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa024d99640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070930 con 0x7fa0080043e0 2023-11-28T18:54:27.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 msgr2=0x7fa0080047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:27.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004d00 legacy=0x7fa008007510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:27.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.186+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 0x7fa0080047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T18:54:27.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.390+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 msgr2=0x7fa0080047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:27.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.390+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004d00 legacy=0x7fa008007510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:27.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.390+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 0x7fa0080047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T18:54:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.794+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 msgr2=0x7fa0080047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:27.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.794+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004d00 legacy=0x7fa008007510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:27.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:27.794+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 0x7fa0080047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T18:54:28.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:28.598+0000 7fa02659c640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 msgr2=0x7fa0080047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-28T18:54:28.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:28.598+0000 7fa01ffff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fa008004d00 legacy=0x7fa008007510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-28T18:54:28.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-28T18:54:28.598+0000 7fa02659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0080043e0 0x7fa0080047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T18:54:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:54:30.052 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T18:54:30.071 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-28T18:54:30.071 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T18:54:30.071 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:54:30.075 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2023-11-28T18:54:30.088 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T18:54:30.088 ERROR:tasks.ceph.mds.c:tried to stop a non-running daemon 2023-11-28T18:54:30.091 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-11-28T18:54:30.091 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.092 INFO:tasks.ceph.mds.e:Stopped 2023-11-28T18:54:30.092 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T18:54:30.092 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.093 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T18:54:30.093 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T18:54:30.093 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.094 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T18:54:30.094 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-11-28T18:54:30.094 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.094 INFO:tasks.ceph.mds.f:Stopped 2023-11-28T18:54:30.095 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-28T18:54:30.095 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-28T18:54:30.095 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.095 INFO:tasks.ceph.osd.0:Stopped 2023-11-28T18:54:30.096 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-28T18:54:30.096 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.096 INFO:tasks.ceph.osd.1:Stopped 2023-11-28T18:54:30.097 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-28T18:54:30.097 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.097 INFO:tasks.ceph.osd.2:Stopped 2023-11-28T18:54:30.098 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-28T18:54:30.098 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.098 INFO:tasks.ceph.osd.3:Stopped 2023-11-28T18:54:30.098 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-28T18:54:30.099 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.099 INFO:tasks.ceph.osd.4:Stopped 2023-11-28T18:54:30.099 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-28T18:54:30.099 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.100 INFO:tasks.ceph.osd.5:Stopped 2023-11-28T18:54:30.101 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-28T18:54:30.101 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.102 INFO:tasks.ceph.osd.6:Stopped 2023-11-28T18:54:30.102 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-28T18:54:30.102 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.103 INFO:tasks.ceph.osd.7:Stopped 2023-11-28T18:54:30.103 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-28T18:54:30.103 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-28T18:54:30.103 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.104 INFO:tasks.ceph.mgr.y:Stopped 2023-11-28T18:54:30.104 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-28T18:54:30.104 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:54:30.105 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2023-11-28T18:54:30.118 INFO:tasks.ceph.mgr.x:Stopped 2023-11-28T18:54:30.119 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-28T18:54:30.119 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T18:54:30.119 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.120 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T18:54:30.120 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-28T18:54:30.120 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.123 INFO:tasks.ceph.mon.b:Stopped 2023-11-28T18:54:30.124 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-28T18:54:30.124 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T18:54:30.125 INFO:tasks.ceph.mon.c:Stopped 2023-11-28T18:54:30.125 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-28T18:54:30.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-11-28T18:54:30.205 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:54:30.205 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-28T18:54:30.490 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:54:30.490 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-28T18:54:30.702 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:54:30.702 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-28T18:54:30.924 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:54:30.924 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-28T18:54:31.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:54:31.197 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-28T18:54:31.344 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:54:31.345 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-28T18:54:31.583 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:54:31.584 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-28T18:54:31.914 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:54:31.915 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-28T18:54:32.129 INFO:tasks.ceph:Archiving mon data... 2023-11-28T18:54:32.131 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/data/mon.a.tgz 2023-11-28T18:54:32.131 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-11-28T18:54:32.160 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.wj9L6Btdtz 2023-11-28T18:54:32.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.wj9L6Btdtz 2023-11-28T18:54:32.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.wj9L6Btdtz 2023-11-28T18:54:32.406 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.wj9L6Btdtz is 264KB 2023-11-28T18:54:32.479 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.wj9L6Btdtz 2023-11-28T18:54:32.489 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/data/mon.b.tgz 2023-11-28T18:54:32.489 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2023-11-28T18:54:32.498 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.7kbFfklKfj 2023-11-28T18:54:32.499 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.7kbFfklKfj 2023-11-28T18:54:32.618 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.7kbFfklKfj 2023-11-28T18:54:32.695 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.7kbFfklKfj is 267KB 2023-11-28T18:54:32.764 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.7kbFfklKfj 2023-11-28T18:54:32.773 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/data/mon.c.tgz 2023-11-28T18:54:32.773 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2023-11-28T18:54:32.830 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.6u72ULh88u 2023-11-28T18:54:32.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.6u72ULh88u 2023-11-28T18:54:32.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.6u72ULh88u 2023-11-28T18:54:33.021 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.6u72ULh88u is 267KB 2023-11-28T18:54:33.084 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.6u72ULh88u 2023-11-28T18:54:33.093 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-28T18:54:33.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T18:54:33.097 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T18:54:33.157 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-28T18:54:33.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T18:54:33.167 DEBUG:teuthology.orchestra.run.smithi193:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T18:54:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T18:54:33.230 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T18:54:33.237 INFO:tasks.ceph:Archiving crash dumps... 2023-11-28T18:54:33.239 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/remote/smithi007/crash 2023-11-28T18:54:33.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T18:54:33.323 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/remote/smithi193/crash 2023-11-28T18:54:33.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T18:54:33.347 INFO:tasks.ceph:Shutting down logrotate 2023-11-28T18:54:33.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:54:33.369 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T18:54:33.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T18:54:33.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T18:54:33.485 INFO:tasks.ceph:Compressing logs... 2023-11-28T18:54:33.485 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T18:54:33.512 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T18:54:33.538 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95038.log 2023-11-28T18:54:33.539 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42928.log 2023-11-28T18:54:33.539 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49339.log 2023-11-28T18:54:33.539 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95038.log.gz 2023-11-28T18:54:33.540 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95641.log 2023-11-28T18:54:33.541 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6%/var/log/ceph/ceph-client.admin.49339.log: -- replaced with /var/log/ceph/ceph-client.admin.42928.log.gz 2023-11-28T18:54:33.542 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96504.log 2023-11-28T18:54:33.542 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49339.log.gz 2023-11-28T18:54:33.542 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95641.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95641.log.gz 2023-11-28T18:54:33.543 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96296.log 2023-11-28T18:54:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96504.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.49495.log -- replaced with /var/log/ceph/ceph-client.admin.96504.log.gz 2023-11-28T18:54:33.544 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-28T18:54:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96296.log: gzip -5 --verbose 87.5% -- -- replaced with /var/log/ceph/ceph-client.admin.96296.log.gz 2023-11-28T18:54:33.545 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.94617.log 2023-11-28T18:54:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50904.log 2023-11-28T18:54:33.546 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49495.log.gz 2023-11-28T18:54:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94617.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94617.log.gz 2023-11-28T18:54:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73695.log 2023-11-28T18:54:33.548 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43361.log 2023-11-28T18:54:33.548 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50904.log.gz 2023-11-28T18:54:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.73695.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73695.log.gz 2023-11-28T18:54:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43582.log 2023-11-28T18:54:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94251.log 2023-11-28T18:54:33.551 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43361.log.gz 2023-11-28T18:54:33.552 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84973.log 2023-11-28T18:54:33.552 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43582.log.gz 2023-11-28T18:54:33.552 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94251.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94251.log.gz 2023-11-28T18:54:33.553 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51945.log 2023-11-28T18:54:33.554 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.84973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44198.log 2023-11-28T18:54:33.554 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84973.log.gz 2023-11-28T18:54:33.554 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72582.log 2023-11-28T18:54:33.554 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51945.log.gz 2023-11-28T18:54:33.557 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48674.log 2023-11-28T18:54:33.558 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44198.log.gz 2023-11-28T18:54:33.558 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.72582.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.72582.log.gz 2023-11-28T18:54:33.558 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.96741.log 2023-11-28T18:54:33.559 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96216.log 2023-11-28T18:54:33.559 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48674.log.gz 2023-11-28T18:54:33.577 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38431.log 2023-11-28T18:54:33.577 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38455.log 2023-11-28T18:54:33.577 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.38431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2023-11-28T18:54:33.578 INFO:teuthology.orchestra.run.smithi193.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38431.log.gz 2023-11-28T18:54:33.578 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.38455.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38455.log.gz 2023-11-28T18:54:33.578 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-28T18:54:33.578 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.38503.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2023-11-28T18:54:33.578 INFO:teuthology.orchestra.run.smithi193.stderr: --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-28T18:54:33.578 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38335.log 2023-11-28T18:54:33.579 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38407.log 2023-11-28T18:54:33.579 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.38335.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38335.log.gz 2023-11-28T18:54:33.580 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-28T18:54:33.580 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.38407.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38407.log.gz 2023-11-28T18:54:33.580 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38383.log 2023-11-28T18:54:33.580 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-28T18:54:33.580 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.38383.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38383.log.gz 2023-11-28T18:54:33.580 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-11-28T18:54:33.580 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T18:54:33.581 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38479.log 2023-11-28T18:54:33.581 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-28T18:54:33.581 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.38479.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38479.log.gz 2023-11-28T18:54:33.581 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-mon.c.log: 93.5% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T18:54:33.584 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T18:54:33.585 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-11-28T18:54:33.585 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-mds.d.log: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T18:54:33.589 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-11-28T18:54:33.590 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74101.log 2023-11-28T18:54:33.590 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96741.log.gz 2023-11-28T18:54:33.590 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96216.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96216.log.gz 2023-11-28T18:54:33.591 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-28T18:54:33.591 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74101.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74101.log.gz 2023-11-28T18:54:33.591 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96008.log 2023-11-28T18:54:33.591 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48837.log 2023-11-28T18:54:33.591 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96008.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.96008.log.gz 2023-11-28T18:54:33.591 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50931.log 2023-11-28T18:54:33.591 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94591.log 2023-11-28T18:54:33.591 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48837.log.gz 2023-11-28T18:54:33.592 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50931.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50931.log.gz 2023-11-28T18:54:33.592 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52075.log 2023-11-28T18:54:33.592 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95875.log 2023-11-28T18:54:33.592 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94591.log.gz 2023-11-28T18:54:33.592 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97057.log 2023-11-28T18:54:33.592 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52075.log.gz 2023-11-28T18:54:33.592 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95875.log: gzip -5 88.2% --verbose -- replaced with /var/log/ceph/ceph-client.admin.95875.log.gz -- 2023-11-28T18:54:33.593 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.58140.log 2023-11-28T18:54:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96478.log 2023-11-28T18:54:33.593 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97057.log.gz 2023-11-28T18:54:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.58140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51483.log 2023-11-28T18:54:33.593 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58140.log.gz 2023-11-28T18:54:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96478.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96478.log.gz 2023-11-28T18:54:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49767.log 2023-11-28T18:54:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44865.log 2023-11-28T18:54:33.594 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51483.log.gz 2023-11-28T18:54:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49767.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.49767.log.gz --verbose 2023-11-28T18:54:33.594 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-osd.0.log 2023-11-28T18:54:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95561.log 2023-11-28T18:54:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2023-11-28T18:54:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74129.log 2023-11-28T18:54:33.595 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95561.log.gz 2023-11-28T18:54:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94435.log 2023-11-28T18:54:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48781.log 2023-11-28T18:54:33.595 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74129.log.gz 2023-11-28T18:54:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94435.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94435.log.gz 2023-11-28T18:54:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43156.log 2023-11-28T18:54:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48781.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48781.log.gz 2023-11-28T18:54:33.595 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.94854.log 2023-11-28T18:54:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83378.log 2023-11-28T18:54:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97004.log 2023-11-28T18:54:33.596 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94854.log.gz 2023-11-28T18:54:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83378.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83378.log.gz 2023-11-28T18:54:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94672.log 2023-11-28T18:54:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48204.log 2023-11-28T18:54:33.596 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97004.log.gz 2023-11-28T18:54:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94672.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43156.log.gz 2023-11-28T18:54:33.597 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51213.log 2023-11-28T18:54:33.597 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94672.log.gz 2023-11-28T18:54:33.597 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96530.log 2023-11-28T18:54:33.597 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95246.log 2023-11-28T18:54:33.597 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51213.log.gz 2023-11-28T18:54:33.597 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48204.log.gz 2023-11-28T18:54:33.597 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95771.log 2023-11-28T18:54:33.598 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96530.log.gz 2023-11-28T18:54:33.598 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95246.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95246.log.gz 2023-11-28T18:54:33.598 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83245.log 2023-11-28T18:54:33.598 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95771.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44865.log.gz 2023-11-28T18:54:33.598 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48514.log 2023-11-28T18:54:33.598 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95771.log.gz 2023-11-28T18:54:33.598 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83352.log 2023-11-28T18:54:33.599 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83245.log.gz 2023-11-28T18:54:33.601 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-28T18:54:33.601 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-28T18:54:33.601 INFO:teuthology.orchestra.run.smithi193.stderr: 95.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-28T18:54:33.601 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38359.log 2023-11-28T18:54:33.601 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.38359.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38359.log.gz 2023-11-28T18:54:33.606 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94461.log 2023-11-28T18:54:33.606 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48514.log.gz 2023-11-28T18:54:33.606 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48908.log 2023-11-28T18:54:33.606 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83352.log.gz 2023-11-28T18:54:33.606 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42431.log 2023-11-28T18:54:33.607 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.94461.log.gz 2023-11-28T18:54:33.609 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95667.log 2023-11-28T18:54:33.609 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48908.log.gz 2023-11-28T18:54:33.609 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43066.log 2023-11-28T18:54:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95667.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95667.log.gz 2023-11-28T18:54:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50849.log 2023-11-28T18:54:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43367.log 2023-11-28T18:54:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50849.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.42362.log -- replaced with /var/log/ceph/ceph-client.admin.50849.log.gz 2023-11-28T18:54:33.610 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-28T18:54:33.610 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50242.log 2023-11-28T18:54:33.610 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42431.log.gz 2023-11-28T18:54:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42362.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43066.log.gz 2023-11-28T18:54:33.611 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43367.log.gz 2023-11-28T18:54:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43167.log 2023-11-28T18:54:33.611 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42362.log.gz 2023-11-28T18:54:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96400.log 2023-11-28T18:54:33.611 INFO:teuthology.orchestra.run.smithi007.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50242.log.gz 2023-11-28T18:54:33.611 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94095.log 2023-11-28T18:54:33.612 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49741.log 2023-11-28T18:54:33.612 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96400.log.gz 2023-11-28T18:54:33.612 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50454.log 2023-11-28T18:54:33.612 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94095.log.gz 2023-11-28T18:54:33.612 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50729.log 2023-11-28T18:54:33.612 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49741.log.gz 2023-11-28T18:54:33.612 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50454.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50454.log.gz 2023-11-28T18:54:33.612 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94513.log 2023-11-28T18:54:33.613 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43167.log.gz 2023-11-28T18:54:33.616 INFO:teuthology.orchestra.run.smithi007.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2023-11-28T18:54:33.617 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52154.log 2023-11-28T18:54:33.617 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50729.log.gz 2023-11-28T18:54:33.617 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46610.log 88.6% 2023-11-28T18:54:33.617 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.94513.log.gz 2023-11-28T18:54:33.617 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97403.log 2023-11-28T18:54:33.617 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52154.log.gz 2023-11-28T18:54:33.617 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.46610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48752.log 2023-11-28T18:54:33.618 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48381.log 2023-11-28T18:54:33.618 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48752.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.83162.log -- replaced with /var/log/ceph/ceph-client.admin.48752.log.gz 2023-11-28T18:54:33.618 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-28T18:54:33.618 INFO:teuthology.orchestra.run.smithi007.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.admin.97403.log.gz 2023-11-28T18:54:33.618 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51918.log 2023-11-28T18:54:33.619 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48381.log.gz 2023-11-28T18:54:33.619 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83162.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83162.log.gz 2023-11-28T18:54:33.619 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50402.log 2023-11-28T18:54:33.619 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45073.log 2023-11-28T18:54:33.619 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51918.log.gz 2023-11-28T18:54:33.619 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50402.log: gzip -5 88.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.50402.log.gz -- 2023-11-28T18:54:33.619 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.44146.log 2023-11-28T18:54:33.622 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.45073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49954.log 2023-11-28T18:54:33.622 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46610.log.gz 2023-11-28T18:54:33.622 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45073.log.gz 2023-11-28T18:54:33.622 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97137.log 2023-11-28T18:54:33.622 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44146.log.gz 2023-11-28T18:54:33.622 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49954.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49954.log.gz 2023-11-28T18:54:33.623 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51760.log 2023-11-28T18:54:33.623 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97137.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.97137.log.gz 2023-11-28T18:54:33.623 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44917.log 2023-11-28T18:54:33.623 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51786.log 2023-11-28T18:54:33.623 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51760.log.gz 2023-11-28T18:54:33.623 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96138.log 2023-11-28T18:54:33.625 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44917.log.gz 2023-11-28T18:54:33.625 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43588.log 88.5% 2023-11-28T18:54:33.625 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51786.log.gz 2023-11-28T18:54:33.625 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51067.log 2023-11-28T18:54:33.625 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96138.log.gz 2023-11-28T18:54:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51428.log 2023-11-28T18:54:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51067.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51067.log.gz 2023-11-28T18:54:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-28T18:54:33.627 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51428.log: gzip -5 88.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.51428.log.gz -- 2023-11-28T18:54:33.627 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.55928.log 2023-11-28T18:54:33.628 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44709.log 2023-11-28T18:54:33.628 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.55928.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43588.log.gz 2023-11-28T18:54:33.629 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55928.log.gz 2023-11-28T18:54:33.629 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95509.log 2023-11-28T18:54:33.630 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50294.log 2023-11-28T18:54:33.631 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51840.log 2023-11-28T18:54:33.631 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95509.log.gz 2023-11-28T18:54:33.632 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44995.log 2023-11-28T18:54:33.632 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50294.log.gz 2023-11-28T18:54:33.633 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51840.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51840.log.gz 2023-11-28T18:54:33.633 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44735.log 2023-11-28T18:54:33.634 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49151.log 2023-11-28T18:54:33.636 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50376.log 2023-11-28T18:54:33.636 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49287.log 2023-11-28T18:54:33.637 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49151.log.gz 2023-11-28T18:54:33.637 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49628.log 2023-11-28T18:54:33.637 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50376.log.gz 2023-11-28T18:54:33.638 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96952.log 2023-11-28T18:54:33.638 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49287.log.gz 2023-11-28T18:54:33.639 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49628.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49628.log.gz 2023-11-28T18:54:33.639 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49313.log 2023-11-28T18:54:33.639 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44995.log.gz 2023-11-28T18:54:33.640 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83432.log 2023-11-28T18:54:33.640 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96952.log.gz 2023-11-28T18:54:33.642 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49313.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49313.log.gz 2023-11-28T18:54:33.643 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94121.log 2023-11-28T18:54:33.645 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83432.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.83432.log.gz 2023-11-28T18:54:33.645 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71956.log 2023-11-28T18:54:33.645 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94121.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.94121.log.gz --verbose 2023-11-28T18:54:33.647 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph.tmp-client.admin.33323.log 2023-11-28T18:54:33.647 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.71956.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-28T18:54:33.648 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71956.log.gz 2023-11-28T18:54:33.659 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.33323.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33323.log.gz 2023-11-28T18:54:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43372.log 2023-11-28T18:54:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94303.log 2023-11-28T18:54:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95849.log 2023-11-28T18:54:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95613.log 2023-11-28T18:54:33.661 INFO:teuthology.orchestra.run.smithi007.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.94303.log.gz 2023-11-28T18:54:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95849.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.95849.log.gz 2023-11-28T18:54:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95901.log 2023-11-28T18:54:33.661 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43372.log.gz 2023-11-28T18:54:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51170.log 2023-11-28T18:54:33.662 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95613.log.gz 2023-11-28T18:54:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95901.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95901.log.gz 2023-11-28T18:54:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43159.log 2023-11-28T18:54:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95327.log 2023-11-28T18:54:33.663 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51170.log.gz 2023-11-28T18:54:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82861.log 2023-11-28T18:54:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43171.log 2023-11-28T18:54:33.664 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95327.log.gz 2023-11-28T18:54:33.664 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.82861.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82861.log.gz 2023-11-28T18:54:33.664 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97377.log 2023-11-28T18:54:33.664 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43159.log.gz 2023-11-28T18:54:33.665 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95535.log 2023-11-28T18:54:33.665 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49366.log 2023-11-28T18:54:33.665 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97377.log.gz 2023-11-28T18:54:33.665 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94331.log 2023-11-28T18:54:33.666 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95535.log.gz 2023-11-28T18:54:33.682 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48270.log 2023-11-28T18:54:33.683 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49366.log.gz 2023-11-28T18:54:33.683 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2023-11-28T18:54:33.683 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94331.log.gz 2023-11-28T18:54:33.683 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43579.log 2023-11-28T18:54:33.683 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43154.log 2023-11-28T18:54:33.683 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42683.log 2023-11-28T18:54:33.683 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51041.log 2023-11-28T18:54:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49822.log 2023-11-28T18:54:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42683.log: /var/log/ceph/ceph-client.admin.51041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59155.log 2023-11-28T18:54:33.684 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51041.log.gz 2023-11-28T18:54:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49822.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49822.log.gz 2023-11-28T18:54:33.684 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43171.log.gz 2023-11-28T18:54:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49392.log 2023-11-28T18:54:33.685 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2023-11-28T18:54:33.685 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5%/var/log/ceph/ceph-client.admin.59155.log: -- replaced with /var/log/ceph/ceph-client.admin.43579.log.gz 2023-11-28T18:54:33.685 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45021.log 2023-11-28T18:54:33.685 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59155.log.gz 2023-11-28T18:54:33.686 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43154.log.gz 2023-11-28T18:54:33.686 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49392.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42683.log.gz 2023-11-28T18:54:33.686 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48808.log 2023-11-28T18:54:33.687 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49392.log.gz 2023-11-28T18:54:33.687 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48270.log.gz 2023-11-28T18:54:33.687 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.45021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42577.log 2023-11-28T18:54:33.687 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43164.log 2023-11-28T18:54:33.688 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48808.log.gz 2023-11-28T18:54:33.688 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49601.log 2023-11-28T18:54:33.688 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73933.log 2023-11-28T18:54:33.689 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49601.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45021.log.gz 2023-11-28T18:54:33.689 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49601.log.gz 2023-11-28T18:54:33.689 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49902.log 2023-11-28T18:54:33.689 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.73933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95693.log 2023-11-28T18:54:33.690 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73933.log.gz 2023-11-28T18:54:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49902.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49902.log.gz 2023-11-28T18:54:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94802.log 2023-11-28T18:54:33.691 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42577.log.gz 2023-11-28T18:54:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95693.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95693.log.gz 2023-11-28T18:54:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46422.log 2023-11-28T18:54:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97030.log 2023-11-28T18:54:33.691 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43164.log.gz 2023-11-28T18:54:33.691 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94802.log.gz 2023-11-28T18:54:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.46422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43938.log 2023-11-28T18:54:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96897.log 2023-11-28T18:54:33.692 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97030.log.gz 2023-11-28T18:54:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51866.log 2023-11-28T18:54:33.692 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43938.log.gz 2023-11-28T18:54:33.706 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50136.log 2023-11-28T18:54:33.706 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96897.log.gz 2023-11-28T18:54:33.707 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51866.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51866.log.gz 2023-11-28T18:54:33.707 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48170.log 2023-11-28T18:54:33.707 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51668.log 2023-11-28T18:54:33.707 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50136.log.gz 2023-11-28T18:54:33.707 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94539.log 2023-11-28T18:54:33.707 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96086.log 2023-11-28T18:54:33.707 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51668.log.gz 2023-11-28T18:54:33.708 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94539.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94539.log.gz 2023-11-28T18:54:33.708 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50875.log 2023-11-28T18:54:33.708 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95457.log 2023-11-28T18:54:33.708 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96086.log.gz 2023-11-28T18:54:33.708 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48170.log.gz 2023-11-28T18:54:33.708 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51812.log 2023-11-28T18:54:33.708 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50875.log.gz 2023-11-28T18:54:33.708 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95457.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95457.log.gz 2023-11-28T18:54:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44016.log 2023-11-28T18:54:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48934.log 2023-11-28T18:54:33.709 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51812.log.gz 2023-11-28T18:54:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94828.log 2023-11-28T18:54:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42551.log 2023-11-28T18:54:33.709 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48934.log.gz 2023-11-28T18:54:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51376.log 2023-11-28T18:54:33.709 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.94828.log.gz 2023-11-28T18:54:33.710 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0%/var/log/ceph/ceph-client.admin.42551.log: -- replaced with /var/log/ceph/ceph-client.admin.44016.log.gz 2023-11-28T18:54:33.710 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50597.log 2023-11-28T18:54:33.710 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97163.log 2023-11-28T18:54:33.710 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51376.log.gz 2023-11-28T18:54:33.710 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94986.log 2023-11-28T18:54:33.711 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50597.log.gz 2023-11-28T18:54:33.711 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42551.log.gz 2023-11-28T18:54:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44969.log 2023-11-28T18:54:33.711 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.97163.log.gz 2023-11-28T18:54:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95194.log 2023-11-28T18:54:33.711 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94986.log.gz 2023-11-28T18:54:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93991.log 2023-11-28T18:54:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47614.log 2023-11-28T18:54:33.712 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95194.log.gz 2023-11-28T18:54:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93991.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44969.log.gz 2023-11-28T18:54:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50649.log 2023-11-28T18:54:33.712 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.93991.log.gz 2023-11-28T18:54:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.47614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74262.log 2023-11-28T18:54:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50649.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-28T18:54:33.712 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50649.log.gz 2023-11-28T18:54:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47408.log 2023-11-28T18:54:33.716 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74262.log.gz 2023-11-28T18:54:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97268.log 2023-11-28T18:54:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.47408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51708.log 2023-11-28T18:54:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43560.log 2023-11-28T18:54:33.717 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97268.log.gz 2023-11-28T18:54:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59483.log 2023-11-28T18:54:33.717 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51708.log.gz 2023-11-28T18:54:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43118.log 2023-11-28T18:54:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.59483.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44709.log.gz 2023-11-28T18:54:33.717 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59483.log.gz 88.0% 2023-11-28T18:54:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -- replaced with /var/log/ceph/ceph-client.admin.44735.log.gz 2023-11-28T18:54:33.717 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52218.log 2023-11-28T18:54:33.718 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43560.log.gz 2023-11-28T18:54:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48461.log 2023-11-28T18:54:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48567.log 2023-11-28T18:54:33.718 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52218.log.gz 2023-11-28T18:54:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48461.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48461.log.gz 2023-11-28T18:54:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97346.log 2023-11-28T18:54:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50959.log 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48567.log.gz 2023-11-28T18:54:33.720 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-28T18:54:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42605.log 2023-11-28T18:54:33.720 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.97346.log.gz 2023-11-28T18:54:33.720 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43118.log.gz 2023-11-28T18:54:33.721 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50959.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50959.log.gz 2023-11-28T18:54:33.721 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2023-11-28T18:54:33.721 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44277.log 2023-11-28T18:54:33.722 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53336.log 2023-11-28T18:54:33.722 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2023-11-28T18:54:33.722 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46422.log.gz 2023-11-28T18:54:33.722 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44277.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47408.log.gz 2023-11-28T18:54:33.723 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47614.log.gz 2023-11-28T18:54:33.723 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51615.log 2023-11-28T18:54:33.723 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42605.log.gz 2023-11-28T18:54:33.724 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44277.log.gz 2023-11-28T18:54:33.724 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.53336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43170.log 2023-11-28T18:54:33.724 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53336.log.gz 2023-11-28T18:54:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51615.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51615.log.gz 2023-11-28T18:54:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44813.log 2023-11-28T18:54:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95379.log 2023-11-28T18:54:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48320.log 2023-11-28T18:54:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95379.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43170.log.gz 2023-11-28T18:54:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97292.log 2023-11-28T18:54:33.728 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95379.log.gz 2023-11-28T18:54:33.728 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94487.log 2023-11-28T18:54:33.729 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48320.log.gz 2023-11-28T18:54:33.729 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43771.log 2023-11-28T18:54:33.729 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97292.log.gz 2023-11-28T18:54:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42765.log 2023-11-28T18:54:33.730 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94487.log.gz 2023-11-28T18:54:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96793.log 2023-11-28T18:54:33.731 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43771.log.gz 2023-11-28T18:54:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93965.log 2023-11-28T18:54:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95927.log 2023-11-28T18:54:33.733 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96793.log.gz 2023-11-28T18:54:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42824.log 2023-11-28T18:54:33.735 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93965.log.gz 2023-11-28T18:54:33.735 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42765.log.gz 2023-11-28T18:54:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95927.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.95927.log.gz 2023-11-28T18:54:33.735 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50814.log 2023-11-28T18:54:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42824.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-28T18:54:33.737 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50814.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42824.log.gz 2023-11-28T18:54:33.737 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.50814.log.gz -5 2023-11-28T18:54:33.737 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50162.log 2023-11-28T18:54:33.738 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44813.log.gz 2023-11-28T18:54:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96426.log 2023-11-28T18:54:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83217.log 2023-11-28T18:54:33.739 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50162.log.gz 2023-11-28T18:54:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51015.log 2023-11-28T18:54:33.740 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96426.log.gz 2023-11-28T18:54:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83217.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83217.log.gz 2023-11-28T18:54:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44224.log 2023-11-28T18:54:33.742 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81403.log 2023-11-28T18:54:33.742 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51015.log.gz 2023-11-28T18:54:33.743 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49012.log 2023-11-28T18:54:33.744 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.81403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56522.log 2023-11-28T18:54:33.744 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81403.log.gz 2023-11-28T18:54:33.745 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48726.log 2023-11-28T18:54:33.745 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44224.log.gz 2023-11-28T18:54:33.746 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49012.log.gz 2023-11-28T18:54:33.746 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.56522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96034.log 2023-11-28T18:54:33.746 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56522.log.gz 2023-11-28T18:54:33.747 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48726.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48726.log.gz 2023-11-28T18:54:33.747 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96871.log 2023-11-28T18:54:33.748 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97090.log 2023-11-28T18:54:33.748 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96034.log.gz 2023-11-28T18:54:33.749 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49715.log 2023-11-28T18:54:33.749 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.96871.log.gz 2023-11-28T18:54:33.750 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44094.log 2023-11-28T18:54:33.751 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97090.log.gz 2023-11-28T18:54:33.751 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49715.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49715.log.gz 2023-11-28T18:54:33.751 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44943.log 2023-11-28T18:54:33.752 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95956.log 2023-11-28T18:54:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74182.log 2023-11-28T18:54:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49125.log 2023-11-28T18:54:33.754 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95956.log.gz 2023-11-28T18:54:33.754 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44094.log.gz 2023-11-28T18:54:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74182.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44943.log.gz 2023-11-28T18:54:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- 89.4% /var/log/ceph/ceph-client.admin.49095.log -- replaced with /var/log/ceph/ceph-client.admin.74182.log.gz 2023-11-28T18:54:33.755 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-28T18:54:33.756 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43172.log 2023-11-28T18:54:33.756 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49125.log.gz 2023-11-28T18:54:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94277.log 2023-11-28T18:54:33.758 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49095.log.gz 2023-11-28T18:54:33.758 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50762.log 2023-11-28T18:54:33.759 INFO:teuthology.orchestra.run.smithi007.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43172.log.gz 2023-11-28T18:54:33.760 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51971.log 2023-11-28T18:54:33.760 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.94277.log.gz 2023-11-28T18:54:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50762.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50762.log.gz 2023-11-28T18:54:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52128.log 2023-11-28T18:54:33.762 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51971.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51971.log.gz 2023-11-28T18:54:33.762 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50428.log 2023-11-28T18:54:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51734.log 2023-11-28T18:54:33.763 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52128.log.gz 2023-11-28T18:54:33.764 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43576.log 2023-11-28T18:54:33.764 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50428.log.gz 2023-11-28T18:54:33.765 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43362.log 2023-11-28T18:54:33.765 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51734.log.gz 2023-11-28T18:54:33.766 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83484.log 2023-11-28T18:54:33.766 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43964.log 2023-11-28T18:54:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83484.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43576.log.gz 2023-11-28T18:54:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45151.log 2023-11-28T18:54:33.768 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83484.log.gz 2023-11-28T18:54:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48648.log 2023-11-28T18:54:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.45151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49521.log 2023-11-28T18:54:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96663.log 2023-11-28T18:54:33.771 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48648.log.gz 2023-11-28T18:54:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94147.log 2023-11-28T18:54:33.771 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49521.log.gz 2023-11-28T18:54:33.773 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43362.log.gz 2023-11-28T18:54:33.773 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51317.log 2023-11-28T18:54:33.773 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96663.log.gz 2023-11-28T18:54:33.774 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43014.log 2023-11-28T18:54:33.774 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.94147.log.gz 2023-11-28T18:54:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48435.log 2023-11-28T18:54:33.776 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43964.log.gz 2023-11-28T18:54:33.776 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51317.log.gz 2023-11-28T18:54:33.778 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83406.log 2023-11-28T18:54:33.778 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2023-11-28T18:54:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50675.log 2023-11-28T18:54:33.779 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48435.log.gz 2023-11-28T18:54:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83406.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83406.log.gz 2023-11-28T18:54:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51239.log 2023-11-28T18:54:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50987.log 2023-11-28T18:54:33.780 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50675.log.gz 2023-11-28T18:54:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51239.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51239.log.gz 2023-11-28T18:54:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50348.log 2023-11-28T18:54:33.781 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43014.log.gz 2023-11-28T18:54:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50987.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50987.log.gz 2023-11-28T18:54:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94958.log 2023-11-28T18:54:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50348.log: gzip -5 --verbose 89.3% -- -- replaced with /var/log/ceph/ceph-client.admin.50348.log.gz /var/log/ceph/ceph-client.admin.47733.log 2023-11-28T18:54:33.783 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-28T18:54:33.783 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94958.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94958.log.gz -5 2023-11-28T18:54:33.783 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.96268.log 2023-11-28T18:54:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.47733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96637.log 2023-11-28T18:54:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97189.log 2023-11-28T18:54:33.786 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47733.log.gz 2023-11-28T18:54:33.786 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96268.log.gz 2023-11-28T18:54:33.787 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96637.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.96637.log.gz -5 2023-11-28T18:54:33.787 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.74156.log 2023-11-28T18:54:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97189.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97189.log.gz 2023-11-28T18:54:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50084.log 2023-11-28T18:54:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74156.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74156.log.gz 2023-11-28T18:54:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83297.log 2023-11-28T18:54:33.790 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44533.log 2023-11-28T18:54:33.790 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50084.log.gz 2023-11-28T18:54:33.791 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83297.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83297.log.gz 2023-11-28T18:54:33.791 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44428.log 2023-11-28T18:54:33.792 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52023.log 2023-11-28T18:54:33.793 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95587.log 2023-11-28T18:54:33.794 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52023.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T18:54:33.795 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52023.log.gz 2023-11-28T18:54:33.796 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49574.log 2023-11-28T18:54:33.796 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95587.log.gz 2023-11-28T18:54:33.797 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94409.log 2023-11-28T18:54:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49421.log 2023-11-28T18:54:33.798 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49574.log.gz 2023-11-28T18:54:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95797.log 2023-11-28T18:54:33.799 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T18:54:33.799 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.94409.log.gz 2023-11-28T18:54:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49421.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49421.log.gz 2023-11-28T18:54:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74075.log 2023-11-28T18:54:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96715.log 2023-11-28T18:54:33.801 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95797.log.gz 2023-11-28T18:54:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95405.log 2023-11-28T18:54:33.803 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74075.log.gz 2023-11-28T18:54:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96715.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96715.log.gz 2023-11-28T18:54:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95116.log 2023-11-28T18:54:33.804 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42631.log 2023-11-28T18:54:33.804 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95405.log.gz 2023-11-28T18:54:33.805 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96819.log 2023-11-28T18:54:33.805 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95116.log.gz 2023-11-28T18:54:33.806 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49796.log 2023-11-28T18:54:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48864.log 2023-11-28T18:54:33.807 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96819.log.gz 2023-11-28T18:54:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94776.log 2023-11-28T18:54:33.808 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49796.log.gz 87.8% 2023-11-28T18:54:33.808 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42631.log.gz 2023-11-28T18:54:33.809 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48700.log 2023-11-28T18:54:33.809 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48864.log.gz 2023-11-28T18:54:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97083.log 2023-11-28T18:54:33.812 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94776.log.gz 2023-11-28T18:54:33.813 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48700.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48700.log.gz 2023-11-28T18:54:33.813 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50188.log 2023-11-28T18:54:33.813 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43379.log 2023-11-28T18:54:33.814 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97083.log.gz 2023-11-28T18:54:33.814 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50188.log: gzip 89.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.50188.log.gz -- 2023-11-28T18:54:33.814 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.52101.log 2023-11-28T18:54:33.814 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44891.log 2023-11-28T18:54:33.816 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55013.log 2023-11-28T18:54:33.818 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52101.log.gz 2023-11-28T18:54:33.820 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44891.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43379.log.gz 2023-11-28T18:54:33.820 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50110.log 2023-11-28T18:54:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.55013.log: gzip -5 87.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.55013.log.gz -- 2023-11-28T18:54:33.821 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.43849.log 2023-11-28T18:54:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50110.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50110.log.gz 2023-11-28T18:54:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43040.log 2023-11-28T18:54:33.822 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43849.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44891.log.gz 2023-11-28T18:54:33.822 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96060.log 2023-11-28T18:54:33.823 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50268.log 2023-11-28T18:54:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96060.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96060.log.gz 2023-11-28T18:54:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94357.log 2023-11-28T18:54:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50268.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50268.log.gz 2023-11-28T18:54:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49984.log 2023-11-28T18:54:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94357.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43849.log.gz 2023-11-28T18:54:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.57064.log 2023-11-28T18:54:33.826 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.94357.log.gz 2023-11-28T18:54:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49984.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49984.log.gz 2023-11-28T18:54:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83323.log 2023-11-28T18:54:33.827 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43040.log.gz 2023-11-28T18:54:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.57064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51537.log 2023-11-28T18:54:33.830 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57064.log.gz 2023-11-28T18:54:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48986.log 2023-11-28T18:54:33.831 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83323.log.gz 2023-11-28T18:54:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97216.log 2023-11-28T18:54:33.831 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51537.log.gz 2023-11-28T18:54:33.834 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48986.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48986.log.gz 2023-11-28T18:54:33.834 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49178.log 2023-11-28T18:54:33.834 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52049.log 2023-11-28T18:54:33.834 INFO:teuthology.orchestra.run.smithi007.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97216.log.gz 2023-11-28T18:54:33.835 INFO:teuthology.orchestra.run.smithi007.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44533.log.gz 2023-11-28T18:54:33.835 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44428.log.gz 2023-11-28T18:54:33.835 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95142.log 2023-11-28T18:54:33.835 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49178.log.gz 2023-11-28T18:54:33.836 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52049.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52049.log.gz 2023-11-28T18:54:33.836 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43380.log 2023-11-28T18:54:33.837 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50480.log 2023-11-28T18:54:33.838 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95142.log.gz 2023-11-28T18:54:33.838 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42467.log 2023-11-28T18:54:33.838 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50480.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50480.log.gz 2023-11-28T18:54:33.838 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50623.log 2023-11-28T18:54:33.839 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43380.log.gz 2023-11-28T18:54:33.840 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72144.log 2023-11-28T18:54:33.840 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94724.log 2023-11-28T18:54:33.840 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50623.log.gz 2023-11-28T18:54:33.841 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.72144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74288.log 2023-11-28T18:54:33.841 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72144.log.gz 2023-11-28T18:54:33.842 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94724.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.94724.log.gz 2023-11-28T18:54:33.842 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94173.log 2023-11-28T18:54:33.842 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74288.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42467.log.gz 2023-11-28T18:54:33.843 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74288.log.gz 2023-11-28T18:54:33.844 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49232.log 2023-11-28T18:54:33.844 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48090.log 2023-11-28T18:54:33.844 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94173.log.gz 2023-11-28T18:54:33.845 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44042.log 2023-11-28T18:54:33.845 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49232.log.gz 2023-11-28T18:54:33.846 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58631.log 2023-11-28T18:54:33.846 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49547.log 2023-11-28T18:54:33.848 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.58631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51124.log 2023-11-28T18:54:33.848 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58631.log.gz 2023-11-28T18:54:33.849 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48238.log 2023-11-28T18:54:33.849 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49547.log.gz 2023-11-28T18:54:33.849 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51124.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48090.log.gz 2023-11-28T18:54:33.850 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44042.log.gz 2023-11-28T18:54:33.850 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51124.log.gz 2023-11-28T18:54:33.851 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49467.log 2023-11-28T18:54:33.851 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96582.log 2023-11-28T18:54:33.852 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48238.log: /var/log/ceph/ceph-client.admin.49467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44251.log 2023-11-28T18:54:33.852 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49467.log.gz 2023-11-28T18:54:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96582.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.48238.log.gz 2023-11-28T18:54:33.854 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44378.log 2023-11-28T18:54:33.854 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96582.log.gz 2023-11-28T18:54:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51641.log 2023-11-28T18:54:33.856 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42988.log 2023-11-28T18:54:33.857 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51641.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51641.log.gz 2023-11-28T18:54:33.857 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48136.log 2023-11-28T18:54:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52412.log 2023-11-28T18:54:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50555.log 2023-11-28T18:54:33.859 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44251.log.gz 2023-11-28T18:54:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52853.log 2023-11-28T18:54:33.860 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52412.log.gz 2023-11-28T18:54:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50555.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.44378.log.gz 2023-11-28T18:54:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44328.log 2023-11-28T18:54:33.861 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50555.log.gz 2023-11-28T18:54:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52853.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52853.log.gz 2023-11-28T18:54:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42962.log 2023-11-28T18:54:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95220.log 2023-11-28T18:54:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51511.log 2023-11-28T18:54:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95220.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T18:54:33.865 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95220.log.gz 2023-11-28T18:54:33.865 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42988.log.gz 2023-11-28T18:54:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48407.log 2023-11-28T18:54:33.866 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51511.log.gz 2023-11-28T18:54:33.867 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-11-28T18:54:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51997.log 2023-11-28T18:54:33.868 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48407.log.gz 2023-11-28T18:54:33.869 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95301.log 2023-11-28T18:54:33.870 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50216.log 2023-11-28T18:54:33.870 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51997.log.gz 2023-11-28T18:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94750.log 2023-11-28T18:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95301.log.gz 2023-11-28T18:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52191.log 2023-11-28T18:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50216.log.gz 2023-11-28T18:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47508.log 2023-11-28T18:54:33.878 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94750.log.gz 2023-11-28T18:54:33.878 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.52191.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52191.log.gz 2023-11-28T18:54:33.878 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-11-28T18:54:33.878 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.47508.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48136.log.gz 2023-11-28T18:54:33.878 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50788.log 2023-11-28T18:54:33.878 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42962.log.gz 2023-11-28T18:54:33.878 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.44328.log.gz 2023-11-28T18:54:33.879 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43587.log 2023-11-28T18:54:33.879 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47508.log.gz 2023-11-28T18:54:33.881 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50788.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50788.log.gz 2023-11-28T18:54:33.881 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49261.log 2023-11-28T18:54:33.881 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96164.log 2023-11-28T18:54:33.881 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95745.log 2023-11-28T18:54:33.881 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49261.log.gz 2023-11-28T18:54:33.882 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94932.log 2023-11-28T18:54:33.882 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.96164.log.gz 2023-11-28T18:54:33.882 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95745.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95745.log.gz 2023-11-28T18:54:33.883 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74236.log 2023-11-28T18:54:33.883 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94932.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94932.log.gz 2023-11-28T18:54:33.884 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49205.log 2023-11-28T18:54:33.884 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43587.log.gz 2023-11-28T18:54:33.884 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96689.log 2023-11-28T18:54:33.885 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74236.log.gz 2023-11-28T18:54:33.885 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49205.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49205.log.gz 2023-11-28T18:54:33.886 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46957.log 2023-11-28T18:54:33.887 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96609.log 2023-11-28T18:54:33.887 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96689.log.gz 2023-11-28T18:54:33.888 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.46957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94199.log 2023-11-28T18:54:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96556.log 2023-11-28T18:54:33.890 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96609.log.gz 2023-11-28T18:54:33.890 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94199.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.94199.log.gzgzip 2023-11-28T18:54:33.891 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50702.log 2023-11-28T18:54:33.891 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96556.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96556.log.gz 2023-11-28T18:54:33.892 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42902.log 2023-11-28T18:54:33.893 INFO:teuthology.orchestra.run.smithi193.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-11-28T18:54:33.894 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50702.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50702.log.gz 2023-11-28T18:54:33.894 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46957.log.gz 2023-11-28T18:54:33.894 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96845.log 2023-11-28T18:54:33.894 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96923.log 2023-11-28T18:54:33.895 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96845.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96845.log.gz 2023-11-28T18:54:33.896 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97443.log 2023-11-28T18:54:33.897 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96923.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42902.log.gz 2023-11-28T18:54:33.897 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51402.log 2023-11-28T18:54:33.897 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96923.log.gz 2023-11-28T18:54:33.898 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95272.log 2023-11-28T18:54:33.899 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51402.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51402.log.gz 2023-11-28T18:54:33.899 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49848.log 2023-11-28T18:54:33.900 INFO:teuthology.orchestra.run.smithi007.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.admin.97443.log.gz 2023-11-28T18:54:33.900 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95719.log 2023-11-28T18:54:33.901 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.95272.log.gz 2023-11-28T18:54:33.901 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49848.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49848.log.gz 2023-11-28T18:54:33.902 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42657.log 2023-11-28T18:54:33.902 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96452.log 2023-11-28T18:54:33.903 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42657.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95719.log.gz 2023-11-28T18:54:33.904 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43378.log 2023-11-28T18:54:33.905 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74314.log 2023-11-28T18:54:33.905 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96452.log.gz 2023-11-28T18:54:33.906 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45047.log 2023-11-28T18:54:33.907 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74314.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42657.log.gz 2023-11-28T18:54:33.907 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45125.log 2023-11-28T18:54:33.907 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74314.log.gz 2023-11-28T18:54:33.908 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.45047.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43378.log.gz 2023-11-28T18:54:33.908 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51265.log 2023-11-28T18:54:33.909 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.45125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55477.log 2023-11-28T18:54:33.910 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43571.log 2023-11-28T18:54:33.910 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51265.log.gz 2023-11-28T18:54:33.911 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45047.log.gz 2023-11-28T18:54:33.911 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.55477.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.55477.log.gz --verbose 2023-11-28T18:54:33.911 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-client.admin.97318.log 2023-11-28T18:54:33.912 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83458.log 2023-11-28T18:54:33.913 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97318.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45125.log.gz 2023-11-28T18:54:33.913 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97318.log.gz 2023-11-28T18:54:33.914 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50057.log 2023-11-28T18:54:33.914 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43571.log.gz 2023-11-28T18:54:33.915 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74340.log 2023-11-28T18:54:33.915 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83458.log.gz 2023-11-28T18:54:33.916 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83271.log 2023-11-28T18:54:33.916 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50057.log.gz 2023-11-28T18:54:33.917 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96190.log 2023-11-28T18:54:33.918 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74340.log.gz 2023-11-28T18:54:33.918 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48487.log 2023-11-28T18:54:33.919 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83271.log.gz 2023-11-28T18:54:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96190.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96190.log.gz 2023-11-28T18:54:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43374.log 2023-11-28T18:54:33.921 INFO:teuthology.orchestra.run.smithi193.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-11-28T18:54:33.921 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84462.log 2023-11-28T18:54:33.921 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48487.log.gz 2023-11-28T18:54:33.922 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50031.log 2023-11-28T18:54:33.923 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.84462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94383.log 2023-11-28T18:54:33.923 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84462.log.gz 2023-11-28T18:54:33.923 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50031.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43374.log.gz 2023-11-28T18:54:33.924 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50031.log.gz 2023-11-28T18:54:33.924 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95431.log 2023-11-28T18:54:33.925 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42737.log 2023-11-28T18:54:33.925 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94383.log.gz 2023-11-28T18:54:33.926 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95168.log 2023-11-28T18:54:33.926 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95431.log.gz 2023-11-28T18:54:33.927 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48960.log 2023-11-28T18:54:33.928 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96348.log 2023-11-28T18:54:33.928 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95168.log.gz 2023-11-28T18:54:33.929 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95090.log 2023-11-28T18:54:33.929 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48960.log.gz 2023-11-28T18:54:33.930 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82447.log 2023-11-28T18:54:33.930 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96348.log.gz 2023-11-28T18:54:33.931 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95090.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95090.log.gz 2023-11-28T18:54:33.931 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50509.log 2023-11-28T18:54:33.932 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.82447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49680.log 2023-11-28T18:54:33.932 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82447.log.gz 2023-11-28T18:54:33.933 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50509.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50509.log.gz 2023-11-28T18:54:33.933 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94698.log 2023-11-28T18:54:33.933 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42737.log.gz 2023-11-28T18:54:33.934 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T18:54:33.934 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48620.log 2023-11-28T18:54:33.935 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49680.log.gz 2023-11-28T18:54:33.935 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49928.log 2023-11-28T18:54:33.935 INFO:teuthology.orchestra.run.smithi007.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94698.log.gz 2023-11-28T18:54:33.936 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48620.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48620.log.gz 2023-11-28T18:54:33.936 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53823.log 2023-11-28T18:54:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96112.log 2023-11-28T18:54:33.938 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49928.log.gz 2023-11-28T18:54:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.53823.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53823.log.gz 2023-11-28T18:54:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42850.log 2023-11-28T18:54:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94880.log 2023-11-28T18:54:33.945 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96112.log.gz 2023-11-28T18:54:33.946 INFO:teuthology.orchestra.run.smithi007.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-11-28T18:54:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82005.log 2023-11-28T18:54:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94880.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94880.log.gz 2023-11-28T18:54:33.946 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42850.log.gz 2023-11-28T18:54:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47327.log 2023-11-28T18:54:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.82005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51291.log 2023-11-28T18:54:33.947 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82005.log.gz 2023-11-28T18:54:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.47327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51343.log 2023-11-28T18:54:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94017.log 2023-11-28T18:54:33.948 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51291.log.gz 2023-11-28T18:54:33.952 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96767.log 2023-11-28T18:54:33.953 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51343.log.gz 2023-11-28T18:54:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44839.log 2023-11-28T18:54:33.953 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.94017.log.gz 2023-11-28T18:54:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94565.log 2023-11-28T18:54:33.953 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96767.log.gz 2023-11-28T18:54:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49876.log 2023-11-28T18:54:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96242.log 2023-11-28T18:54:33.954 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94565.log.gz 2023-11-28T18:54:33.954 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44839.log.gz 2023-11-28T18:54:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44478.log 2023-11-28T18:54:33.954 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49876.log.gz 2023-11-28T18:54:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96242.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96242.log.gz 2023-11-28T18:54:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96978.log 2023-11-28T18:54:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95353.log 2023-11-28T18:54:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94069.log 2023-11-28T18:54:33.957 INFO:teuthology.orchestra.run.smithi007.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.96978.log.gz 2023-11-28T18:54:33.958 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95353.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95353.log.gz 2023-11-28T18:54:33.958 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51093.log 2023-11-28T18:54:33.959 INFO:teuthology.orchestra.run.smithi007.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44478.log.gz 2023-11-28T18:54:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94069.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.94069.log.gz 2023-11-28T18:54:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96374.log 2023-11-28T18:54:33.966 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47193.log 2023-11-28T18:54:33.967 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51093.log.gz 2023-11-28T18:54:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96374.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96374.log.gz 2023-11-28T18:54:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2023-11-28T18:54:33.968 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.47193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94225.log 2023-11-28T18:54:33.968 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44120.log 2023-11-28T18:54:33.969 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42798.log 2023-11-28T18:54:33.969 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94225.log.gz 2023-11-28T18:54:33.969 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51589.log 2023-11-28T18:54:33.969 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38293.log 2023-11-28T18:54:33.970 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44120.log.gz 2023-11-28T18:54:33.970 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51589.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51589.log.gz 2023-11-28T18:54:33.970 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94043.log 2023-11-28T18:54:33.970 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51892.log 2023-11-28T18:54:33.973 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95823.log 2023-11-28T18:54:33.974 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.94043.log.gz 2023-11-28T18:54:33.974 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43092.log 2023-11-28T18:54:33.974 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51892.log.gz 2023-11-28T18:54:33.975 INFO:teuthology.orchestra.run.smithi007.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42798.log.gz 2023-11-28T18:54:33.975 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95823.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95823.log.gz 2023-11-28T18:54:33.975 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49654.log 2023-11-28T18:54:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74210.log 2023-11-28T18:54:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48355.log 2023-11-28T18:54:33.976 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49654.log.gz 2023-11-28T18:54:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74210.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74210.log.gz 2023-11-28T18:54:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81541.log 2023-11-28T18:54:33.977 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43092.log.gz 2023-11-28T18:54:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48355.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48355.log.gz 2023-11-28T18:54:33.978 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46351.log 2023-11-28T18:54:33.981 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.81541.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81541.log.gz 2023-11-28T18:54:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95064.log 2023-11-28T18:54:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.46351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47655.log 2023-11-28T18:54:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95064.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46351.log.gz 2023-11-28T18:54:33.983 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95064.log.gz 2023-11-28T18:54:33.983 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83510.log 2023-11-28T18:54:33.983 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.47655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44068.log 2023-11-28T18:54:33.984 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.83510.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47327.log.gz 2023-11-28T18:54:33.984 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43990.log 2023-11-28T18:54:33.985 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83510.log.gz 2023-11-28T18:54:33.985 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47655.log.gz 2023-11-28T18:54:33.985 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74707.log 2023-11-28T18:54:33.985 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45099.log 2023-11-28T18:54:33.986 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38293.log.gz 2023-11-28T18:54:33.986 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.74707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96322.log 2023-11-28T18:54:33.991 INFO:teuthology.orchestra.run.smithi007.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74707.log.gz 2023-11-28T18:54:33.991 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.45099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51457.log 2023-11-28T18:54:33.992 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2023-11-28T18:54:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.96322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60035.log 2023-11-28T18:54:33.992 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44068.log.gz 2023-11-28T18:54:33.992 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96322.log.gz 2023-11-28T18:54:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51457.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51457.log.gz 2023-11-28T18:54:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44657.log 2023-11-28T18:54:33.993 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43990.log.gz 2023-11-28T18:54:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.60035.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45099.log.gz 2023-11-28T18:54:33.994 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49066.log 2023-11-28T18:54:33.994 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60035.log.gz 2023-11-28T18:54:33.994 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.0.44657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51563.log 2023-11-28T18:54:33.994 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47193.log.gz 2023-11-28T18:54:33.995 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54871.log 2023-11-28T18:54:33.995 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49066.log.gz 2023-11-28T18:54:33.996 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.51563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44172.log 2023-11-28T18:54:33.996 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51563.log.gz 2023-11-28T18:54:33.996 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.54871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95012.log 2023-11-28T18:54:33.997 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54871.log.gz 2023-11-28T18:54:33.997 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95483.log 2023-11-28T18:54:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95012.log: gzip -5 87.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.95012.log.gz -- 2023-11-28T18:54:34.009 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.43911.log 2023-11-28T18:54:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95483.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44172.log.gz 2023-11-28T18:54:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44787.log 2023-11-28T18:54:34.010 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95483.log.gz 2023-11-28T18:54:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43911.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-11-28T18:54:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97242.log 2023-11-28T18:54:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43797.log 2023-11-28T18:54:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.97242.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43911.log.gz 2023-11-28T18:54:34.010 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97242.log.gz 2023-11-28T18:54:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48593.log 2023-11-28T18:54:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43823.log 2023-11-28T18:54:34.011 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44787.log.gz 2023-11-28T18:54:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.48593.log: gzip -5 88.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.48593.log.gz -- 2023-11-28T18:54:34.011 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.95982.log 2023-11-28T18:54:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50320.log 2023-11-28T18:54:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.95982.log: gzip -5 --verbose 87.7% -- /var/log/ceph/ceph-client.admin.57594.log -- replaced with /var/log/ceph/ceph-client.admin.95982.log.gz 2023-11-28T18:54:34.011 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-28T18:54:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.50320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43883.log 2023-11-28T18:54:34.012 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50320.log.gz 2023-11-28T18:54:34.012 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43797.log.gz 2023-11-28T18:54:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.57594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44761.log 2023-11-28T18:54:34.012 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57594.log.gz 2023-11-28T18:54:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.43883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42876.log 2023-11-28T18:54:34.012 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43823.log.gz 2023-11-28T18:54:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.44761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73221.log 2023-11-28T18:54:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54339.log 2023-11-28T18:54:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.73221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94646.log 2023-11-28T18:54:34.013 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73221.log.gz 2023-11-28T18:54:34.013 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43883.log.gz 2023-11-28T18:54:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.54339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94906.log 2023-11-28T18:54:34.013 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54339.log.gz 2023-11-28T18:54:34.013 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44761.log.gz 2023-11-28T18:54:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42709.log 2023-11-28T18:54:34.014 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.94646.log.gz 2023-11-28T18:54:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49039.log 2023-11-28T18:54:34.016 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94906.log.gz 2023-11-28T18:54:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42709.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42876.log.gz 2023-11-28T18:54:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.49039.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49039.log.gz 2023-11-28T18:54:34.016 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42709.log.gz 2023-11-28T18:54:34.135 INFO:teuthology.orchestra.run.smithi193.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-28T18:54:34.343 INFO:teuthology.orchestra.run.smithi193.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-28T18:54:34.527 INFO:teuthology.orchestra.run.smithi007.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-11-28T18:54:34.839 INFO:teuthology.orchestra.run.smithi193.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-11-28T18:54:36.811 INFO:teuthology.orchestra.run.smithi007.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-11-28T18:54:37.026 INFO:teuthology.orchestra.run.smithi007.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-28T18:54:42.911 INFO:teuthology.orchestra.run.smithi007.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-28T18:54:43.350 INFO:teuthology.orchestra.run.smithi193.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-28T18:54:43.417 INFO:teuthology.orchestra.run.smithi193.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-28T18:54:46.189 INFO:teuthology.orchestra.run.smithi007.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-11-28T18:54:46.192 INFO:teuthology.orchestra.run.smithi193.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-28T18:54:46.863 INFO:teuthology.orchestra.run.smithi007.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-28T18:54:47.876 INFO:teuthology.orchestra.run.smithi193.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-28T18:54:47.877 INFO:teuthology.orchestra.run.smithi193.stderr: 2023-11-28T18:54:47.878 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m14.346s 2023-11-28T18:54:47.878 INFO:teuthology.orchestra.run.smithi193.stderr:user 0m45.598s 2023-11-28T18:54:47.878 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m3.869s 2023-11-28T18:54:51.525 INFO:teuthology.orchestra.run.smithi007.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-28T18:54:51.782 INFO:teuthology.orchestra.run.smithi007.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-28T18:57:17.668 INFO:teuthology.orchestra.run.smithi007.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.44657.log.gz 2023-11-28T18:57:17.670 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-28T18:57:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:real 2m44.154s 2023-11-28T18:57:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:user 3m14.147s 2023-11-28T18:57:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m19.274s 2023-11-28T18:57:17.671 INFO:tasks.ceph:Archiving logs... 2023-11-28T18:57:17.671 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/remote/smithi007/log 2023-11-28T18:57:17.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T18:57:36.683 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/remote/smithi193/log 2023-11-28T18:57:36.685 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T18:57:42.790 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T18:57:42.802 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-28T18:57:42.818 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T18:57:42.822 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-28T18:57:42.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T18:57:42.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T18:57:42.913 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-28T18:57:42.925 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-28T18:57:42.926 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T18:57:42.931 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T18:57:42.947 INFO:teuthology.orchestra.run.smithi193.stdout: remote refid st t when poll reach delay offset jitter 2023-11-28T18:57:42.948 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================== 2023-11-28T18:57:42.948 INFO:teuthology.orchestra.run.smithi193.stdout:*hv01.front.sepi 44.4.53.2 2 u 20 128 377 0.108 -2.116 0.306 2023-11-28T18:57:42.948 INFO:teuthology.orchestra.run.smithi193.stdout:+hv02.front.sepi 63.231.80.2 3 u 24 128 377 0.063 -1.658 0.431 2023-11-28T18:57:42.948 INFO:teuthology.orchestra.run.smithi193.stdout:+hv03.front.sepi 74.6.168.72 3 u 70 128 377 0.086 -2.162 0.378 2023-11-28T18:57:42.948 INFO:teuthology.orchestra.run.smithi193.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-11-28T18:57:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2023-11-28T18:57:42.983 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2023-11-28T18:57:42.984 INFO:teuthology.orchestra.run.smithi007.stdout:*hv01.front.sepi 44.4.53.2 2 u 17 64 377 0.083 -2.586 0.198 2023-11-28T18:57:42.984 INFO:teuthology.orchestra.run.smithi007.stdout:+hv02.front.sepi 63.231.80.2 3 u 25 64 377 0.070 -1.746 0.224 2023-11-28T18:57:42.984 INFO:teuthology.orchestra.run.smithi007.stdout:+hv03.front.sepi 74.6.168.72 3 u 9 64 377 0.089 -2.380 0.138 2023-11-28T18:57:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-11-28T18:57:42.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-28T18:57:42.994 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-28T18:57:42.995 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-28T18:57:43.026 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-28T18:57:43.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-28T18:57:43.119 INFO:teuthology.task.internal:Duration was 2852.400497 seconds 2023-11-28T18:57:43.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-28T18:57:43.151 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-28T18:57:43.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T18:57:43.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T18:57:43.385 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-28T18:57:43.385 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2023-11-28T18:57:43.385 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T18:57:43.410 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2023-11-28T18:57:43.410 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T18:57:43.435 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-28T18:57:43.435 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T18:57:43.460 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T18:57:43.514 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-28T18:57:43.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T18:57:43.544 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T18:57:43.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-28T18:57:43.688 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-28T18:57:43.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T18:57:43.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T18:57:43.776 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-28T18:57:43.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T18:57:43.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T18:57:43.826 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2023-11-28T18:57:43.836 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2023-11-28T18:57:43.872 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T18:57:43.921 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi007.front.sepia.ceph.com, flagging run as failed 2023-11-28T18:57:43.921 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T18:57:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:57:43.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-28T18:57:43.942 INFO:teuthology.task.internal:Transferring archived files... 2023-11-28T18:57:43.944 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/remote/smithi007 2023-11-28T18:57:43.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T18:57:52.447 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-11-28T18:57:52.463 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/remote/smithi007/coredump/1701197116.59497.core.gz: gzip compressed data, was "1701197116.59497.core", last modified: Tue Nov 28 18:57:44 2023, max compression, original size modulo 2^32 264523776 2023-11-28T18:57:52.464 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-11-28T18:57:53.538 INFO:teuthology.task.internal: core looks like: /tmp/tmpk3mj62ls: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/ceph-mds', platform: 'x86_64' 2023-11-28T18:57:53.580 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i a 2023-11-28T18:57:53.581 DEBUG:teuthology.orchestra.run.smithi007:> which 'ceph-mds -f --cluster ceph -i a' 2023-11-28T18:57:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:57:53.594 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2023-11-28T18:57:53.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-28T18:57:53.702 INFO:teuthology.task.internal:Not uploading archives. 2023-11-28T18:57:53.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-28T18:57:53.744 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-28T18:57:53.745 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T18:57:53.757 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T18:57:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: 657881 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 28 18:57 /home/ubuntu/cephtest 2023-11-28T18:57:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: 657882 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Nov 28 18:10 /home/ubuntu/cephtest/archive 2023-11-28T18:57:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: 657885 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 18:57 /home/ubuntu/cephtest/archive/syslog 2023-11-28T18:57:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: 657887 4 -rw-rw-rw- 1 ubuntu ubuntu 1982 Nov 28 18:54 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-28T18:57:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: 657886 108 -rw-rw-r-- 1 ubuntu ubuntu 110028 Nov 28 18:57 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T18:57:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: 657888 100 -rw-rw-rw- 1 ubuntu ubuntu 99226 Nov 28 18:57 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-28T18:57:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: 657883 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 18:45 /home/ubuntu/cephtest/archive/coredump 2023-11-28T18:57:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: 661936 37728 -rw------- 1 root root 264523776 Nov 28 18:45 /home/ubuntu/cephtest/archive/coredump/1701197116.59497.core 2023-11-28T18:57:53.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T18:57:53.776 INFO:teuthology.orchestra.run.smithi007.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-28T18:57:53.777 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-28T18:57:53.804 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-28T18:57:53.818 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-28T18:57:53.856 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T18:57:53.857 INFO:teuthology.nuke:Checking targets against current locks 2023-11-28T18:57:53.865 INFO:teuthology.orchestra.run.smithi193.stdout: 658042 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 28 18:57 /home/ubuntu/cephtest 2023-11-28T18:57:53.866 INFO:teuthology.orchestra.run.smithi193.stdout: 658043 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 28 18:57 /home/ubuntu/cephtest/archive 2023-11-28T18:57:53.866 INFO:teuthology.orchestra.run.smithi193.stdout: 658046 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 18:57 /home/ubuntu/cephtest/archive/syslog 2023-11-28T18:57:53.866 INFO:teuthology.orchestra.run.smithi193.stdout: 658048 4 -rw-rw-rw- 1 ubuntu ubuntu 1847 Nov 28 18:54 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-28T18:57:53.867 INFO:teuthology.orchestra.run.smithi193.stdout: 658047 100 -rw-rw-r-- 1 ubuntu ubuntu 100134 Nov 28 18:57 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T18:57:53.867 INFO:teuthology.orchestra.run.smithi193.stdout: 658049 84 -rw-rw-rw- 1 ubuntu ubuntu 85501 Nov 28 18:57 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-28T18:57:53.868 INFO:teuthology.orchestra.run.smithi193.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-28T18:57:53.893 DEBUG:teuthology.nuke:shortname: smithi007 2023-11-28T18:57:53.894 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T18:57:53.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 17:57:36.705776', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ6dCvqETJOdN7f1Epo6cniZht7c5goNuYAOZUBrh9thfzQcM2F3QKi5mYaL+zO29hg6o7/fjXX4Y1jFJrsaJyc='} 2023-11-28T18:57:53.921 DEBUG:teuthology.nuke:shortname: smithi193 2023-11-28T18:57:53.921 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T18:57:53.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 17:57:36.703938', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBgCbSH1Nj/42ryCnkWgSNJF0gbI6Oe+/+KDqHSBCnArI3dbGEbt4LSUv3/4QJAvpTe6bSYl7wBvSQEb1HWidsk='} 2023-11-28T18:57:53.991 INFO:teuthology.orchestra.console.smithi007:Power off 2023-11-28T18:57:53.992 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T18:57:54.020 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2023-11-28T18:57:54.021 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T18:57:54.058 INFO:teuthology.orchestra.console.smithi193:Power off 2023-11-28T18:57:54.059 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T18:57:54.072 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-11-28T18:57:54.087 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2023-11-28T18:57:54.088 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T18:57:54.116 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2023-11-28T18:57:58.074 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T18:57:58.117 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T18:57:58.186 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-11-28T18:57:58.229 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2023-11-28T18:58:02.192 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T18:58:02.230 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T18:58:02.304 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-11-28T18:58:02.344 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2023-11-28T18:58:02.445 INFO:teuthology.orchestra.console.smithi193:Power off completed 2023-11-28T18:58:06.308 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T18:58:06.421 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2023-11-28T18:58:06.522 INFO:teuthology.orchestra.console.smithi007:Power off completed 2023-11-28T18:58:06.629 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} duration: 2852.4004967212677 failure_reason: 'Command failed (workunit test suites/ffsb.sh) on smithi007 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh''' flavor: default mon.a-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 mon.b-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d9d491af047e42f68657ad2fc302775b status: fail success: false 2023-11-28T18:58:06.629 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T18:58:06.710 INFO:teuthology.run:FAIL